Word Embeddings for Natural Language Processing
"Learn about word embeddings and how they are essential for text processing with neural networks. Explore the concepts of word vectors, relationship preservation, and similarity in embedding words. Discover the properties and questions surrounding word embeddings, including space efficiency, finding the encoding function, and trade-offs in embedding space capacity. Dive into the process of learning word embeddings through meaningful sentence modifications and optimization techniques."
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Word embeddings Text processing with current NNs requires encoding into vectors. One-hot encoding: N words encoded by length N vectors. A word gets a vector with exactly one entry = 1, others 0. Very space inefficient, no natural language structure. Bag of words: Collection of words (along with number of occurences). No word order, no natural language structure.
Word embeddings: idea Idea: learn an embedding from words into vectors Prior work: Learning representations by back-propagating errors. (Rumelhart et al., 1986) A neural probabilistic language model (Bengio et al., 2003) NLP (almost) from Scratch (Collobert & Weston, 2008) A recent, even simpler and faster model: word2vec (Mikolov et al. 2013) http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/
Word embeddings: similarity Hope to have similar words nearby http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/
Word embeddings: relationships Hope to preserve some language structure (relationships between words). http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/
Word embeddings: properties Need to have a function W(word) that returns a vector encoding that word. Similarity of words corresponds to nearby vectors. Director chairman, scratched scraped Relationships between words correspond to difference between vectors. Big bigger, small smaller
Word embeddings: properties Relationships between words correspond to difference between vectors. http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/
Word embeddings: questions How big should the embedding space be? Trade-offs like any other machine learning problem greater capacity versus efficiency and overfitting. How do we find W? Often as part of a prediction or classification task involving neighboring words.
Learning word embeddings First attempt: Input data is sets of 5 words from a meaningful sentence. E.g., one of the best places . Modify half of them by replacing middle word with a random word. one of function best places W is a map (depending on parameters, Q) from words to 50 dim l vectors. E.g., a look-up table or an RNN. Feed 5 embeddings into a module R to determine valid or invalid Optimize over Q to predict better https://arxiv.org/ftp/arxiv/papers/1102/1102.1808.pdf http://colah.github.io/posts/2014-07-NLP-RNNs-Representations/
