Understanding Words in Context: Distributional Modeling by Katrin Erk

Distributional models Katrin Erk

You can get an idea of what a word means from observing it in context He filled the wampimuk, passed it around, and we all drank some We found a little hairy wampimuk sleeping behind a tree. (examples by Marco Baroni) Distributional modeling: Represent the meaning of a word through the contexts in which it is observed Similar words appear in similar contexts 2

What words can appear in these contexts? Word 1: drown, bathroom, shower, fill, fall, lie, electrocute, toilet, whirlpool, iron, gin Word 2: eat, fall, pick, slice, peel, tree, throw, fruit, pie, bite, crab, grate Word 3: advocate, overthrow, establish, citizen, ideal, representative, dictatorship, campaign, bastion, freedom Word 4: spend, enjoy, remember, last, pass, end, die, happen, brighten, relive

What words can appear in these contexts? bathtub apple Word 1: drown, bathroom, shower, fill, fall, lie, electrocute, toilet, whirlpool, iron, gin Word 2: eat, fall, pick, slice, peel, tree, throw, fruit, pie, bite, crab, grate democracy Word 3: advocate, overthrow, establish, citizen, ideal, representative, dictatorship, campaign, bastion, freedom day Word 4: spend, enjoy, remember, last, pass, end, die, happen, brighten, relive

What can you say about word number 5? bathtub appl e Word 1: drown, bathroom, shower, fill, fall, lie, electrocute, toilet, whirlpool, iron, gin Word 2: eat, fall, ripe, slice, peel, tree, throw, fruit, pie, bite, crab, grate day Word 4: spend, enjoy, remember, last, pass, end, die, happen, brighten, relive democracy Word 3: advocate, overthrow, establish, citizen, ideal, representative, dictatorship, campaign, bastion, freedom Word 5: eat, paint, peel, apple, fruit, juice, lemon, blue, grow

What can you say about word number 5? bathtub appl e Word 1: drown, bathroom, shower, fill, fall, lie, electrocute, toilet, whirlpool, iron, gin Word 2: eat, fall, ripe, slice, peel, tree, throw, fruit, pie, bite, crab, grate day Word 4: spend, enjoy, remember, last, pass, end, die, happen, brighten, relive democracy Word 3: advocate, overthrow, establish, citizen, ideal, representative, dictatorship, campaign, bastion, freedom orange Word 5: eat, paint, peel, apple, fruit, juice, lemon, blue, grow

Describing meaning through context Similar words appear in similar contexts apple, orange Measure similarity in meaning as similarity in contexts Caveat: If a word has multiple meanings, like orange , it will appear in a mixture of contexts How to describe the contexts of a word? Count other words nearby

Counting context words for apple They picked up red apples that had fallen to the ground She atea red apple Pick an apple. Eating apples is healthy Word count, 3-word context window, lemmatized a 2 be 1 eat 2 fall 1 have 1 healthy 1 pick 2 red 2 that 1 up 1

How can we compare two context word counts? Count how often apple occurs close to other words in a large text collection (corpus): eat 794 fall 244 ripe 47 slice 221 peel 208 tree 160 throw 145 fruit pie 156 bite 104 crab 88 109 Interpret counts as coordinates: Every context word becomes a dimension. fall apple eat

How can we compare two context word counts? Count how often apple occurs close to other words in a large text collection (corpus): eat 794 fall 244 ripe 47 slice 221 peel 208 tree 160 throw 145 fruit pie 156 bite 104 crab 88 109 Do the same for orange : eat 265 fall 22 ripe 25 slice 62 peel 220 tree 64 throw 74 fruit pie 111 bite 4 crab 8 4

How can we compare two context word counts? Then visualize both count tables as vectors in the same space: eat 794 fall 244 ripe 47 slice 221 peel 208 tree 160 throw 145 fruit pie 156 bite 104 crab 88 109 eat 265 fall 22 ripe 25 slice 62 peel 220 tree 64 throw 74 fruit pie 111 bite 4 crab 8 4 fall apple Similarity between two words as proximity in space orange eat

How can we compare two context word counts? Then visualize both count tables as vectors in the same space: eat 794 fall 244 ripe 47 slice 221 peel 208 tree 160 throw 145 fruit pie 156 bite 104 crab 88 109 eat 265 fall 22 ripe 25 slice 62 peel 220 tree 64 throw 74 fruit pie 111 bite 4 crab 8 4 fall apple Similarity between two words as proximity in space orange eat

What do we mean by similarity of vectors? Euclidean distance (a dissimilarity measure!): apple orange

Problem with Euclidean distance: very sensitive to word frequency! apple Braeburn

What do we mean by similarity of vectors? Cosine similarity: apple Use angle between vectors instead of point distance to get around word frequency issues orange

Using distributional models Predicting word similarity (WordSim353): doctor nurse 7.00 professor doctor 6.62 student professor 6.81 smart student 4.62 smart stupid 5.81 Similar: synonyms antonyms topically related words Doing the TOEFL test: provisions 1. stipulations 2. interrelations 3. jurisdictions 4. interpretations

Using distributional models Predicting priming effects free associations Finding (near-)synonyms: automatically building a thesaurus Related: use distributional similarity of documents (containing similar words) in Information Retrieval

Corpora in which to count words Corpus = text collection What works best for computing a distributional model? Moby Dick 2 years of the Wall Street Journal A collection of dating ads A collection of webpage texts

Corpora in which to count words Need to be available electronically Best possible match for today s English language in general Mixture of genres Mixture of authors Spoken and written Larger is better

Corpora in which to count (English) words Brown Corpus 1 million words Balanced corpus, mixture of genres British National Corpus 100 million words Balanced corpus, mixture of genres, spoken and written

Corpora in which to count (English) words English Gigaword corpus 1 billion words Short news articles Wikipedia dump 2 billion words UKWaC (UK web as corpus) 2 billion words Collection of webpages ending in .uk

What can we do with our word counts? blackboard

Background in philosophy of language: Wittgenstein, meaning as use Man kan fu r eine gro e Klasse von Fa llen der Benu tzung des Wortes Bedeutung wenn auch nicht fu r alle Fa lle seiner Benu tzung dieses Wort so erkla ren: Die Bedeutung eines Wortes ist sein Gebrauch in der Sprache. -- Wittgenstein, Philosophical Investigations For a large class of cases though not for all in which we employ the word meaning it can be explained thus: the meaning of a word is its use in the language. (translation: Anscombe/Stokhof)

Background in linguistics: Harris and Firth Zellig Harris (1957): Classify linguistic units by observing the contexts they occur in phonemes morphemes phrase types: noun phrase, verb phrase Not specifically about semantics John Firth (1957): collocations identify senses of a word by looking at groups of contexts in which it appears You shall know a word by the company it keeps

Background in psychology Landauer/Dumais 1998: A solution to Plato s problem How come you know so many words? A typical American seventh grader knows the meaning of 10-15 words today that she did not know yesterday. She must have acquired most of them as a result of reading because (a) the majority of English words are used only in print, (b) she already knew well almost all the words she would have encoun- tered in speech, and (c) she learned less than one word by direct instruction.

Background in psychology Many phenomena to do with word meaning can be simulated with distributional models Word similarity ratings (WordSim353): doctor nurse 7.00 professor doctor 6.62 student professor 6.81 smart student 4.62 smart stupid 5.81 How would you simulate this with a distributional model?

Background in psychology Many phenomena to do with word meaning can be simulated with distributional models Priming Hearing one word makes you react faster to a related word Hodgson data: election vote dove peace vase flower NOT: election peace How would you simulate that with a distributional model?

Background in psychology Are our mental representations of words, of concepts distributional? Do they represent the contexts in which a word has been encountered? Does the meaning of a word depend on the contexts in which it is used?

Background in psychology Concepts as distributional: Yes (Landauer and others): How else would we learn so many words? Text as one of the main ways in which we encounter words No (Barsalou and others): Perception is central to how we represent concepts Distributional and perceptual (Andrews/Vigliocco/Vinson and others) Both types of information seems to be relevant Simulations: Combining both makes for a better model