Approaches to Domain Adaptation for NLP Analysis

Adapting Text instead of the Model : An Open Domain Approach Gourab Kundu, Dan Roth University of Illinois at Urbana-Champaign 1

Motivating Example #1 predicate Semantic role Original Sentence Wrong Scotty gazed at ugly gray slums . AM-LOC Transformed Sentence Correct! Scotty looked at ugly gray slums . A1 2

Motivating Example #2 Original Sentence He was discharged from the hospital after a two-day checkup and he and his parents had what Mr. Mckinley described as a celebration lunch in the campus. AM-TMP Predicate Wrong 3

Motivating Example #2 Transformed Sentence He was discharged from the hospital after a two-day examination and he and his parents had what Mr. Mckinley described as a celebration lunch in the campus. Correct! Predicate AM-TMP 4

Research Question Can text perturbation be done in an automatic way to yield better NLP analysis? We study this question in the context of semantic role labeling. We focus on improving the performance of SRL on a different domain 5

Outline Overview of Domain Adaptation Overview of Adaptation Using Transformations (ADUT) Transformation Functions Combination Strategy Experimental Results Conclusion 6

Domain Adaptation Models trained on one domain perform significantly worse on another domain Semantic Role Labeling: WSJ domain (76%), Fiction domain (65%) Important Problem for wide scale NLP Adaptation is a problem for many tasks of NLP There are many different domains where natural language varies Labeling is expensive and time consuming 7

Current Approaches to Domain Adaptation ChelbaAc04, Adaptation of a maximum entropy capitalizer: Little data can help a lot Daume07, Frustratingly Easy domain adaptation FinkelMa09, Hierarchical Bayesian domain adaptation Labeled Adaptation Uses labeled data from new domain Unlabeled Adaptation BlitzerMcPe06, Domain Adaptation with Structural Correspondence Learning HuangYa09, Distributional Representations for Handling Sparsity in Supervised Sequence Labeling Uses unlabeled data from new domain Combined Adaptation Combines labeled and unlabeled data JiangZh07, Instance Weighting for Domain Adaptation in NLP ChangCoRo10, The necessity of combining adaptation methods 8

Limitations (Retraining takes time) Limitations: Need to retrain the model -- can take a long time Retrain NLP Tool 1 NLP Tool 2 NLP Tool N Source Domain Unlabeled Data Target Domain Unlabeled Data Model SRL: 20 hours Retrain 9

Limitations (Some tools are hard to retrain) Limitations: Need to retrain other people s tools -- may need implementation No option for retraining NLP Tool 1 NLP Tool 2 NLP Tool N Source Domain Unlabeled Data Target Domain Unlabeled Data Model 10

Limitations (Insufficient Unlabeled Data) Limitations: Need significant unlabeled data -- may not be available (e.g. website) May not be sufficient NLP Tool 1 NLP Tool 2 NLP Tool N Source Domain Unlabeled Data Target Domain Unlabeled Data Model 11

Outline Overview of Domain Adaptation Overview of Adaptation Using Transformations (ADUT) Transformation Functions Combination Strategy Experimental Results Conclusion 12

ADaptation Using Transformations (ADUT) Traditional Approach: Adapt model for the new text Our Approach: Adapt text for the old model Transformed Sentences Model Outputs o1 o2 t1 t2 Tool Tool Sentence s Output o Transformation Module Combination Module Model tk ok Old System 13

Transformation Functions Definition: A Function that maps an instance to a set of instances Example: Replacement of a word with synonyms that are common in training data Properties: Label (Semantic role) Preserving Output examples are more likely to appear in Old Domain than input example 14

Categorization of Transformation Functions Resource Based Transformation Uses resources and prior knowledge Learned Transformations Learned from training data 15

Resource Based Transformation Replacement of Infrequent Predicate Replacement/Removal of Quoted String Replacement of Unknown Word (Word Cluster, WordNet) Sentence Simplification 16

Replacement of Infrequent Predicate (VerbNet) Intuition: Model makes better prediction over frequent predicates. Input Sentence Scotty gazed at ugly gray slums . Transformed Sentence Scotty looked at ugly gray slums . 17

Replacement/Removal of Quoted String Intuition: Parser works better on simplified quoted sentences. Transformed Sentences We just sit quiet. Input Sentence We just sit quiet , he said . He said, We just sit quiet . He said, This is good . 18

Replacing Unknown Word(Word Cluster, WordNet) Intuition: Parser & Model works better on known words. Input Sentence He was released after a two-day checkup. Transformed Sentence He was released after a two-day examination. 19

Sentence Simplification (1) Intuition: Parser & Model work better on simplified sentences. Input Sentence The science teacher and the students discussed the issue at the classroom . Delete PP Transformed Sentence The science teacher and the students discussed the issue. 20

Sentence Simplification (2) Input Sentence Simplify NP The science teacher and the students discussed the issue. Transformed Sentence The teacher discussed the issue. 21

Learned Transformation Rules Motivation: Identify a specific context in the input sentence Transfer the candidate argument to a simpler context in which the SRL is more robust Input Sentence Mr. Mckinley was entitled to a discount . pattern p=[-2,NP, ][-1,AUX, ][1, ,to] -2 -1 0 1 2. PP, to AUX, was NP, Mckinley ` A2

Context Component of Rules Input Sentence Mr. Mckinley was entitled to a discount . -1 0 1 2. -2 Rule: predicate p=entitle pattern p=[-2,NP, ][-1,AUX, ][1, ,to] Location of Source Phrase ns=-2 Replacement Sentence st= But he did not sing. Location of Replacement Phrase nt=-3 Label Correspondence function f={(A0,A2),(Ai,Ai, i 0)}

Replacement Component of Rules Motivation: Replacement Sentence But he did not sing . -4 -3 -2 -1 0 1 Rule: predicate p=entitle pattern p=[-2,NP, ][-1,AUX, ][1, ,to] Location of Source Phrase ns=-2 Replacement Sentence st= But he did not sing. Location of Replacement Phrase nt=-3 Label Correspondence function f={(A0,A2),(Ai,Ai, i 0)} 24

Semantic Role mapping component of Rule Input Sentence Transformed Sentence Replacement Sentence But he did not sing . Mr. Mckinley was entitled to a discount . -2 -1 0 1 2 -4 -3 -2 -1 0 1 A2 Apply SRL System Gold Annotation A0 Rule: predicate p=entitle pattern p=[-2,NP, ][-1,AUX, ][1, ,to] Location of Source Phrase ns=-2 Replacement Sentence st= But he did not sing. Location of Replacement Phrase nt=-3 Label Correspondence function f={(A0,A2),(Ai,Ai, i 0)} 25

Transforming a sentence by using rules for each phrase p in input sentence s for each rule R if applies to p R is the set of rules, learned from training data sentence t = transform( , p) r = semantic role of p in t using SRL model semantic role of p in s = map( , r) 26

Learning Transformation Rules Input: Predicate p, Semantic role r RGet Initial Rules (p, r) repeat S Expand Rules (R) Sort R S based on accuracy R Top rules in R S 27

Learning Transformation Rules Input: Predicate p, Semantic role r RGet Initial Rules (p, r) repeat S Expand Rules (R) Sort R S based on accuracy R Top rules in R S 28

Get Initial Rules (entitle, A2) Replacement Sentence Mr. Mckinley was entitled to a discount . But he did not sing . A2 Replacement Sentence I asked the man . 29

Learning Transformation Rules Input: Predicate p, Semantic role r RGet Initial Rules (p, r) repeat S Expand Rules (R) Sort R S based on accuracy R Top rules in R S 30

Expand Rule (?) Rule ?: Does not apply st = But he did not sing . nt = -3 p = ask sp = [-1,NP,I][0,VBD,asked][1,NP,man] ns = 1 f = {(A0,A2), (Ai,Ai, i 0)} He asked the man Applies Neighbor Rule of ?: st = ?.st nt = ?.nt p = ?.p sp=[-1,NP, ][0,VBD,asked][1,NP,man] ns = ?.ns f = ?.f 31

Learning Transformation Rules Input: Predicate p, Semantic role r RGet Initial Rules (p, r) repeat S Expand Rules (R) Sort R S based on accuracy R Top rules in R S 32

Calculate Accuracy (?) Example: A rule is correct Input Sentence Transformed Sentence Replacement Sentence he But did not sing . Mr. Mckinley was entitled to a discount . -1 0 -2 -2 1 2. -4 -3 -2 -1 0 1 Gold Annotation Apply SRL System A0 A2 A2 = f (A0) Correct! 33 33

Calculate Accuracy (?) Example: A rule makes a mistake Transformed Sentence Replacement Sentence Input Sentence But he did not sing . The movie was entitled a big success . -2 -2 -1 0 1 2. -4 -3 -2 -1 0 1 Gold Annotation Apply SRL System A0 A1 A2 = f (A0) Wrong 34 34

Outline Overview of Domain Adaptation Overview of Adaptation Using Transformation (ADUT) Transformation Functions Combination Strategy Experimental Results Conclusion 35

Combination using Integer Linear Programming Step1: Compute distribution of scores over labels for argument candidates Our SRL system classifies each phrase as a semantic role The system assigns a probability distribution over semantic roles for each argument For same argument in different sentences, compute the average A1 0.3 Transformed Sentence 1 Scotty gazed at ugly gray slums . AM-LOC 0.4 A1 0.6 AM-LOC 0.1 Transformed Sentence 2 Scotty looked at ugly gray slums . A1 0.45 Average AM-LOC 0.25 36 36

Inference via Integer Linear Programming Example of a Constraint: Two arguments can not overlap Input Sentence Scotty gazed at ugly gray slums. Goal: Find maximum likely semantic role assignment to all arguments without violating the constraints Solve an ILP 37 37

Outline Overview of Domain Adaptation Overview of Adaptation Using Transformations (ADUT) Transformation Functions Combination Strategy Experimental Results Conclusion 38

Results for Single Parse System (F1) Baseline ADUT 69.3(+3.8) 65.7(+2.8) 65.5 62.9 Charniak Parse based SRL Stanford Parse based SRL 39

Results for Multi Parse System (1) Punyakanok08 Toutanova08 Surdeanu07 (Cons) ADUT-Combined Huang10 73.8(+3.3) (Retrain) 70.5 69.2(-1.3) 68.8(-1.7) 67.8(-2.7) F1 40

Effect of each Transformation Baseline Replacement of Unknown words Replacement of Predicate Replacement of Quotes Sentence Simplification Transformation By Rules Together 69.3 67 66.8 66.4 66.2 66.1 65.5 F1 41

Conclusion Current Work We suggested a framework for adapting text to yield better SRL analysis We showed that adaptation is possible without retraining and unlabeled data We showed that simple transformations yield 13% error reduction for SRL Future Work: Applying framework to other domains and tasks Using unlabeled data to improve transformations Thank You. 42