Unsupervised Cross-Task Generalization for Multi-Task NLP Models
Enhance the multi-task NLP models with unsupervised cross-task generalization through retrieval augmentation. Explore frameworks like ReCross to improve model performance across diverse tasks. Dive into the world of upstream training and hidden dimensions in this cutting-edge approach to natural language processing.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Unsupervised Cross-Task Generalization via Retrieval Augmentation https://inklab.usc.edu/ReCross/
Massively Multi-Task NLP Models Upstream Training upstreamtasks Input: Is this review positive or negative? Review: Best cast ... ? Output: Positive : amazon_plority ?? Make a title for news: British mathematician Stephen Haw... Input: Output: Mathematician Hawking ... Multi-Task NLP Model ?1, ,?? ?? : gigawords
CrossFit (Ye et al. 2021) upstreamtasks Multi-Task NLP Model T0 (Sanh et al. 2022) FLAN (Wei et al. 2022)
Unsupervised Cross-Task Generalization upstreamtasks Multi-Task NLP Model I want to ski! ??: an unseen task w/ a few unlabeled data Fred is ... He .... In the sentence, does the pronoun "he" refer to Fred? Yes, no? Input: Unsupervised Cross-Task Generalization Does "her" refer to Alice or Mary in this sentence: Alice ... Mary... and her ... Input: ??
Retrieval Augmentation ??: an unseen task w/ a few unlabeled data upstreamtasks Retrieve some additional data for re-learning relevant skills.
??: an unseen task w/ a few unlabeled data Fred is ... He .... In the sentence, does the pronoun "he" refer to Fred? Yes, no? Input: Unsupervised Cross-Task Generalization Does "her" refer to Alice or Mary in this sentence: Alice ... Mary... and her ... Input: ?? The ReCross Framework ?i ? ?? ? Reranker topK Encoders MIPS Dense Index ??
pair-wise score hidden dim #examples Dense Index Decoders Reranker cand. query upstreamdata distant sup. ? Encoders Make a title for news: British mathemat ...
Evaluation 1/8 size 5 different query sets: Q
Conclusion ReCross: a retrieval augmentation method for unsupervised cross-task generalization. Two ranking stages: simple & intuitive Significant improvement. Future directions More rigorous task/example representation for indexing. Parameter-efficient tuning for faster adaptation. Methods to use unlabeled data for guiding re-learning. .
