Learning to Talk to Agents for Complex Task Solving
Explore how AI can solve complex tasks by interacting with agents in natural language, using the CommaQA dataset. Discover ways to break down complex tasks, reuse existing models, and improve interpretability in AI systems.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
https://github.com/allenai/CommaQA Hey AI, Can You Solve Complex Tasks by Talking to Agents? Not yet, but we can help Tushar Khot, Kyle Richardson, Daniel Khashabi, Ashish Sabharwal
Motivation: Solving Tasks by Talking to Agents Hey AI, Can you buy me the book series that has the kid with that lightning scar? 1. Solve a complex task by breaking them down into agent s capabilities Hey Google, Which book series has a kid with a lightning scar? Harry Potter 2. Interact with existing agents in their expected and natural language Hey Google, Which are the books in Harry Potter series? Can AI systems do the same? Hey Alexa, Buy Harry Potter and the Sorcerer's Stone Hey Alexa, Buy Harry Potter and the Sorcerer's Stone Hey Alexa, Buy Harry Potter and the Sorcerer's Stone Ordered!
Paper Summary New Task: Learning to Talk to Agents to Solve Complex Tasks 1. Solve a complex task by breaking them down into agent s capabilities Green AI: Reuse existing expensive and even proprietary models Better Long-Term Bet: No need to learn every task from scratch Interpretability: Naturally modular and interpretable systems 2. Interact with existing agents in their expected and natural language Technical Challenge: Search for solutions by interacting with NL agents How can we build AI systems to solve this task? Can AI systems do the same? New Dataset: CommaQA: Communicating with Agents for QA o Synthetic Multi-hop QA Dataset solvable using agents o Challenging for current black-box models and task baselines
Learning to Talk to Agents: Example Task Which Harry Potter Book has a car on its cover? What is the list of books in LOTR series Which movies are in the Fast & Furious series What is the list of books in the Harry Potter series? Harry Potter and the Philosopher s Stone, Cover of The Hobbit book Poster of the 2 Fast 2 Furious movie foreach Cover of the Harry Potter and the Philosopher s Stone book Cover of the Harry Potter and the Philosopher s Stone book foreach Is there a car in ? Is there a truck in ? Is there a car in ? Yes No filter Harry Potter and the Chamber of Secrets Is there a mountain in ? 4
Learning to Talk to Agents: Task Definition Given: Should NOT rely on: Agents Training Data of the Agents Is there a truck in ? What is the list of books in LOTR series Cover of The Hobbit book Millions of egs ?? GPT-3 Examples of Valid Inputs Agents Internal Knowledge Which Harry Potter Book has a car on its cover? => Harry Potter and the Is it okay to play chess at 3 AM? => Yes Chamber of Secrets Complex Task Supervision GPT-3 ?? To Do: Conversation Supervision Learn to solve the complex task by talking to agents 5 This is not a standard QA task to answer the question a system must communicate with the agents
CommaQA Dataset: Communicating with Agents for QA CommaQA-I Multi-hop QA Dataset designed for the learning to talk task Agents: Single-hop QA over different modalities Three different styles of multi-hop reasoning: KBQA TextQA What is IPhone a type of? What objects does Apple make? What objects has Steve Jobs likely used? => [ Cell What awards have the actors of the Oscar-winning movies received? => [ Oscar , ] Phone , ] CommaQA-E Implicit Decomposition Inspired by OpenBookQA (Mihaylov et al. 18), StrategyQA TableQA TextQA (Geva et al. 21) What awards has Beyonc won? Which movies have won Oscar? What awards has Sian Heder won? Who acted in Godfather? CommaQA-N MathQA TextQA What is the maximum value in [35.0, 15.0, 25.0]? What was the length of javelin throws by Michael Shey ? What awards have the actors of the Oscar-winning movies What awards have the actors of the Oscar-winning movies received? => [ Oscar , ] received? => [ Oscar , ] What was the gap between the longest javelin throws from USA and Ireland? What awards have the actors of the Oscar-winning movies received? => [ Oscar , ] => 2.0 Explicit Decomposition Inspired by HotpotQA(Yang et al. 18), MusiQue (Trivedi et al. 21) Numeric Decomposition Inspired by DROP (Dua et al. 19) 6
CommaQA Dataset: Key Properties No conflicts with world knowledge -- Replace entities with made-up words, e.g., What awards have the actors of the Glag-winning movies received? TableQA TextQA What awards has Beyonc won? What awards has Sian Heder won? Needs dynamic interaction with agents -- Different agents may have the answer in each context foreach Is there a car in ? Needs handling of structured answers from agents -- Introduce novel executable operators for NL questions over list/dict answers Is there a car in ? Fast and noise-free agents for quick progress -- Agents implemented as fast Python functions (instead of LMs) Auxiliary supervision as stepping-stones -- Provide rich annotations on the KB, textual facts, decompositions 7
Results Unsolved using the current baselines that talk to agents Oracle Black-box models struggle even when given access to the agent s private knowledge KB Oracle But solvable by training on conversation supervision (oracle upper bound) TMN: Khot et al, 21 T5: Raffelet al. 20 UQA: Khashabi et al. 20 8
Conclusion Rather than learning a new model for each task, can we learn to use existing models/agents? Proposed Learning to Talk to Agents Task How do we study this problem and develop methods for this task without other confounding factors ? Proposed CommaQA Dataset Future Research Directions: Use semantics of the question in the search Reward-based approaches Please come to our poster for more details! Code/Data/Paper: https://github.com/allenai/CommaQA Questions? Email us: {tushark,kyler,danielk,ashishs}@allenai.org 9
UNUSED SLIDES UNUSED SLIDES 10
Can you buy me the book series that has the kid with that lightning scar? 1. Solved a task by breaking them down into sub-tasks Which book series has a kid with a lightning scar? 2. Interacted with existing agents in natural language Harry Potter What are the books in Harry Potter series? Can we build AI systems to do the same? Hey Alexa, Buy Harry Potter and the Sorcerer's Stone Hey Alexa, Buy Harry Potter and the Sorcerer's Stone Hey Alexa, Buy Harry Potter and the Sorcerer's Stone Ordered! 13
Motivation: Solve Tasks by using Agents TODO: Add text + animation Buy the entire book series with the kid with the lightning scar Hey Google, which book series has a kid with a lightning scar? Harry Potter Hey Google, what are the books in Harry Potter series? 1. Harry Potter and the Sorcerer s Stone 2. Harry Potter and the Chamber of Secrets ... Alexa, Buy the book "Harry Potter and the Sorcerer s Stone" and the Sorcerer s Stone" Alexa, Buy the book "Harry Potter Added Added 14
