Principles of Mixed-Initiative User Interfaces Explained

Principles of Mixed-Initiative User Interfaces Eric Horvitz Hangxiao Zhu, 09/27/2024

Introduction Background & Motivation: Recent debate has centered on whether user- interface research should focus on developing new metaphors and tools for enhancing users ability to directly manipulate objects, or on creating interface agents that provide automation. The aim is to enhance user experiences by balancing direct manipulation and automation to address the limitations of relying solely on one approach.

Principles for Mixed-Initiative UI 1. Developing significant value-added automation 2. Considering uncertainty about a user s goals 3. Considering the status of a user s attention in the timing of services 4. Inferring ideal action in light of costs, benefits, and uncertainties 5. Employing dialog to resolve key uncertainties 6. Allowing efficient direct invocation and termination 7. Minimizing the cost of poor guesses about action and timing 8. Scoping precision of service to match uncertainty variation in goals 9. Providing mechanisms for efficient agent-user collaboration to refine results 10.Employing socially appropriate behaviors for agent-user interaction 11.Maintaining working memory of recent interactions 12.Continuing to learn by observing

LookOut Project Calendaring and Scheduling The system analyzes email content to infer scheduling-related information, such as dates and times, and automatically fills in appointment fields in the user's calendar The system allows users to invoke or adjust automated services, enhancing efficiency while maintaining user control over scheduling tasks

LookOut Project Decision Making under Uncertainty The system uses probabilistic models to infer the likelihood that a user intends to schedule or review a calendar based on email content Actions are guided by the expected utility of each option, balancing the costs and benefits of acting under uncertainty to improve user interaction

LookOut Project Multiple Interaction Modalities Manual Operation: Lookout can be manually invoked by the user through an icon, providing control over when the system takes action, allowing users to engage the system only when needed. Basic Automated-Assistance Mode: In this mode, Lookout automatically analyzes emails and populates calendar entries, reducing the need for manual input while still allowing users to adjust or confirm details as needed. Social-Agent Modality: This mode introduces an animated character that interacts with users via speech, utilizing speech recognition and text-to-speech for hands- free operation, offering a more conversational and less intrusive form of assistance.

LookOut Project Handling Invocation Failures If Lookout fails to infer that the user needs scheduling assistance, the user can manually invoke the system. If Lookout misinterprets the user s intent, it respectfully backs off, making minimal disruption

LookOut Project Inference and Decisions The system uses probabilistic models to infer user intentions based on email content and user behavior The current version of Lookout employs an efficient linear SVM approximation method to assign probabilities to user intentions. This allows the system to anticipate whether a user wants to schedule an appointment or review a calendar, enabling intelligent action under uncertainty Decision about action versus inaction should be directed by expected utility

LookOut Project Inference and Decisions Likelihood of a user's goals given observed evidence: The system calculates the probability that a user has a specific goal based on observed actions and context (e.g., email content). Utility definition: Utility is used to measure the value of different outcomes, such as taking action when the user has a goal or refraining from action when they do not. Expected utility equation: This equation calculates the overall value of an action by considering the likelihood of the user's goal and the associated utility for each possible outcome. Threshold probability: A specific probability point is calculated, beyond which it is better for the system to take action. If the likelihood of a user having a goal is higher than the threshold, the system acts; otherwise, it does not. Integrating dialog as an option for action: When uncertainty about the user s goal is high, the system can engage in a dialog with the user to clarify intentions before acting, providing a balance between direct action and inquiry.

LookOut Project User Attention and the Timing of Service By understanding the relationship between message size and preferred response time, Lookout can delay or time its actions to provide assistance when users are most ready to receive it

LookOut Project Machinery for Life-long Learning The system refines its probabilistic models and attention timing based on ongoing observations This life-long learning capability allows Lookout to adapt to individual user preferences and become more efficient and accurate in providing assistance as it gathers more data

Results and Contributions Integration of Automation and Direct Manipulation: The research demonstrates that combining automated reasoning with direct user manipulation can significantly enhance human-computer interaction, allowing users to retain control while benefiting from intelligent assistance Principles for Mixed-Initiative Interfaces: The paper presents a set of guiding principles for designing mixed- initiative user interfaces Lookout System: The development of the Lookout system showcases these principles in action, offering practical solutions for calendaring and scheduling tasks within Microsoft Outlook.