Comprehensive Guide to Evaluating Chatbot Quality with ISO 9241 Standards

Chatbot Use Case Questionnaire

Scope 1. Complete Mapping of Use Cases with ISO 9241 2. Questionnaire - User - Admin 3. List of Scenarios for Each Questions in the Questionnaire 2

Reference Knowledge Base from CMC Tree Mapping from ISO 9241 in Evaluating Quality of Chatbots and Intelligent Conversational Agents by Nicole Radziwill and Morgan Benton ISO 9241 s concept of usability: a. Efficiency b. Effectiveness c. Satisfaction 3

ISO 9241 - Efficiency Efficiency Category Quality Attribute Reference Performance Graceful degradation Robustness to manipulation Robustness to unexpected input Avoid inappropriate utterances and be able to perform damage control Effective function allocation, provides appropriate escalation channels to humans Cohen & Lane (2016) Thieltges (2016) Kluwer (2011) Morrissey and Kirakowski (2013) Staven (2017) 4

ISO 9241 - Effectiveness Effectiveness Category Quality Attribute Reference Functionality Accurate speech synthesis Interprets commands accurately Use appropriate degrees of formality, linguistic register Linguistic accuracy of outputs Execute requested tasks Facilitate transactions and follows up with status reports General ease of use Engage in on-the-fly problem solving Contains breadth of knowledge, is flexible in interpreting it Kuligowska (2015) Eeuwen (2017) Morrissey & Kirakowski (2013) Wallace (2003) Ramos (2017) Eeuwen (2017) Solomon (2017) Cohen & Lane (2016) Humanity Passes the Turing test Does not have to pass the Turing Test Transparent to inspection, discloses its chatbot identity Include errors to increase realism Convincing, satisfying, & natural interaction Able to respond to specific questions Able to maintain themed discussion Weizenbaum (1966); Wallace (2003) Ramos (2017) Bostrom & Yudkowski (2014) Coniam (2014) Morrissey & Kirakowski (2013) 5

ISO 9241 - Satisfaction Satisfaction Category Quality Attribute Reference Affect Provide greetings, convey personality Give conversational cues Provide emotional information through tone, inflection, and expressivity Exude warmth and authenticity Make tasks more fun and interesting Entertain and/or enable participant to enjoy the interaction Read and respond to moods of human participant Morrissey & Kirakowski (2013) Pauletto et al. (2013) Solomon (2017) Eeuwen (2017) Ramos (2017) Meira & Canuto (2015) Ethics and Behaviour Respect, inclusion, and preservation of dignity (linked to choice of training set) Ethics and cultural knowledge of users Protect and respect privacy Nondeception Sensitivity to safety and social concerns Trustworthiness (linked to perceived quality) Awareness of trends and social context Neff & Nagy (2016) Applin & Fischer (2015) Eeuwen (2017) Isaac & Bridewell (2014) Miner et al. (2016) Herzum et al. (2002) Vetter (2002) Accessibility Responds to social cues or lack thereof Can detect meaning or intent Meets neurodiverse needs such as extra response time and text interface Morrissey and Kirakowski (2013) Wilson et al. (2017) Radziwill & Benton (2017) 6

Overview of ISO 9241 Efficiency Performance Functionality Effectiveness ISO 9241 Humanity Affect Ethics and Behaviour Satisfaction Accessibility 7

ISO 9241 - Efficiency Efficiency Performance The chatbot should be able to respond immediately (max within 1 min excluding network factor) Functionality Effectiveness ISO 9241 Humanity Affect Ethics and Behaviour Satisfaction User Accessibility The Chatbot is able to trigger response from the user after 30 seconds of silence Efficiency Performance Live Chat Escalation process for unanswered questions Admin E-mail Call 8

ISO 9241 - Effectiveness Efficiency Performance Humanity Effectiveness ISO 9241 Functionality Affect Ethics and Behaviour Satisfaction Accesibility The chatbot able to get the context of the sentence Humanity User Effectiveness User Functionality Admin 9

ISO 9241 Effectiveness (Cont.) The chatbot should understand the intent of the questions, regardless of different ways of asking. Humanity BM EN Effectiveness Mix The bots are able to answer lengthy and complex questions User BM Functionality EN Admin BM Formal Language Mix BI Chatbot is able to provide predictive texts for auto suggest Language BM Informal Language EN Mix User Effectiveness Functionality BM Grammatical error EN Mix The chatbot is able to answer the questions without prior language selection and is able to handle/overcome grammatical errors, incomplete sentences and typos. BM Incomplete sentences EN Mix BM Incomplete punctuations EN Mix BM Short form/ social media languages EN Mix Emoji 10

ISO 9241 Effectiveness (Cont.) Humanity First impression Consistency Navigation of screen sequence Effectiveness Organization of dashboard components Look and Feel User Easy to understand terms Functionality Usage of terms throughout the admin dashboard Admin Walkthrough tutorial of system usage functionality BM Add EN BM Easy to handle the training content such as add, update or delete Update EN Delete BM Functionality Effectiveness Admin EN The process of training the Chatbot does not require technical knowledge/skills Programming Basic computer literacy Simultaneous admin management Basic background understanding in Artificial Intelligence Able to edit the confidence level of response 12

ISO 9241 Effectiveness (Cont.) Excel View by period Humanity Search Word Admin can monitor on-going or previous chat log Create in form of file PDF Effectiveness Category Daily Weekly Monthly Annually User Period Functionality Unanswered questions Admin Category Daily Weekly Period Monthly Most popular questions Sort Effectiveness Functionality Admin Annually Category Daily Weekly Monthly Annually Location User demography Period The admin can generate and view the analytical reports Daily Weekly Monthly Annually Time Hits Period Sort Daily Weekly Monthly Category of most popular products Period Annually Others Word Excel PDF Create in form of file 13

ISO 9241 - Satisfaction Sad Happy Angry The response given by the bot should be tailored according to the current emotion of the user. Efficiency Performance Functionality User The chatbot should be able to support links and multimedia content Effectiveness Links ISO 9241 Humanity File Attachment The chatbot can populate most popular questions and display at the welcome page as guidance to new users. Affect Image Video Ethics and Behaviour Satisfaction The chatbot should guide the user (clickable button) and also allow non-guided (allow free text input) for the user Accesibility Affect Context The user are able to give rating and comment at the end of conversation Emotion User details Able to extract information during live conversation for escalation purpose Admin Others Satisfaction News The chatbot can also notify user Promotions User Does the chatbot make user feel safe to share non-sensitive personal information Announcements Ethics and Behaviour Can the chatbot detect fraud attempt by the user e.g. (When the user is trying to request TAC number via chatbot) Admin Can the chatbot ensure escalated cases are followed up by live agent and know the case s status (Issues ticket number and is updated until the case is settled) Web Accessibility Social media The user are able to access the bot from different platform User Mobile Others 14

Thank You 15