Italian Textual Dataset for Emotion Recognition in HRI
Explore the creation and evaluation of an Italian textual dataset for emotion recognition in Human-Robot Interactions (HRI). Emotions play a crucial role in HRI, and this study delves into the challenges and intricacies of recognizing emotions in different contexts. Discover the state-of-the-art English and Italian datasets used in emotion research, and how ad-hoc dialogues and AI technologies shape the study's methodology.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Building and Assessing an Italian Textual Dataset for Emotion Recognition in Human-Robot Interactions Alessia Fantini1,5, Antonino Asta2, Alfredo Cuzzocrea3,4, Giovanni Pilato5 1University of Pisa, Pisa, Italy 2University of Palermo, Palermo, Italy 3iDEALab, University of Calabria, Rende, Italy 4Dept. of Computer Science, University of Paris City, Paris, France 5ICAR-CNR, Italian National Research Council, Palermo, Italy The 35th International Conference on Software Engineering & Knowledge Engineering
Outline Introduction Results Our Study Conclusions Methodology
Introduction - The context: Emotions are key factors during Human-Robot Interaction (HRI) but, at the same time, one of the most difficult tasks for robots during interaction with humans is emotion recognition Multidimensional nature Their understanding depends on the context
JOY SADNESS - - - Emotions can overlap They can be different from person to person The same context can generate one emotion at one time and a different emotion at another time But in general, we tend to be able to identify objective situations to which specific emotions are linked.
Introduction - State of the Art ENGLISH DATASET: EmotionX, Affect-Intensity Lexicon and Emotion Dataset (AILA), CrowdFlower s Emotion Dataset, Friends, EmoBank, SemEval-2018 Task 1: Affect in Tweets (AIT-2018), Sentiment140, Emotion Intensity Dataset (EmoInt), The International Survey on Emotion Antecedents and Reactions (ISEAR), The Stanford Sentiment and Emotion Classification (SSEC), The DEAP (Database for Emotion Analysis using Physiological Signals) Ekman s basic emotions News paper, books or dialogues found on the Internet, including those found from social media Movies or physiological signals ITALIAN DATASET: SEMEVAL-ITA-2018, ITA-EVALITA-2020, EmoLexIta, The STS-ITA (Sentences in the Wild - Italian), News-ITA
Our study - Features Use examples of interactions between people in which the emotions we want to focus on Avoid data from social media or newspaper articles Dialogue structure Ad hoc dialogues: provide the specific context in which certain emotions may emerge The labeling was not done directly by us: ChatGPT + Psychologist Basic emotions + emotions that may emerge during HRI (joy, sadness, anger, fear, surprise, disgust, frustration, embarrassment, boredom, nervousness, melancholy, guilt, hope, and stress) Analysis on dataset balance (distribution of gender, type of context, consistency between emotion and context, and quality of the interaction)
Our study - Purpose Build an Italian dataset for dialog-based emotion recognition We defined methodological criteria We exploited ChatGPT to help us develop them Psychology experts reviewed each conversation to analyze the balance of the dataset Methodology comprises three stages: dialogue generation procedure, data analysis, and improvements
Methodology - Procedure ChatGPT: 25 short conversation, of about five lines, between two people in which a specific emotion emerges. 14 Emotions Joy Sadness Anger Fear Surprise Disgust Frustration Embarrassment Boredom Nervousness Melancholy Guilt Hope Stress The original dialogues generated were retained, but we created a copy to edit them after performing the analysis 5 not using the word corresponding to the emotion. We labeled these kind of dialogues Without Word (W.W.) Tot. 420 Dialogues
Methodology -Analysis Consistency between context and emotion Y/N Gender distribution N, M, F Type of context Work, Leisure, Luck, Interpersonal sphere, Generic Quality of interaction Sufficient, Not much, No Percent relative frequency
Methodology - Improvements Several modifications, both grammatically and in terms of content About context: Distribution of the type of contexts: selecting those most inherent to interpersonal and social scenarios and generate new scenarios in order to make contexts heterogeneous The importance of the human expert intervening in all phases to direct ChatGPT to generate more focused dialogues Generate dialogues in which a specific emotion can emerge, considering the selected scenarios Select those dialogues that were more consistent with HRI
NOT CONSISTENT Results Global Analisys Consistency 86% Gender N=30% M=56% F=14% In a couple of cases confusion Context Overall appears heterogeneous, but it is unbalanced when observed in relation to specific emotions. E.g. Joy : Work , Leisure , Luck . No personal context Quality of interaction 65% sufficient , 25% not much , 10% no
alessia.fantini@icar.cnr.it alessia.fantini@phd.unipi.it Conclusions 1 2 3 4 5 Consistency between context and emotion is the main strength Type of contexts not much heterogeneous Gender distribution: large imbalance in favor of the male gender Reflect more generally on training AI systems that need to be as heterogeneous as possible Quality of the interaction almost always needs modification by the human operator Interpersonal and social spheres that reflect possible HRI situations Fast data generation Correct bias Gender bias and stereotype Grammatical errors, to adjust the language to the emotion, to make the dialogues more natural
