User Validation in Ontology Alignment: Challenges and Solutions
This content discusses the importance of user validation in ontology alignment, highlighting issues related to user profiles, system services, and identified challenges. It emphasizes the role of user involvement in improving the performance of automated systems and presents insights into how state-of-the-art systems are dealing with these issues through qualitative evaluations and experiments.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
User validation in ontology alignment Zlatan Dragisic1, Valentina Ivanova1, Patrick Lambrix1, Daniel Faria2, Ernesto Jim nez-Ruiz3 and Catia Pesquita4 1 Link ping University 2Gulbenkian Science Institute, Portugal 3University of Oxford, UK 4 Universidade de Lisboa, Portugal
Motivation Many automatic ontology alignment systems Limits to the performance of automated systems Users involvement in the alignment process enables the detection and removal of erroneous mappings the addition of alternative and potentially new mappings the adjustment of system settings OAEI - Best ranked systems 2011-2016 1 0.95 0.9 0.85 Anatomy F-measure Conference 0.8 Interactive anatomy 0.75 Interactive conference 0.7 0.65 0.6 2011 2012 2013 2014 2015 2016
Contributions Identification of issues related to user validation in ontology alignment User profile Interface System services Qualitative evaluation evaluating how state-of-the- art systems deal with the issues Experiments evaluating how systems deal with erroneous input from the user
Identified issues regarding user user profile profile Domain expertise of the user Technical expertise of the user Expertise with the alignment system Users can be expected to make mistakes
Identified issues regarding system services system services Issue CogZ Prompt Agreement Maker AlViz AML COMA LogMap SAMBO RepOSE Stage of involvement before, after, during, iterative a+i a a d b,a+i a b,a+i a+i Suggestions selection Threshold/ advanced filtering - - - - - System services Recomputation - - - Feedback Propagation Conflict detection/blocking/ revalidation - - - - - - - - - - ( 2, B ) ( 3, F ) ( 6, D ) ( 4, C ) ( 5, C ) ( 5, E ) sim suggest th discard
Identified issues regarding system services system services Issue CogZ Prompt Agreement Maker AlViz AML COMA LogMap SAMBO RepOSE Stage of involvement before, after, during, iterative a+i a a d b,a+i a b,a+i a+i Suggestions selection Threshold/ advanced filtering - - - - - System services Recomputation - - - Feedback Propagation Conflict detection/blocking/ revalidation - - - - - - - - - - (2,C) 0.5 0.5 (5,F) (6,F)
Identified issues regarding user interface user interface Issue CogZ Prompt Agreement Maker AlViz AML COMA LogMap SAMBO RepOSE - - - - - - - Visual information seeking tasks - - Visual analytics - - - - - - Overview Zoom Filter Details-on-demand Relate History Extract Alternative views - - - Grouping - - - Alignment presentation - - - - - - Validated/candidate mappings - - - - Metadata & context - - Ranking/ recommendations - - - - - - - - - - Provenance & justification - - - - - - - - - - - - - Mapping explanation Impact/ consequences - - - - - - - - - -
Identified issues regarding user interface user interface Issue CogZ Prompt Agreement Maker AlViz AML COMA LogMap SAMBO RepOSE - - - - - - - Visual information seeking tasks - - Visual analytics - - - - - - Alternative views - - - Grouping - - - Alignment presentation - - - - - - Validated/candidate mappings - - - - Metadata & context - - Ranking/ recommendations - - - - - - - - - - Provenance & justification - - - - - - - - - - - - - Mapping explanation Impact/ consequences - - - - - - - - - -
Identified issues regarding user interface user interface Issue CogZ Prompt Agreement Maker AlViz AML COMA LogMap SAMBO RepOSE - - - - - - - Visual information seeking tasks - - Visual analytics - - - - - - Alternative views - - - Grouping - - - Alignment presentation - - - - - - Validated/candidate mappings - - - - Metadata & context - - Ranking/ recommendations - - - - - - - - - - Provenance & justification - - - - - - - - - - - - - Mapping explanation Impact/ consequences - - - - - - - - - -
Identified issues regarding user interface user interface Issue CogZ Prompt Agreement Maker AlViz AML COMA LogMap SAMBO RepOSE - - - - - - - Visual information seeking tasks - - Visual analytics - - - - - - Alternative views - - - Grouping - - - Alignment presentation - - - - - - Validated/candidate mappings - - - - Metadata & context - - Ranking/ recommendations - - - - - - - - - - Provenance & justification - - - - - - - - - - - - - Mapping explanation Impact/ consequences - - - - - - - - - - Frame names or synonyms are similar
Identified issues regarding user interface user interface Issue CogZ Prompt Agreement Maker AlViz AML COMA LogMap SAMBO RepOSE - - Accept/reject - Create/refine - - Search - - Alignment Interaction User annotation - - - - - - - - - - - Session - Temporary mappings - - - - -
Experiments Purpose Simulate users with different expertise levels Show how systems service are affected by and cope with errors Interaction simulated via reference alignment Varying degrees of errors: 0%, 10%, 20%, 30% Anatomy (mid-size ontologies) and Conference track (small ontologies) Measures: Precision and recall Number of interactions Systems: AML JarvisOM LogMap ServOMBI
Experiments - results F-measure All tools improve with the all-knowing expert highest Jarvis 0.96 0.91 0.95 0.95 0.89 0.94 0.88 0.94 0.88 0.93 0.87 0.85 0.76 0.75 0.75 0.68 0.71 0.75 0.66 0.65 0.6 0.6 0.55 When increasing the error, performance deteriorates most affected Jarvis 0.49 0.45 0.35 0.25 0.17 0.15 non-interactive 0 10 20 30 AML JarvisOM LogMap ServOMBI Recall Precision 1 1 1 1 0.99 0.99 0.96 0.95 0.85 0.95 0.94 0.94 0.93 0.85 0.98 0.95 0.94 0.9 0.97 0.94 0.96 0.9 0.93 0.83 0.82 0.82 0.8 0.86 0.92 0.8 0.71 0.7 0.67 0.67 0.76 0.62 0.6 0.7 0.62 0.55 0.53 0.43 0.5 0.49 0.6 0.4 0.53 0.51 0.5 0.3 0.4 0.2 0.36 0.11 0.1 0.3 non-interactive 0 10 20 30 non-interactive 0 10 20 30 AML JarvisOM LogMap ServOMBI AML JarvisOM LogMap ServOMBI
Experiments - results F-measure More improvement w.r.t. recall: AML 0.96 0.91 0.95 0.95 0.89 0.94 0.88 0.94 0.88 0.93 0.87 0.85 0.76 0.75 0.75 0.68 0.71 More improvement w.r.t. precision: LogMap, ServOMBI 0.75 0.66 0.65 0.6 0.6 0.55 0.49 0.45 0.35 0.25 0.17 0.15 non-interactive 0 10 20 30 AML JarvisOM LogMap ServOMBI Recall Precision 1 1 1 1 0.99 0.99 0.96 0.95 0.85 0.95 0.94 0.94 0.93 0.85 0.98 0.95 0.94 0.9 0.97 0.94 0.96 0.9 0.93 0.83 0.82 0.82 0.8 0.86 0.92 0.8 0.71 0.7 0.67 0.67 0.76 0.62 0.6 0.7 0.62 0.55 0.53 0.43 0.5 0.49 0.6 0.4 0.53 0.51 0.5 0.3 0.4 0.2 0.36 0.11 0.1 0.3 non-interactive 0 10 20 30 non-interactive 0 10 20 30 AML JarvisOM LogMap ServOMBI AML JarvisOM LogMap ServOMBI
Experiments - results F-measure Errors impact precision: AML, JarvisOM 0.96 0.91 0.95 0.95 0.89 0.94 0.88 0.94 0.88 0.93 0.87 0.85 0.76 0.75 0.75 0.68 0.71 Errors impact recall: ServOMBI 0.75 0.66 0.65 0.6 0.6 0.55 0.49 0.45 0.35 0.25 0.17 0.15 non-interactive 0 10 20 30 AML JarvisOM LogMap ServOMBI Recall Precision 1 1 1 1 0.99 0.99 0.96 0.95 0.85 0.95 0.94 0.94 0.93 0.85 0.98 0.95 0.94 0.9 0.97 0.94 0.96 0.9 0.93 0.83 0.82 0.82 0.8 0.86 0.92 0.8 0.71 0.7 0.67 0.67 0.76 0.62 0.6 0.7 0.62 0.55 0.53 0.43 0.5 0.49 0.6 0.4 0.53 0.51 0.5 0.3 0.4 0.2 0.36 0.11 0.1 0.3 non-interactive 0 10 20 30 non-interactive 0 10 20 30 AML JarvisOM LogMap ServOMBI AML JarvisOM LogMap ServOMBI
Experiments - results Interactions Redundant requests Extrapolation from user feedback Balance in TP/TN 2500 2000 1500 1000 500 0 30 20 10 0 Total Distinct TP TN FP FN
Conclusions A step towards guidelines and best practices for good user interface design User profiles need to be supported (errors) Systems need to prioritize which mappings to present to the user Extrapolate knowledge by feedback propagation (double-edged sword) The presented mappings need to be explained Consequences Justification
