Performance Evaluation in Information Retrieval Systems
Dive into performance evaluation of information retrieval systems in Lecture #23 by Dr. Adnan Abid. Explore the difficulties in evaluating IR systems, measures for search engines, and methods to measure user happiness. Discover why system evaluation is crucial and the challenges involved in determining effectiveness and relevancy.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
INFORMATION RETRIEVAL INFORMATION RETRIEVAL TECHNIQUES TECHNIQUES BY DR. ADNAN ABID Lecture # 23 Performance Evaluation of Information Retrieval Systems 1
ACKNOWLEDGEMENTS ACKNOWLEDGEMENTS The presentation of this lecture has been taken from the underline sources 1. Introduction to information retrieval by Prabhakar Raghavan, Christopher D. Manning, and Hinrich Sch tze 2. Managing gigabytes by Ian H. Witten, Alistair Moffat, Timothy C. Bell 3. Modern information retrieval by Baeza-Yates Ricardo, 4. Web Information Retrieval by Stefano Ceri, Alessandro Bozzon, Marco Brambilla
Outline Outline Why System Evaluation? Difficulties in Evaluating IR Systems Measures for a search engine Measuring user happiness How do you tell if users are happy? 3
Why System Evaluation? Why System Evaluation? There are many retrieval models/ algorithms/ systems, which one is the best? What is the best component for: Ranking function (dot-product, cosine, ) Term selection (stopword removal, stemming ) Term weighting (TF, TF-IDF, ) How far down the ranked list will a user need to look to find some/all relevant documents? 4
Difficulties in Evaluating IR Systems Difficulties in Evaluating IR Systems Effectiveness is related to the relevancy of retrieved items. Relevancy is not typically binary but continuous. Even if relevancy is binary, it can be a difficult judgment to make. Relevancy, from a human standpoint, is: Subjective: Depends upon a specific user s judgment. Situational: Relates to user s current needs. Cognitive: Depends on human perception and behavior. Dynamic: Changes over time. 5
Measures for a search engine Measures for a search engine How fast does it index Number of documents/hour (Average document size) How fast does it search Latency as a function of index size Expressiveness of query language Ability to express complex information needs Speed on complex queries Uncluttered UI Is it free? 6
Measuring user happiness Measuring user happiness Issue: who is the user we are trying to make happy? Depends on the setting Web engine: User finds what s/he wants and returns to the engine Can measure rate of return users User completes task search as a means, not end See Russell http://dmrussell.googlepages.com/JCDL-talk- June-2007-short.pdf eCommerce site: user finds what s/he wants and buys Is it the end-user, or the eCommerce site, whose happiness we measure? Measure time to purchase, or fraction of searchers who become buyers? 7
Measuring user happiness Measuring user happiness Enterprise (company/govt/academic): Care about user productivity How much time do my users save when looking for information? Many other criteria having to do with breadth of access, secure access, etc. 8
How do you tell if users are happy? How do you tell if users are happy? Search returns products relevant to users How do you assess this at scale? Search results get clicked a lot Misleading titles/summaries can cause users to click Users buy after using the search engine Or, users spend a lot of $ after using the search engine Repeat visitors/buyers Do users leave soon after searching? Do they come back within a week/month/ ? 9
