Business Intelligence and Analytics: Visualizing Model Performance Session 9
Explore the concepts of ranking, profit curves, ROC graphs, AUC, LIFT curves, and more in Session 9 of the ISYS8036 Business Intelligence and Analytics course. Understand how these visualizations can optimize model performance for better decision-making.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Course : ISYS8036-Business Intelligence and Analytics VISUALIZING MODEL PERFORMANCE Session 9
Agenda Instance Ranking Profit Curves ROC Graph and ROC Curve Area Under ROC (AUC) LIFT Curves
Ranking Instead of Classifying
Profit Curves Therearetwocriticalconditionsunderlyingtheprofitcalculation: Theclasspriors Theproportionofpositiveandnegativeinstancesinthetarget population Thecostsandbenefits Theexpectedprofitisspecificallysensitivetotherelativelevelsofcosts andbenefitsforthedifferentcellsofthecost-benefitmatrix Reality..???
Generating ROC curve: Algorithm Sortthetestsetbythemodelpredictions Startwithcutoff=max(prediction) Decreasecutoff,aftereachstepcountthenumberoftruepositives TP(positiveswithpredictionabovethecutoff)andfalsepositivesFP (negativesabovethecutoff) CalculateTPrate(TP/P)andFP(FP/N)rate PlotcurrentnumberofTP/PasafunctionofcurrentFP/N
ROC Graphs and Curves ROCgraphsdecoupleclassifierperformancefromtheconditions underwhichtheclassifierswillbeused ROCgraphsareindependentoftheclassproportionsaswellasthe costsandbenefits Notthemostintuitivevisualizationformanybusinessstakeholders
Area Under the ROC Curve (AUC) Thearea under a classifier s curve expressed as a fraction of the unitsquare Itsvaluerangesfromzerotoone TheAUCisusefulwhenasinglenumberisneededtosummarize performance,orwhennothingisknownabouttheoperating conditions AROCcurveprovidesmoreinformationthanitsarea EquivalenttotheMann-Whitney-Wilcoxonmeasure AlsoequivalenttotheGiniCoefficient(withaminoralgebraic transformation) Bothareequivalenttotheprobabilitythatarandomlychosenpositive instancewillberankedaheadofarandomlychosennegativeinstance
Lets focus back in on actually mining the data.. WhichmodelshouldTelCo selectinordertotarget customerswithaspecialoffer, priortocontractexpiration?
Performance Evaluation Training Set: Model Accuracy 95% Classification Tree Logistic Regression ?-Nearest Neighbors Na ve Bays 93% 100% 76% Test Set: Model Accuracy 91.8% 0.0 AUC ClassificationTree LogisticRegression ?-NearestNeighbors Na veBays 0.614 0.014 93.0% 0.1 0.574 0.023 93.0% 0.0 0.537 0.015 76.5% 0.6 0.632 0.019
Performance Evaluation Na ve Bayes confusion matrix: p n 127 (3%) 200 (4%) 848 (18%) Y N 3518 (75%) ?-Nearest Neighbors confusion matrix: p n 3 (0%) 324 (7%) 15 (0%) Y N 4351 (93%)
References Provost, F.; Fawcett, T.: Data Science for Business; Fundamental Principles of Data Mining and Data- Analytic Thinking. O Reilly, CA 95472, 2013.
Thank You Thank You
