Optimizing Classifier Performance through Threshold Analysis

Examining classifier performance. 1

2

develop_a chd2018_a The AUROC summarizes the predictive performance at all unique thresholds. 3

1 = = = ix p Pr( 1| ) Y i p 1 exp + * x 0 j ij = 1 j 0 1 We predict the observation is not an event We predict the observation is an event = iy 4

Using thresholds for prediction = p C alc ulate 1,..., i n i ( ) Select a threshol d . . .5 e g p t p p 1 0 if if p p i t = y i i t 5

The confusion matrix. 6

The essential calculations are based on the confusion matrix. 7

The confusion matrix is a crosstabulation of the actual and predicted classes. y 0 1 y 0 1 It quantifies the confusion of the classifier for a single threshold 8

Re-do the final model for the develop_a data set 9

%let screened= MIPhone MICCBal Dep MM ILS MTGBal Income POS CD IRA brclus1 Sav NSF Age SavBal LOCBal NSFAmt Inv MIHMVal CRScore MIAcctAg InvBal DirDep CCPurc SDB CashBk AcctAge InArea ATMAmt b_DDABal DDA brclus2 CC HMOwn DepAmt Phone ATM LORes brclus4; /*do all subsets*/ ods output NObs=NObs bestsubsets=score; proc proc logistic logistic data=d.develop_a; model ins(event="1")=&screened resr resu / selection=score best=1 1; run run; /*Calculate Schwarz Bayes criterion */ data data _NULL_; set NObs; where label = 'Number of Observations Used'; call symput('obs',n); run run; data data subset; set score; sbc=-scorechisq+log(&obs)*(numberofvariables+1 1); run run; /*select model with lowest SBC put variable names in a macro variable*/ proc proc sql sql; select VariablesInModel into :selected from subset having sbc=min(sbc); quit quit; 10

Score the develop_a data set adjusting for over sampling, and define two, somewhat arbitrary, threshold predictions 11

/*run the selected model and score the model using priorevent option*/ proc proc logistic logistic data=d.develop_a descending plots=roc; model ins=&selected; score data=d.develop_a out=develop_scored priorevent=.02 run run; proc proc univariate univariate data=develop_scored; var p_1; histogram p_1; run run; .02; /*two arbitrary thresholds*/ data data develop_scored; set develop_scored; yhat1=p_1 ge .05 yhat2=p_1 ge .1 label yhat1= "Threshold=.05" run run; proc proc freq freq data=develop_scored; tables ins*(yhat1 yhat2); run run; .05; .1; yhat2= "Threshold=.10"; 12

13

14

Three essential quantities. = = = sensitivity y Pr( 1| 1) y = = = specificity y Pr( 0| 0) y = = Pr( 1) p y Over sampled data provide unbiased estimates of sensitivity and specificity.

In this case, the probability is known. It is the proportion of y=1 before oversampling = = = Pr( 1) .02 p y 16

= = = sensitivity y Pr( 1| 1) y 17

= = = specificity y Pr( 0| 0) y 18

Some additional measures. 19

True and false positive rate = = = = = = y y True Positive Rate=Pr( *sensitivity False Positive (1 )*(1 specifi ity p 1| 1) Pr( 1| 1 )Pr( 1) y y y p = = = = = = y y Rate= Pr( 1| 0) Pr( 0| 1)Pr( 0 ) y y y c ) 20

True and false negative rate = = = = = = y y True Negative Rate=Pr( (1 )*specificity False Negative Rate=P *(1 sensitivit p 0| 0) Pr( 0| 0)Pr( 0) y y y p = = = = = = y y r( 0| 1) Pr( 1| 0)Pr( 1) y y y y ) 21

Depth depth=(True Positive Rate)+(False Positive Rate) 22

Positive Predicted Value -- What is the percentage of true 1s among those we predict as 1 s? true positive rate depth Positive Predicted Value= 23

Lift: Positive Predicted Value p 24

Use the outroc= option in the score statement to get the essential measures. proc proc logistic logistic data=d.develop_a descending plots=roc; model ins=&selected; score data=d.develop_a out=develop_scored priorevent=.02 run run; proc proc contents contents data=roc;run .02 outroc=roc; run; 25

26

Calculate the measures data data roc1; set roc(rename=(_sensit_=sensitivity _prob_=threshold)); p=.02 .02;/*probability y=1 in original sample*/ specificity=1 1-_1MSPEC_;/*specificity*/ true_pos=p*sensitivity; false_neg=p*(1 1-sensitivity); true_neg=(1 1-p)*specificity; false_pos=(1 1-p)*(1 1-specificity); depth=true_pos+false_pos; pos_pred_val=true_pos/depth; neg_pred_val=true_neg/(1 1-depth); accuracy=true_pos+true_neg; lift=pos_pred_val/p; drop _1mspec_ _FALNEG_ _FALPOS_ _neg_ _pos_; run run; proc proc contents contents data=roc1;run run; 27

A Gains chart. Plot positive predicted value vs depth proc proc sgplot sgplot data=roc1; /*restict to a reasonable range of probabilities to consider*/ where threshold > .005 .005 and threshold<.5 title "Gains Chart"; title2 "Plot positive predicted value vs depth"; series x=depth y=pos_pred_val/ lineattrs=(thickness=2 2); refline 1 1/axis=y; run run; title; .5; 28

29

Lift chart plot lift vs depth proc proc sgplot sgplot data=roc1; /*restict to a reasonable range of probabilities to consider*/ where threshold > .005 .005 and threshold<.5 title "Lift Chart"; title2 "Plot lift vs depth"; series x=depth y=lift/ lineattrs=(thickness=2 2); yaxis values=(0 0 to 7 7 by 1 1); refline 1 1/axis=y; run run; title; .5; 30

31

Ideally, one would like large values of all these statistics. The context of the problem determines which of these measures is the primary concern. A database marketer might be most concerned with PV+ because it gives the response rate for the customers that receive an offer. A fraud investigator might be most concerned with sensitivity because it gives the proportion of frauds that would be detected. 32