Hierarchical Generalized Linear Models for Behavioral Health Readmission Rates

Biography Allen Hom, Ph.D. has been a Senior Research Analyst at Optum, a division of UnitedHealth Group (UHG) for over 7 years. Dr. Hom s areas of specialty are advanced statistics and behavioral health analytics, with more than 15 years of SAS programming experience. Prior to joining Optum, he worked as a consultant SAS programmer at McKesson Specialty and Abbott Vascular Devices. He has authored and co-authored several peer reviewed journal articles and scientific abstracts. Allen Hom, PhD Senior Research Analyst Optum Consumer Solutions Group Healthcare Analytics 425 Market Street, San Francisco 94105 (415) 547-5813 Allen.Hom@optum.com

Hierarchical generalized linear models for behavioral health risk-standardized 30- day and 90-day readmission rates Allen Hom, PhD Western Users Of SAS Software, Inc. 24th Annual Educational Forum And Conference San Francisco, California September 7-9, 2016

Introduction The modeling of readmission rates that the Centers for Medicare & Medicaid Services (CMS) uses has been limited to physical health problems without complete consideration of mental health and substance abuse related issues. The goal of this risk adjustment exercise is to develop readmission rate models. specially for mental health and substance abuse disorders. in a way that accounts for differences across hospitals in patient demographic and clinical characteristics that might be related to readmission rates. 3

Modelling approach Hospital-level 30-day and 90-day Risk Standardized Readmission Rates (RSRR) for Substance Use Disorders (SUD) and Mental Health (MH) readmissions were estimated with Hierarchical Generalized Linear Models (HGLM) to adjust for patient clustering (hierarchically correlated) effects around hospitals. HGLM simultaneously models two levels (patient and hospital) - Patient level models log-odds of hospital readmission using age, sex, public sector, selected clinical diagnosis covariates, and a hospital-specific intercept. - Hospital level random intercept after accounting for correlated patient risks. Patient level effects depend on the variance of the random hospital effect. - Hospitals are declared random subjects (in PROC GLIMMIX) because they are selected from a larger population to which we want to generalize. Slopes are assumed to be fixed. That is, hospitals are assumed to respond the same way to SUD and MH readmissions. - In addition, the random effects are assumed to be normally distributed. 4

Standardized Readmission Ratio (SRR) The Standardized Readmission Ratio (SRR) is calculated as the ratio of the predicted number of readmissions to the number of expected readmissions for a given hospital. Each model results in the (a) adjusted number of readmissions which are predicted (numerator) on the basis of the hospital's performance with its observed clinical/patient mix and (b) number of readmissions expected (denominator) on the basis on the national performance with that hospital's clinical/patient mix. The MH sample (N= 63,975) was 81% of the 30-day readmission sample (n=78,761) and 81% of the 90-day (N=60,547) readmission sample (n=74,540). The SUD sample (N=14,786) was 19% of the 30-day readmit sample, and 19% of the 90-day (N=13,993) readmit sample. In short, the MH sample was four times larger than the SUD sample. For a time period (e.g., 30-day), two models (MH or SUD) are run based on the index admission diagnosis. SRRCj = predictCj / expectCj c = admissions in cohort (SUD, MH) j = hospital 5

Standardized Readmission Ratio (SRR) and Risk Standardized RR (RSRR) Two SRRCj are computed: SRR_SUD and SRR_MH. SUD RSRR = SUD SSR x SUD national raw readmission rate. MH RSRR = MH SSR x MH national raw readmission rate. To report a single SRR for Hospital-Wide Readmission (SRR_HWR), the SUD and MH SRRs are pooled to compute a composite hospital volume weighted (by admissions) logarithmic mean: SRRj = exp ( mcj log(Rcj))/Smcj) = SRR_HWR c = Admissions in cohort (SUD, MH) j = Hospital mcj = Admissions per cohort for hospital Rcj = SRR for the condition SRR HWR < 1 indicates a lower-than-expected readmission rate (better). SRR HWR > 1 indicates a higher-than-expected readmission rate (worse). A hospital-wide RSRR (e.g., 30-day) is computed as: SRR_HWR x national average readmission rate = 30-day RSRR for a hospital. The method is repeated to calculate the 90-day RSRR for a hospital. See Appendix: PROC SQL code to merge 30-day SUD and MH RSRR data and Appendix: SAS code to compute hospital-wide SRR_HWR and RSRR. 6

Definition of eligible admissions and exclusion criteria Definition of Eligible Admissions Patient must have continuous eligibility with the health plan for 12 months prior to the initial admission. Eligibility at least one day after the discharge date is required to ensure that the absence of a readmission is not due to loss of the behavioral health benefit. For 30-day readmission, the readmission had to be greater than or equal to two days and less than or equal to thirty days. Exclusion criteria for 30-day readmission In-hospital death Admissions with no paid amounts Patient is without at least 30 days post discharge continuous enrollment Patient is without at least 365 days of continuous enrollment prior to admission start date Admissions which occur within 30 days of discharge from another index admission will not be considered index admissions Transferred to another acute care facility Discharged against medical advice (AMA) Same day discharges Similar criteria was used to investigate 90 day readmission patterns 7

PROC GLIMMIX example for the SUD 30-day HGLM model ods graphics on; procglimmix data=pr_readmit30_sud (where =(&therapy=1)) method=rmpl empirical=classical maxopt=50 inititer = 50 noclprint maxlmmupdate=100 namelen=60 ; class rpt_fac_id ; ods output parameterestimates= &lib..read30_&therapy._est_adj_2014 (keep=effect estimate stderr); DX1_SCHIZO_RISK DX1_ANXIETY_RISK / dist=binary link=logit solution cl stdcoef ddfm=bw ; xbeta=_xbeta_; linp=_linp_; random intercept/subject=rpt_fac_id solution; random _residual_ ; output out=readm30_adj_sud pred(blup ilink)=predprob pred(noblup ilink)=expprob ; id xbeta linp admit_key pers_sys_id rpt_fac_id rpt_fac_region rpt_fac_st readmit30; /* variables selection based upon significant variables from previous logistic regression */ model readmit30 (event='1') = agecat publicsec sex DX1_DEPRESSIVE_RISK DX1_BIPOLAR_RISK run; ods graphics off ; nloptions tech=nmsimp maxiter=200 ; ods exclude solutionr; title "HGLM for &therapy 2014"; 8

Most important variables by cohort type in the HGLM analysis Rank 30-day SUD Readmit HGLM 1 SUD diagnosis in the past year 90-day SUD Readmit HGLM SUD diagnosis in the past year 2 Bipolar and related disorder diagnosis in the past year Depressive disorder diagnosis in past year Medicaid/Medicare PS health insurance 3 Bipolar and related disorder diagnosis in past year 4 Medicaid/Medicare - PS health insurance Depressive disorder diagnosis in the past year Rank 30-day MH Readmit HGLM 1 Bipolar related disorder diagnosis in past year 2 Schizophrenic spectrum and other psychotic disorder diagnosis in past year 90-day MH Readmit HGLM Depressive disorder diagnosis in past year Medicaid/Medicare PS health insurance 3 Depressive disorder diagnosis in past year Medicaid/Medicare PS health insurance Bipolar and related disorder diagnosis in past year SUD diagnosis in the past year SUD diagnosis in the past year Schizophrenic spectrum and other psychotic disorder diagnosis in past year 4 5 See Appendix Tables 1-4 for odds ratios and other statistics. 9

Cross-validation with revised RSRR models In latter half of 2015, Mathematica Policy Research evaluated the case-mix adjustment for three measures (readmissions, BH spending, and length of stay) to increase their credibility. Revised sample size 1) 30-day, n=79,906, and 2) 90-day, n=75,697 Summary of changes Mathematica suggested for the RSRR models: Remove from Readmit 30-day and Readmit 90-day Involuntary admission to hospital and Electro-Convulsive Therapy (ECT) Change the Age variable by disaggregating the 26 64 and 65+ categories Original: 1) <=12, 2) 13-17, 3) 18-25, 4) 26-64, 5) 65+ Revised: 1) <=12, 2) 13-17, 3) 18-25, 4) 26-39, 5) 40-64, 6) 65-74, 7) 75-102 Change the Product Type by adding new categories Original: 0 = Commercial 1 = Medicare or Medicaid Revised: 1 = Commercial 2 = Medicare 3 = Adult Medicaid (age GE 18) 4 = Youth Medicaid (where age LT 18) 5 = Dual Eligible 10

Cross-validation with revised RSRR models (continued) and results Add interaction terms: New interaction terms were combinations of the top 5 variables: 1) SUD 2) Bipolar Disorders 3) Depressive Disorders 4) Schizophrenic Disorders, and 5) Product Type A present / past time factor: interactions among current diagnoses and diagnoses in the past 90-days or past year. Age category and gender (demographic variables) were also involved in many interaction terms. Revised RSRR models (below left) produced very similar predicted outcome values (average rate, standard deviation) as our original RSRR models. Readmit 30-day: 11.5% 1.98% => 11.4% 1.94% Readmit 90-day: 20.2% 2.68% => 20.9% 2.83% The HGLM regression model configuration of the mental health disorders is robust and stable. The decision was made to retain our original RSRR models to remain consistent with the CMS approach. See 1) Appendix Table 5: Summary of cross-validation results, 2) Appendix Figure1: Distribution of original 30-day RSRR for hospitals with >= 25 episodes, and 3) Appendix Figure 2: Distribution of original 90-day RSRR for hospitals with >= 25 episodes. 11

Summary and conclusions The goal of this risk adjustment exercise is to develop readmission rate models specially for mental health and substance abuse disorders. in a way that accounts for differences across hospitals in patient demographic and clinical characteristics that might be related to readmission rates Based upon the CMS methods for Hospital-Wide Risk-Standardized Readmission Metrics, a robust hierarchical general linear case-mix model for 30-day and 90-day readmission was created based on behavioral health admissions. The cross-validation of the RSRR models with a sample that was 6 months apart showed that the means and variances were very similar, despite new interaction terms in the revised RSRR models. The case-mix models allow for comparison across hospitals by adjusting for patient demographic and clinical characteristics for actual types of cases that it handles. Further research is needed to investigate the reliability and power of the metrics for the 30-day and 90-day readmission models. 12

Thank you Allen Hom, PhD Senior Research Analyst Consumer Solutions Group Healthcare Analytics 425 Market Street, San Francisco 94105 (415) 547-5813 Allen.Hom@optum.com

APPENDICES

Appendix: defining substance use disorder and mental health cohorts Behavioral health admissions were divided into two categories based upon acute inpatient treatment programs within hospitals: Substance Use Disorder and Mental Health therapy. The MH group was composed of index admissions for all other DSM-5 primary or secondary diagnosis categories that were not SUD: Depressive Disorders Bipolar and Related Disorders Schizophrenia Spectrum & Other Psychotic Disorders Neurocognitive Disorders Anxiety Disorders Disruptive, Impulse-Control and Conduct Disorders Neuro-developmental Disorders Feeding and Eating Disorders Personality Disorders SUD describes a problematic pattern of using alcohol or another substance that results in impairment in daily life or noticeable distress. A person with this disorder will often continue to use the substance despite negative consequences. About 73% of the SUD group had DSM-5 primary or secondary diagnosis category of substance related and addictive disorder in the past year. In 2014, a national survey on drug use and health in the United States from the Substance Abuse and Mental Health Services Administration (SAMHSA) found that among adults with SUD diagnosis in the past year, 39.1% also had a co-occurring mental illness diagnosis in the past year. 15

Appendix: PROC SQL code to compute SUD SRR and SUD RSRR * ---- DERIVING SUD SRR) and SUD RSRRFOR EACH HOSPITAL ---- * Put the mean 30 day SUD national raw readmission rate into a macro variable ; procsql noprint; select mean(readmit30) into: ybar from readm30_adj_sud ; quit; %put &YBAR; /* mean 30 day SUD national raw readmission rate = 0.085351 */ * Create the permanent SUD Hospital level Risk Standardized Readmission Rates (RSRR) ; procsql; create table &lib..hosp_rsrr_sud_30day as select distinct rpt_fac_id as rpt_fac_id_sud, sum(readmit30) as readmit_cnt_sud, count(admit_key) as volume_sud, mean(readmit30) as sud_mod_30day_readmit_pct, mean(predprob) as pred_sud, mean(expprob) as exp_sud, (calculated pred_sud) / (calculated exp_sud) as srr_sud, /* SRR_SUD */ (calculated srr_sud) * &ybar as sud_rsk_adj_30day_readmit_pct /* SUD_RSRR */ from readm30_adj_sud group by rpt_fac_id_sud; quit; 16

Appendix: PROC SQL code to merge 30-day SUD and MH RSRR data * Put the mean 30 day readmission rate into a macro variable; procsql noprint; select mean(readmit30) into: HWYBAR from combine_30day; quit; %put &HWYBAR; /* mean combined 30 day readmission rate = 0.112877 */ /* merge the two MH and SUD files together; Need to retain facilities that have only a SUD RSRR or only a MH RSRR */ procsql; quit; create table combine2_30day as select a.rpt_fac_id_sud, a.readmit_cnt_sud, a.volume_sud, a.sud_mod_30day_readmit_pct, a.pred_sud, a.exp_sud, a.srr_sud, a.sud_rsk_adj_30day_readmit_pct, b.rpt_fac_id_mh, b.readmit_cnt_mh, b.volume_mh, b.mh_mod_30day_readmit_pct, b.pred_mh, b.exp_mh, b.srr_mh, b.mh_rsk_adj_30day_readmit_pct from &lib..hosp_rsrr_sud_30day a FULL JOIN &lib..hosp_rsrr_mh_30day b on a.rpt_fac_id_sud=b.rpt_fac_id_mh; 17

Appendix: SAS code to compute hospital-wide SRR_HWR and RSRR data &lib..srr_combined_30day (drop=rpt_fac_id_mh rpt_fac_id_sud ); retain rpt_fac_id ; set combine2_30day; rpt_fac_id=rpt_fac_id_mh; if rpt_fac_id_mh=' ' then rpt_fac_id=rpt_fac_id_sud; /* replace missing MH facility ID with SUD facility ID */ run; procsort data= &lib..srr_combined_30day; by rpt_fac_id ; run; data &lib..Rsrr_all_fac_30day (keep=rpt_fac_id mod_30day_readmit_admit_cnt readmit30_total rsk_adj_30day_readmit_pct mod_30day_readmit_pct); set &lib..srr_combined_30day; by rpt_fac_id ; ARRAY vars readmit_cnt_sud--mh_rsk_adj_30day_readmit_pct ; DO OVER vars; IF vars = . THEN vars = 0; /* zero out variables that have missing values */ END; /* To report a single SRR for Hospital-Wide Readmission (SRR_HWR), the SUD and MH SRRs are pooled to compute a composite hospital volume weighted (by admissions) logarithmic mean */ mod_30day_readmit_admit_cnt = volume_sud + volume_mh; readmit30_total = readmit_cnt_sud + readmit_cnt_mh; IF volume_sud >0 THEN SUD_NUM = volume_sud * LOG(SRR_SUD); ELSE SUD_NUM=0; IF volume_mh >0 THEN MH_NUM = volume_mh * LOG(SRR_MH); ELSE MH_NUM=0; total_num=sum(sud_num, mh_num); srr_hwr=exp(total_num/mod_30day_readmit_admit_cnt); /* SRR_HWR = SRRj = exp ( mcj log(Rcj))/Smcj) */ rsk_adj_30day_readmit_pct=(srr_hwr*&hwybar); /* Hospital-wide 30-day RSRR = SRR_HWR x national average readmission rate */ mod_30day_readmit_pct=(readmit30_total/mod_30day_readmit_admit_cnt);/* Unadjusted 30-day Readmit Rate */ run; 18

Appendix Table 1: Production facility readmit 30-day model for SUD % Acute Admissions Logistic Regression HGLM HGLM N=14,786 Variable Description For Variable OR (95% CL) T Value F Value Presence of primary or secondary substance- related and addictive disorder DX in past year Presence of primary or secondary bipolar and related disorder DX in the past year Presence of primary or secondary depressive disorder DX in the past year PublicSector 1= Medicare / Medicaid 0=Commercial 72.70% 2.19 (1.86, 2.58) 8.43 71.03 SUD Risk Bipolar DisorderRisk Depressive Disorder Risk 9.89% 1.85 (1.58, 2.18) 7.65 58.54 34.34% 1.64 (1.45, 1.86) 7.02 49.26 18.61% 1.69 (1.46, 1.95) 5.08 25.76 Gender female =1 male=0 37.47% 0.77 (0.68, 0.87) -3.81 14.55 Presence of primary or secondary anxiety disorder DX in the past year Presence of primary or secondary schizophrenic spectrum and other psychotic disorder DX in the past year Anxiety Disorder Risk 15.01% 1.46 (1.26, 1.69) 3.80 14.45 Schizophrenia Disorder Risk 4.60% 1.37 (1.09, 1.72) 2.61 6.80 <=12, 13-17 (123, .8%) 18-25 (4,255, 30.2%) 26-64 (9,210, 66.2%), >=65 (407, 2.9%) Age Category 0.82 (0.73, 0.92) -2.39 5.73 19

Appendix Table 2: Production facility readmit 90-day model for SUD % Acute Admissions For Variable Logistic Regression OR (95% CL) N= 13,993 Variable HGLM HGLM Description T Value F Value Presence of primary / secondary substance- related and addictive disorder DX in past year PublicSector 1= Medicare or Medicaid 0=Commercial Bipolar DisorderRisk related disorder DX in the past year Depressive Disorder Risk disorder DX in the past year <=12, 3-17 (118, .8%) 18-25 (4,255, 30.4%), 26-64 (9,210, 65.8%) >=65 (407, 3%) Anxiety Disorder Risk disorder DX in the past year Schizophrenia Disorder Risk and other psychotic disorder DX in past year Other Mental Disorder Risk disorder DX in the past year Gender female =1 male=0 Neurocognitive Disorder Risk neurocognitive disorder DX in the past year 72.49% 2.25 (2.00, 2.53) 10.92 119.24 SUD Risk 18.26% 1.73 (1.55, 1.94) 6.77 45.78 Presence of primary / secondary bipolar and 9.83% 1.58 (1.38, 1.80) 6.71 45.07 Presence of primary / secondary depressive 34.34% 1.47 (1.33, 1.61) 6.42 41.16 AgeCategory 0.73 (0.67, 0.80) -5.14 26.46 Presence of primary or secondary anxiety 14.92% 1.43 (1.27, 1.60) 4.94 24.38 Presence of primary / secondary schizophrenic 4.6% 1.39 (1.15, 1.67) 3.45 11.89 Presence of primary / secondary other mental 2.17% 1.54 (1.20, 1.99) 2.55 6.51 37.5% 0.92 (0.84, 1.01) -1.37 1.89 Presence of primary / secondary 0.81% 1.06 (0.69, 1.65) 0.38 0.14 20

Appendix Table 3: Production facility readmit 30-day model for MH % Acute Admissions Logistic Regression HGLM HGLM N= 63,975 Variable Top 8 Variables Description For Variable OR (95% CL) T Value F Value Bipolar DisorderRisk Schizophrenia Disorder Risk Depressive Disorder Risk Presence of primary / secondary bipolar and related disorder DX in the past year Presence of primary / secondary schizophrenic and other psychotic disorder DX in past year Presence of primary or secondary depressive disorder DX in the past year 25.64% 1.60 (1.51, 1.70) 13.7 187.65 23.6% 1.55 (1.45, 1.64) 13.46 181.30 50.24% 1.46 (1.38, 1.54) 13.27 176.18 1= Medicare / Medicaid 0=Commercial 33.53% 1.57 (1.48, 1.66) 12.99 168.84 Public Sector Presence of primary / secondary substance- related & addictive disorder DX in past year 19.00% 1.44 (1.36, 1.52) 10.31 106.23 SUD Risk Presence of primary or secondary other mental disorder DX in the past year Presence of primary or secondary anxiety disorder DX in the past year <=12 (2,318, 3.6%), 13-17 (12,909, 20.2%), 18-25 (12,320, 19.3%), 26-64 (30,373, 47.5%, >=65 (6,055, 9.5%) Other Mental Disorder Risk Anxiety Disorder Risk 6.68% 1.48 (1.37, 1.61) 7.76 60.22 19.00% 1.17 (1.10, 1.25) 4.95 24.48 AgeCategory 0.92 (0.89, 0.95) -4.74 22.44 21

Appendix Table 4: Production facility readmit 90-day model for MH % Acute Admissions Logistic Regression N= 60,547 Variable Top 8 Variables Description HGLM HGLM OR (95% CL) For Variable T Value F Value Presence of primary / secondary depressive disorder DX in the past year Depressive Disorder Risk Public Sector 1= Medicare or Medicaid 0=Commercial Presence of primary/secondary schizophrenic spectrum and other psychotic disorder DX in the past year Bipolar Disorder Risk related disorder DX in the past year Presence of primary / secondary substance- related & addictive disorder DX in past year <=12 (2,318, 3.6%), 13-17 (12,909, 20.5%), 18-25 (12,320, 19.5%), 26-64 (30,373, 47.3%) >=65 (6,055, 9.1%) Other Mental Disorder Risk disorder DX in the past year Trauma or Stress-Related Disorder Risk 50.32% 1.58 (1.51, 1.65) 19.06 363.23 32.89% 1.68 (1.60, 1.77) 18.44 339.61 Schizophrenia Disorder Risk 23.40% 1.69 (1.61, 1.78) 18.4 338.61 Presence of primary / secondary bipolar and 25.62% 1.60 (1.53, 1.69) 17.45 305.03 18.91% 1.50 (1.42, 1.57) 12.63 159.68 SUD Risk 0.88 (0.85, 0.90) -8.67 75.25 Age Category Presence of primary / secondary other mental 6.60% 1.43 (1.33, 1.54) 8.21 67.37 Presence of primary / secondary trauma or stressor-related disorder DX in the past year 16.18% 1.19 (1.13, 1.26) 5.82 33.71 22

Appendix Table 5: Summary of cross-validation results 10th 25th 50th 75th 90th 99th Facilities Mean SD Min Max Percentile Percentile Percentile Percentile Percentile Percentile Original 30-Day RSRR Model with >= 25 Episodes 785 1.98% 5.6% 9.7% 10.4% 11.3% 12.2% 13.5% 17.9% 35.9% 11.5% Revised 30 Day RSRR Model with >= 25 Episodes 805 1.94% 5.5% 9.7% 10.3% 11.1% 12.2% 13.2% 17.4% 30.3% 11.4% Original 90-Day RSRR Model with >= 25 Episodes 753 2.68% 9.9% 17.9% 18.9% 20.0% 21.1% 22.8% 31.0% 46.5% 20.2% Revised 90 Day RSRR Model with >= 25 Episodes 775 2.83% 10.4% 18.3% 19.3% 20.6% 22.0% 23.6% 32.3% 45.2% 20.9% 23

Appendix Figure1: Distribution of original 30-day RSRR for hospitals with >= 25 episodes 24

Appendix Figure 2: Distribution of original 90-day RSRR for hospitals with >= 25 episodes 25

Appendix: Bootstrap distribution and the bootstrap procedure To estimate the confidence interval around hospital (a random intercept for HGLM) means, CMS used a bootstrap procedure because of complications with determining the standard error for the random effect in the HGLM model. The bootstrap procedure was adapted from a CMS SAS macro for repeated resampling of hospitals with replacement . Hospitals with less than five inpatient admissions were excluded and resample size was increased to 6000 hospitals for each replication to minimize problems with the HGLM solution. Hospitals represent the hierarchical random effect and a random multiplier (to SE of random effect) was also computed during each replication. 3000 replications each were run for 30-day RSRR and 90-day RSRR bootstrap To eliminate extreme outliers (an artifact of bootstrapping), the 95th percentile for each overall bootstrap distribution was used as a cutoff value. The bootstrap means were equivalent to the respective RSRR means of the production validation sample of hospitals with 25 or more admits (from Table 5, original 30-Day and 90-Day RSRR Models). 26

Appendix Table 6: Bootstrap distributions compared to original RSRR model for hospitals with 25 or more readmissions 25th Percentile 50th Percentile 75th Percentile RSRR Model Facilities Mean SD Min Max Original 30-day 11.5% 1.98% 5.6% 785 10.4% 11.3% 12.2% 35.9% Bootstrap 30-day 4,036,081 11.5% 5.61% 0.0% 7.3% 11.1% 15.3% 26.2% Original 90-day 20.2% 2.68% 9.9% 753 18.9% 20.0% 21.1% 46.5% Bootstrap 90-day 4,048,198 20.0% 7.33% 0.2% 14.8% 20.0% 25.0% 37.5% 27

Appendix: Bootstrap procedure and 95% confidence intervals Hospital Specific Confidence Intervals (CI). Each hospital had more than 2,900 bootstrapped values for RSRR. The 2.5th and 97.5th percentiles were used to obtain the lower and upper confidence limits (LCL, UCL) for the 95% CI for each hospital. The average regional RSRR was computed from the original data for four United States regions: Central, Northeast, Southeast, and West. A hospital s 95% CI were evaluated against their average regional RSRR and the also the national average readmissions. A hospital received a passing score if one of two conditions were met: 1) If the UCL is lower than the corresponding average regional RSRR, or 2) If the average regional RSRR is within the confidence interval Due to the time-consuming nature of the bootstrap procedure (more than 8 hours for 3000 replications), the regional median method was retained. See Appendix: Bootstrap macro for 30-day readmit, random sampling and Appendix: Bootstrap macro for 30-day readmit, random standard error for two key concepts for generating simulated data. 28

Appendix Table 7: 95% Confidence interval comparisons for hospitals with 25 or more readmissions Measure Evaluation Frequency Percent Original Pass - Regional Median RSRR 323 51.43 30-Day Readmission 95% CI Pass - Regional Average RSRR 452 71.97 95% CI Pass - National Average Readmission 333 53.03 Original Pass - Regional Median RSRR 322 51.27 90-Day Readmission 95% CI Pass - Regional Average RSRR 444 70.7 95% CI Pass - National Average Readmission 288 45.86 29

Appendix: Bootstrap macro for 30-day readmit, random sampling %MACRO HWR_BOOTSTRAP (RESAMP= ); ODS RESULTS OFF; data R; * create a random seed value ; call streaminit(0); R = round((rand("Uniform") * 10),2); output; run; proc sql noprint; select R into: seed from R; quit; sasfile hospital load; /* SAMPLING HOSPITALS with volume GE 5 */ PROC SURVEYSELECT DATA=hospital (where=(volume GE 5)) METHOD=URS noprint sampsize=6000 OUT=Hosp seed=%eval(&SEED) rep=&RESAMP; run; sasfile hospital close; DATA Hosp2; RUN; proc sort data = hosp2; by rpt_fac_id; run; PROC SQL NOPRINT; order by b.replicate; SET Hosp; H=. ; DO I=1 TO NUMBERHITS; END; OUTPUT; H + 1; CREATE VIEW BSHP AS SELECT B.H, b.replicate, A.* FROM test3 AS A INNER JOIN Hosp2 AS B ON A.rpt_fac_id=B.rpt_fac_id QUIT; 30

Appendix: Bootstrap macro for 30-day readmit, random standard error data R; * create a random seed value ; call streaminit(0); R = round((rand("Uniform") * 10),2); output; run; proc sql noprint; quit; DATA RANDOM_EFFECT; RUN; proc sort data = readm30_adj_sud ; BY replicate rpt_fac_id admit_key ; run; sasfile readm30_adj_sud load; select R into: seed from R; SET SOLUTIONRS (KEEP=REPLICATE SUBJECT ESTIMATE STDERRPRED); LENGTH rpt_fac_id $12.; rpt_fac_id=SUBSTR(SUBJECT, 12); multiplier=rannor(&SEED); m_stderr=stderrpred*multiplier; DROP SUBJECT; DATA ALL_SUD (compress=char reuse=yes); MERGE readm30_adj_sud (IN=A) RANDOM_EFFECT; BY replicate rpt_fac_id; RUN; sasfile readm30_adj_sud close; LINP=XBETA + ESTIMATE; LINP_BS=LINP + m_stderr; P_XBETA=EXP(XBETA)/(1 + EXP(XBETA)); P_LINP=EXP(LINP)/(1 + EXP(LINP)); P_LINP_BS=EXP(LINP_BS)/(1 + EXP(LINP_BS)); IF readmit30 ^=. AND P_LINP_BS ^=. AND P_XBETA ^=.; KEEP replicate admit_key readmit30 P_XBETA m_stderr stderrpred P_LINP_BS rpt_fac_id ; 31