Performance Improvement Projects in Behavioral Health Concepts

2021 -2022 PIP Tools Amy McCurry Schwartz, PIP Consultant September 24, 2021 Behavioral Health Concepts, Inc.

Introduction Purpose Update audience on revisions to PIP Development Tool Update audience on revisions to PIP Validation Tool Approach Applicable for MHP and DMC-ODS ( BHP ) Behavioral Health Concepts, Inc.

What is a Performance Improvement Project (PIP)? CMS: A PIP is a project conducted by the BHP that is designed to achieve significant improvement, sustained over time, in health outcomes and enrollee satisfaction. A PIP may be designed to change behavior at a member, provider, and/or BHP/system level. Source: CMS External Quality Review (EQR) Protocols, October 2019, Protocol 1. Validation of Performance Improvement Projects, p. 27. Behavioral Health Concepts, Inc.

Types of PIPs Clinical Non-Clinical At least one intervention relates to the direct treatment of beneficiaries There is not an intervention that relates to the direct treatment of beneficiaries Reminder calls; texts; emails; newsletters; new scheduling system . At least one indicator that measures (or is a proxy for) a clinical outcome or impact Requires at least one beneficiary- level indicator There must be meaningful association with outcomes or improved care for beneficiaries Behavioral Health Concepts, Inc.

What are these tools of which you speak? PIP Development Tool: Instructions PIP Steps PIP Worksheets PIP Validation Tool: Validation Questions Validation Ratings Who completes the tools? Behavioral Health Concepts, Inc.

PIP DEVELOPMENT TOOL WORKSHEETS 1 9: COUNTY RESPONSIBILITY VALIDATION TOOL STEPS 1 9: COUNTY RESPONSIBILITY SECTIONS 1 11: EQRO RESPONSIBILITY Step 1: Identify the PIP Topic Worksheet 1: PIP Topic Section 1: Review the Selected PIP Topic Step 2: Develop the Aim Statement Worksheet 2: Aim Statement Section 2: Review the PIP Aim Statement Step 3: Identify the PIP Study Population Worksheet 3: PIP Study Population Section 3: Review the Identified PIP Population Step 4: Describe the Sampling Plan Worksheet 4: Sampling Plan Section 4: Review the Sampling Method Step 5: Select the PIP Variables and Performance Measures Step 6: Describe the Improvement Strategy (Intervention) and Implementation Plan (CMS Identifies this as Step 8) Worksheet 5: PIP Variables and Performance Measures Worksheet 6: Improvement Strategy (Intervention) and Implementation Plan (CMS Identifies this as Worksheet 8) Section 5: Review the Selected PIP Variables and Performance Measures Section 6: Assess the Improvement Strategies (CMS Identifies this as Activity 1, Step 8) Step 7: Describe the Data Collection Procedures (CMS Identifies this as Step 6) Worksheet 7: Data Collection Procedures (CMS Identifies this as Worksheet 6) Section 7: Review the Data Collection Procedures (CMS Identifies this as Activity 1, Step 6) Step 8: Describe the Data Analysis and Interpretation of PIP Results (CMS Identifies this as Step 7) Step 9: Address the Likelihood of Significant and Sustained Improvement Through the PIP Worksheet 8: Data Analysis and Interpretation of PIP Results (CMS Identifies this as Worksheet 7) Section 8: Review Data Analysis and Interpretation of PIP Results (CMS Identifies this as Activity 1, Step 7) Worksheet 9: Likelihood of Significant and Sustained Improvement through the PIP Section 9: Assess the Likelihood that Significant and Sustained Improvement Occurred Section 10: Perform Overall Validation and Reporting of PIP Results Section 11: Perform Overall Validation and Reporting of PIP Results Behavioral Health Concepts, Inc.

Developed from the CMS EQR Protocols CMS EXTERNAL QUALITY REVIEW (EQR) PROTOCOLS October 2019 https://www.medicaid.gov/medicaid/quality-of-care/medicaid-managed-care/quality-of-care-external-quality- review/index.html Behavioral Health Concepts, Inc.

2021 2022 PIP Development Tool Changes Why Simplify Instructions More easily understood Remove Duplicative Questions Ease of Completion MHPs/DMC-ODSs may find it more helpful to organize the development (and writing) of the PIP in a different order than the order of the CMS Steps. Behavioral Health Concepts, Inc.

PERFORMANCEIMPROVEMENTPROJECT(PIP) DEVELOPMENTTOOL CalEQRO FY21-22 Reviews The Performance Improvement Project (PIP) Documentation Tool provides a structure for development and submission of PIPs. Based on the Centers for Medicare & Medicaid Services (CMS) EQR Protocol 1: Validation of Performance Improvement Projects (PIPs), the tool is designed to assist the MHP/DMC-ODS to address all required elements of a PIP. BACKGROUND PIPs are designed to achieve significant improvement, sustained over time, in health outcomes and enrollee satisfaction. They should have a direct beneficiary impact and may be designed to create improvement at a member, provider, and/or MHP/DMC-ODS system level. All MHPs/DMC-ODSs are required to have one active and ongoing clinical PIP and one active and ongoing non-clinical PIP each year as a part of the plan s quality assessment and performance improvement (QAPI) program, per 42 C.F.R. 438.330 and 457.1240(b). Each PIP will be evaluated annually by CalEQRO; every section should be reviewed and updated as needed to ensure continued relevance and to address changes to the study, including new interventions. Counties are encouraged to seek technical assistance (TA) throughout the year. Behavioral Health Concepts, Inc.

CMS Steps and PIP Worksheets CMS STEPS 1 9 WORKSHEETS 1 9 Step 1: Identify the PIP Topic Worksheet 1: PIP Topic Step 2: Develop the Aim Statement Worksheet 2: Aim Statement Step 3: Identify the PIP Population Worksheet 3: PIP Study Population Step 4: Describe the Sampling Method Worksheet 4: Sampling Plan Step 5: Select the PIP Variables and Performance Measures Worksheet 5: PIP Variables and Performance Measures Step 6: Describe the Data Collection Procedures Worksheet 6: Improvement Strategy (Intervention) and Implementation Plan Step 7: Describe the Data Analysis and Interpretation of PIP Results Worksheet 7: Data Collection Procedures Step 8: Describe the Improvement Strategy (Intervention) and Implementation Plan Worksheet 8: Data Analysis and Interpretation of PIP Results Step 9: Address the Likelihood of Significant and Sustained Improvement Through the PIP Worksheet 9: Likelihood of Significant and Sustained Improvement Through the PIP Behavioral Health Concepts, Inc.

Worksheets ordered for Development of a PIP Worksheet 1: The PIP Topic What is the problem? Worksheet 2: The Aim Statement What do we want to do? Worksheet 3: The Study Population Who do we intend to help? Worksheet 4: The Sampling Method How do we select a smaller group to study? Worksheet 5: PIP Variables and Performance Measures How will we know if what we are doing makes a difference? Behavioral Health Concepts, Inc.

Worksheets ordered for Development of a PIP Worksheet 6: Improvement Strategy (Intervention) and Implementation Plan What specifically will we do to cause the change? Data Collection Procedures Worksheet 7: What data do we need, and how will we get it? Worksheet 8: Worksheet 9: Data Analysis and Interpretation of PIP Results what did we learn? Likelihood of Significant and Sustained Improvement will it have an ongoing impact? What does the data tell us, and Did we make a difference, and Behavioral Health Concepts, Inc.

Step 1: Identify the PIP Topic What is the Problem? STEP 1: IDENTIFY THE PIP TOPIC What is the problem? A PIP is intended to target improvement in either a clinical service or non-clinical process with the goal of directly impacting beneficiary health and/or functional status. The PIP topic ought to reflect high-volume or high-risk conditions of the population served. Behavioral Health Concepts, Inc.

Worksheet 1: Identify the PIP Topic What is the Problem? 1.1 What is the goal/problem this PIP proposes to solve? How does it affect beneficiary health, functional status, or satisfaction with care? Click or tap here to enter text. 1.2 Who was involved in identifying the problem? (Roles, such as providers or enrollees are sufficient; proper names are not needed). How were beneficiaries or the stakeholders who are affected by/concerned with the issue included? Click or tap here to enter text. 1.3 What MHP/DMC-ODS data have been reviewed that suggest the issue is a problem? Provide the data. Click or tap here to enter text. 1.4 Are there state or national standards or benchmarks related to the problem? If so, what are they? How does the MHP/DMC-ODS s data/performance compare? Click or tap here to enter text. 1.5 What are the provisional or potential root causes of the problem as suggested by quantitative information that the MHP/DMC-ODS chose to address and why? Click or tap here to enter text. 1.6 Briefly state the intervention(s) selected to address the root causes. Click or tap here to enter text. Behavioral Health Concepts, Inc.

What is the reviewer looking for in Step 1? Question Yes No N/A Comments 1.1 Was the PIP topic selected through a comprehensive analysis of beneficiary needs, care, and services? Did selection of the PIP topic consider performance on the CMS Child and Adult Core Set measures? 1.2 1.3 Did the selection of the PIP topic consider input from beneficiaries or providers who are users of, or concerned with, specific service areas? 1.4 Did the PIP topic address care of special populations or high priority services 1.5 Did the PIP topic align with priority areas identified by HHS and/or CMS? 1.6 Overall assessment: In the comments section, note any recommendations for improving the PIP topic. TOTAL of 5 items Behavioral Health Concepts, Inc.

Step 2: Develop the Aim Statement What do we want to do? STEP 2: DEVELOP THE AIM STATEMENT Whatdo we want to do? The PIP aim statement identifies the focus of the PIP (i.e., what the PIP is supposed to accomplish), a goal, and establishes the framework for data collection and analysis. It should provide the improvement strategy, population, and time-period of the study; be clear and concise; and include interventions with a measurable impact. The statement is measurable when it specifies variables and outcomes for a defined improvement strategy, population, and time-period. Examples of Aim Statements Potential sources of information to help form the PIP aim statement include: For ( ( ( by ( ), will ) improve ) as measured ) over ( State data relevant to the topic being studied MHP/DMC-ODS data relevant to the topic being studied CMS Child and Adult Core Sets Enrollee focus groups or surveys Clinical literature on recommended care and external benchmarks. )? Over the next ( ), the ( ) will receive ( to improve ( measured by ( ) ) as ). Behavioral Health Concepts, Inc.

Worksheet 2: Develop the Aim Statement What do we want to do? 2.1 What is the aim/goal of this PIP? The statement should define succinctly: the improvement strategy, population, and time-period of the study. (The statement should be clear and concise; the impact of interventions should be measurable.) Click or tap here to enter text. Behavioral Health Concepts, Inc.

What is the reviewer looking for when validating PIP Aim Statement? Question Yes No N/A Comments 2.1 Did the aim statement clearly specify the improvement strategy, population, and time period for the PIP? Was the PIP aim statement concise? 2.2 2.3 Was the PIP aim statement answerable? Was the PIP aim statement measurable? Overall assessment: In the comments section, note any recommendations for improving the PIP aim statement. TOTAL of 4 items 2.4 2.5 Behavioral Health Concepts, Inc.

Step 2: Develop the Aim Statement What do we want to do? Include: The focus of the PIP The plan for improvement Description of who is being helped Goal Time frame Behavioral Health Concepts, Inc.

STEP 3: IDENTIFY THE PIP STUDY POPULATION Who do we intend to help? In this step, the MHP/DMC-ODS provides detail on the population identified in the aim statement. This step provides information on the relevant beneficiary population, such as age, length of enrollment, frequency of service use, frequency of the problem/issue, type of treatment, diagnoses, and/or other characteristics. Behavioral Health Concepts, Inc.

WORKSHEET 3: PIP STUDY POPULATION Who do we intend to help? 3.1 Describe the beneficiary or enrollee population affected by the problem. Provide information such as age, length of enrollment, diagnosis, and other relevant characteristics. Click or tap here to enter text. 3.2 Will all affected beneficiaries/enrollees receive the intervention(s) and be included in the PIP study population? Yes No 3.3 If no,who would be included? (May be a representative sample, a pilot location, or some other subset of the affected population that will serve as an initial pilot).Click or tap here to enter text. Behavioral Health Concepts, Inc.

What is the reviewer looking for when validating the PIP population? Question Yes No N/A Comments 3.1 Was the project population clearly defined in terms of the identified PIP question (e.g., age, length of the PIP population s participation, diagnoses, procedures, other characteristics) Was the entire MHP/DMC-ODS population included in the PIP? If the entire population was included in the PIP, did the data collection approach capture all beneficiaries to whom the PIP question applied? Was a sample used? (If yes, use Worksheet 1.4 to review sampling methods) Overall assessment: In the comments section, note any recommendations for identifying the project population TOTAL of 4 items 3.2 3.3 3.4 3.5 Behavioral Health Concepts, Inc.

Questions Behavioral Health Concepts, Inc.

STEP 4: DESCRIBE THE SAMPLING PLAN How do we select a smaller group to study? If the entire population of beneficiaries is being included in the PIP, there is no need to describe the sampling method. If a portion of the population will be receiving the intervention or benefit, a sampling approach is in order. A sampling frame is a representative subset of the population based on certain criteria. It includes the universe of members of the target population, such as individuals, caregivers, households, encounters, providers, or other population units that are eligible to be included in the PIP. For example, the sampling frame could be youth with depression or adult beneficiaries who engage in three or more services after assessment but end service before three months. The completeness, recency, and accuracy of the sampling frame are key to the representativeness of the sample. Behavioral Health Concepts, Inc.

STEP 5: SELECT THE PIP VARIABLES AND PERFORMANCE MEASURES How will we know if what we re doing makes a difference? Variables are measurable characteristics, qualities, traits, or behaviors of an individual or process being studied; they can take a variety of forms and will be specific to the issue(s) addressed by the PIP. When choosing variables, select ones that are best suited to the available data, resources, and intended outcomes. Clearly defined, objectively and reliably measured variables lead to higher confidence in results. Performance measures monitor the performance of the MHP/DMC-ODS at a point in time, track performance over time, and inform the evaluation of impact of the interventions used to quantify the outcomes. The performance measure should be consistent with the MHP/DMC-ODS s desired goal for the PIP. When selecting performance measures, the MHP/DMC-ODS should first consider established measures. Behavioral Health Concepts, Inc.

WORKSHEET 5: PIP VARIABLES AND PERFORMANCE MEASURES How will we know if what we re doing makes a difference? 5.1 What are the variables used to track the intervention(s)? Click or tap here to enter text. 5.2 What are the performance measures used to track the outcomes? Please describe how the performance measures assess an important aspect of care that will make a difference to beneficiary health or functional status. Click or tap here to enter text. TABLE 5.1 VARIABLES AND PERFORMANCE MEASURES Variables (Indicators) Performance Measures (Outcomes) Target Improvement Rate Goal Interventions 1. Motivation Enhancement and Engagement Therapy (MEET) intervention during discharge planning 2. Warm hand-offs from discharging program to admitting program Example 1: Increase timely connection rates for clients discharged with referral to a lower level of care (LOC) 1. Documented MEET intervention sessions 2. Billed case management or TCS sessions for warm hand- off visits Admissions to a lower LOC within 10 calendar days of discharge with referral Increase in five percentage points (to 32 percent) over 2019/20 baseline of 28 percent Example 2: Decrease anxiety and improve daily functioning among teens diagnosed with anxiety Mindfulness and other DBT coping skills # of DBT group sessions attended by adolescents GAD-7 and self-report Reduce average score by 25% (from 13 to below 10) Behavioral Health Concepts, Inc.

What is the reviewer looking for when validating the PIP variables and PIP performance measures? Question Yes No N/ Comments A PIP Variables 5.1 Were the variables adequate to answer the PIP question? Objective, clearly defined, time-specific Available to measure performance and track improvement over time Performance measures 5.2 Did the performance measure assess an important aspect of care that will make a difference to beneficiaries health or functional status? (list assessed health or functional status) 5.3 Were the performance measures appropriate based on the availability of data and resources to collect the data (administrative data, medical records, or other sources)? Behavioral Health Concepts, Inc.

What is the reviewer looking for when validating 5.4 the PIP variables and PIP performance measures? Were the measures based on current clinical knowledge or health services research? (Examples may include: hospital admissions, emergency department visits, adverse incidents, appropriate medication use) Did the performance measures: Monitor the performance of MHP/DMC-ODSs at a point in time? Track MHP/DMC-ODS performance over time? Compare performance among MHP/DMC-ODSs over time? Inform the selection and evaluation of quality improvement activities? Did the MHP/DMC-ODS consider existing state or national quality measures? If there were gaps in existing measures, did the MHP/DMC- ODS consider the following when developing new measures based on current clinical practice guidelines or health services research? Accepted relevant clinical guidelines Important aspect of care or operations that was meaningful to beneficiaries Available data sources that allow the MHP/DMC-ODS to reliably and accurately calculate the measure Clearly defined performance measure criteria 5.5 5.6 5.7 Behavioral Health Concepts, Inc.

What is the reviewer looking for when validating the PIP variables and PIP performance measures? 5.8 Did the measures capture changes in enrollee satisfaction or experience of care? (Note that improvement in satisfaction should not be the only measured outcome of a clinical project. Some improvement in health or functional status should also be addressed. For non-clinical PIPs, measurement of health or functional status is preferred Did the measures include a strategy to ensure inter-rater reliability (if applicable)? If process measures were used, is there strong clinical evidence (based on published guidelines) indicating that the process being measured is meaningfully associated with outcomes? Overall assessment: In the comments section, note any recommendations for improving the selected PIP variables and performance measures. TOTAL of 10 items 5.9 5.10 5.11 Behavioral Health Concepts, Inc.

Step 6: Improvement Strategy & Implementation Plan What will we do to cause change? This step describes the improvement strategy (also referred to as an intervention) and how it will be carried out. Selected PIP strategies should be evidence-based or best practices for which there is documented evidence of promise; that is, there should be existing evidence (published or unpublished) suggesting that the intervention would likely lead to the desired improvement in processes or outcomes (as measured by the variables and performance measures). Behavioral Health Concepts, Inc.

Step 6: Improvement Strategy & Implementation Plan What will we do to cause change? Describe the improvement strategy/strategies Translation: Intervention Why were they selected? Evidence-based? Address cultural and linguistic needs? How will they be implemented? Who will apply the interventions? How does the MHP/DMC-ODS verify the interventions are applied as intended? Behavioral Health Concepts, Inc.

WORKSHEET 6: IMPROVEMENT STRATEGY (INTERVENTION) AND IMPLEMENTATION PLAN What, specifically, will we do to cause the change? 6.1 Describe the improvement strategy/intervention. (Distinguish between the intervention(s) and the training and administrative supports required prior to implementation). Click or tap here to enter text. Include pre-intervention process description, if relevant. Click or tap here to enter text. Describe when and how often the intervention will be applied. Click or tap here to enter text. 6.2 What was the quantitative or qualitative evidence (published or unpublished) suggesting that the intervention(s) would address the identified causes/barriers and thereby lead to improvements in processes or outcomes? Click or tap here to enter text. 6.3 Does the improvement strategy specifically address cultural and linguistic needs for the population/beneficiaries? If so, in what way? Click or tap here to enter text. 6.4 Who is involved in applying the intervention? What are their qualifications? Click or tap here to enter text. 6.5 How is the MHP/DMC-ODS ensuring consistency and/or fidelity during implementation of the intervention? Click or tap here to enter text. Behavioral Health Concepts, Inc.

WORKSHEET 6: IMPROVEMENT STRATEGY (INTERVENTION) AND IMPLEMENTATION PLAN What, specifically, will we do to cause the change? Date Intervention Began Example 1: 1a: Addition of two navigators to engage with all admitted clients to adult residential treatment 1b: Addition of barriers on treatment plan and types of barriers Corresponding Variable (Indicator) # Intervention Frequency of Intervention 1a: Minimum weekly individual contact and optimal three times per week before discharge and week after discharge until two face to face appointments in lower level of care 1b: Documented in medical record and summarized by supervisor for QI review. # and type 1a: Navigator visits and type of visits 1b: # of treatment plans with barriers and report with types of barriers weekly and quarterly 1a: 4/1/2021 1b: 4/1/2021 # of DBT group sessions attended by adolescents Example 2: Mindfulness and other DBT coping skills 4/1/2021 Weekly groups 1 2 3 4 Behavioral Health Concepts, Inc.

What is the reviewer looking for when validating the PIP improvement strategy and implementation plan? Question Yes No N/A Comments 6.1 Was the selected improvement strategy evidence-based, suggesting that the test of change (performance measure) would likely to lead to the desired improvement in processes or outcomes (as measured by the PIP variables)? Was the strategy designed to address root causes or barriers identified through data analysis and quality improvement processes? (It is expected that interventions should be measurable on an ongoing basis, e.g., quarterly, monthly, to monitor intervention progress) Was the rapid-cycle PDSA approach used to test the selected improvement strategy? (If tests of change were not successful, i.e., did not achieve significant improvement, a process to identify possible causes and implement solutions should be identified) Was the strategy culturally and linguistically appropriate? 6.2 6.3 6.4 Behavioral Health Concepts, Inc.

What is the reviewer looking for when validating the PIP improvement strategy and implementation plan? 6.5 Was the implementation of the strategy designed to account or adjust for any major confounding variables that could have an obvious impact on PIP outcomes (e.g., patient risk factors, Medicaid program changes, provider education, clinic policies or practices)? 6.6 Did the PIP assess the extent to which the improvement strategy was successful and identify potential follow- up activities? 6.7 Overall assessment: In the comments section, note any recommendations for improving the implementation strategies. TOTAL of 6 items Behavioral Health Concepts, Inc.

Questions Behavioral Health Concepts, Inc.

STEP 7: DESCRIBE THE DATA COLLECTION PROCEDURES What data do we need, and how will we get it? In this step, the MHP/DMC-ODS identifies the data to be collected, including data collection procedures, in order to ensure the validity and reliability of the data used for the PIP. Validity means that the data are measuring what is intended to be measured. Reliability means that the data are producing consistent results. To ensure validity and reliability of the data collected as part of the PIP, the data collection plan should specify: The data sources for the PIP (where the data are located (e.g., the EHR)) The data to be collected Who will collect the data How and when the data are to be collected Frequency of data collection Who (which staff) will enter the data Instruments used to collect the data for analysis Behavioral Health Concepts, Inc.

WORKSHEET 7: DATA COLLECTION PROCEDURES What data do we need, and how will we get it? 7.1 Describe the (planned) methods for ensuring the collection of valid and reliable data. Include MHP/DMC-ODS data entry and collection processes. Click or tap here to enter text. 7.2 What data elements are being collected? Click or tap here to enter text. 7.3 Who is collecting the data? How are they qualified for this task? How will you ensure that all staff collecting data do so in accordance with the plan? Click or tap here to enter text. 7.4 What data collection instruments and electronic data collection/analytic systems are being used (i.e., tools with which raw, original data are collected and/or downloaded for analysis)? Please note if the MHP/DMC-ODS has created any instruments for this PIP. Click or tap here to enter text. Behavioral Health Concepts, Inc.

WORKSHEET 7: DATA COLLECTION PROCEDURES What data do we need, and how will we get it? Frequency of Collection # Variable or PM Data Source Example 1: 1a: Navigator contacts 1b: Treatment Plan Barriers 1a: EHR & Billing logs 1b: EHR 1a: Monthly 1b: Weekly Example 2: # of DBT group sessions attended by adolescents EHR Monthly 1 2 3 4 5 6 Behavioral Health Concepts, Inc.

What is the reviewer looking for when validating the data collection procedures? Question Yes No N/A Comments Assessment of Overall Data Collection Procedures 7.1 Did the PIP design specify a systematic method for collecting valid and reliable data that represents the population in the PIP? 7.2 Did the PIP design specify the frequency of data collection? If yes, what was the frequency (for example, semi-annually)? 7.3 Did the PIP design clearly specify the data sources (e.g., encounter and claims systems, medical records, tracking logs, surveys, provider and/or enrollee interviews) 7.4 Did the PIP design clearly define the data elements to be collected (including numerical definitions and units of measure)? 7.5 Did the data collection plan link to the data analysis plan to ensure that appropriate data would be available for the PIP? 7.6 Did the data collection instruments allow for consistent and accurate data collection over the time periods studied? 7.7 If qualitative data collection methods were used (such as interviews or focus groups), were the methods well-defined and designed to collect meaningful and useful information from respondents? 7.8 Overall assessment: In the comments section, note any recommendations for improving the data collection procedures. Note: Include assessment of data collection procedures for administrative data sources and medical record review noted below. Behavioral Health Concepts, Inc.

What is the reviewer looking for when validating the data collection procedures? Assessment of Overall Data Collection Procedures for Administrative Data Sources 7.9 If inpatient data was used, did the data system capture all inpatient admissions/discharges? 7.10 If ancillary data was used, did ancillary service providers submit encounter or utilization data for all services provided? 7.11 If EHR data was used, were patient, clinical, service, or quality metrics validated for accuracy and completeness as well as comparability across systems? Assessment of Data Collection Procedures for Medical Record Review 7.12 Was a list of data collection personnel and their relevant qualifications provided? 7.13 For medical record review, was inter-rater and intra- rater reliability described? 7.14 For medical record review, were guidelines for obtaining and recording the data developed? TOTAL of 13 items Behavioral Health Concepts, Inc.

STEP 8: DESCRIBE THE DATA ANALYSIS AND INTERPRETATION OF PIP RESULTS What do the data tell us, and what did we learn? In this step, the MHP/DMC-ODS should describe the plan for data analysis and interpretation of PIP results. The data collection plan described in Step 7should link to the data analysis plan. The data analysis plan should be based on a CQI philosophy and reflect an understanding of lessons learned and opportunities for improvement. Interpretation of the PIP results should involve assessing the causes of less- than-optimal project results. The primary source for interpretation of findings should be the results of the analysis. The MHP/DMC-ODS should include basic descriptive analysis, including both baseline and repeated measurements of PIP outcomes. In addition, reasonable benchmarks or comparable data should be included where possible, such as state-level data, data from other counties, or industry benchmarks. The CMS protocol requires the analysis to assess the extent to which any change in performance is statistically significant; however, it does not specify the level of statistical significance that must be met. Behavioral Health Concepts, Inc.

WORKSHEET 8: DATA ANALYSIS AND INTERPRETATION OF PIP RESULTS What do the data tell us, and what did we learn? 8.1 How often were the data analyzed? Plan: Click or tap here to enter text. Actual: Click or tap here to enter text. 8.2 Who conducted the data analysis, and how are they qualified to do so? Plan: Click or tap here to enter text. Actual: Click or tap here to enter text. 8.3 How was change/improvement assessed? Plan: Click or tap here to enter text. Actual: Click or tap here to enter text. 8.4 To what extent was the data collection plan followed were complete and sufficient data available for analysis? Click or tap here to enter text. 8.5 Were any statistical analyses conducted? If so, which ones? Provide target level of significance for each measure. Click or tap here to enter text. 8.6 Were factors considered that could threaten the internal or external validity of the findings examined? Click or tap here to enter text. Behavioral Health Concepts, Inc.

WORKSHEET 8: DATA ANALYSIS AND INTERPRETATION OF PIP RESULTS What do the data tell us, and what did we learn? Performance Performance Measure Measure Description Description Rate Rate rate rate Target Target Performance Performance Baseline Baseline Dates/Year Dates/Year Baseline Baseline sample size and sample size and 1st 1st 1st 1st Most recent Most recent Remeasure Remeasure Dates/Year Dates/Year Most Most recent recent Remeasure Remeasure sample size sample size and rate and rate Demonstrated Demonstrated performance performance improvement improvement (Yes/No) (Yes/No) Statistically Statistically significant significant change in change in performance performance (Yes/No) Specify (Yes/No) Specify P P- -value value Yes No P-value: <.01 <.05 Remeasure Remeasure Dates/Year Dates/Year Remeasure Remeasure sample size sample size and rate and rate Yes No Timeliness of first assessment appt. 90% Average <10 b.days Q1 -Q2 7/20- 12/20 n=505 72% Average 14 b.days Q3-Q4 1/21- 6/21 n=470 84% Average 12 b.days Q1 -Q2 7/21- 12/21 n=517 92% Average 9.7 b.days Other: Yes No PM 1. Click or tap here to enter text. Yes No P-value: <.01 <.05 Other: Yes No PM 2. Click or tap here to enter text. Yes No P-value: <.01 <.05 Other: Yes No PM 3. Click or tap here to enter text. Yes No P-value: <.01 <.05 Behavioral Health Concepts, Inc. Other:

What is the reviewer looking for when validating the data analysis and interpretation of results? Question Yes No N/A Comments 8.1 Was the analysis conducted in accordance with the data analysis plan? 8.2 Did the analysis include baseline and repeat measurements of project outcomes? 8.3 Did the analysis assess the statistical significance of any differences between the initial and repeat measurements? 8.4 Did the analysis account for factors that may influence the comparability of initial and repeat measurements? 8.5 Did the analysis account for factors that may threaten the internal or external validity of the findings? Behavioral Health Concepts, Inc.

What is the reviewer looking for when validating the data analysis and interpretation of results? 8.6 Did the PIP compare the results across multiple entities, such as different patient subgroups, provider sites, or MHP/DMC-ODSs? 8.7 Were PIP results and findings presented in a concise and easily understood manner? 8.8 Did the analysis and interpretation of the PIP data include lessons learned about less-than-optimal performance? Overall assessment: In the comments section, note any recommendations for improving the analysis and interpretation of PIP results. 8.9 TOTAL of 8 items Behavioral Health Concepts, Inc.

STEP 9: ADDRESS THE LIKELIHOOD OF SIGNIFICANT AND SUSTAINED IMPROVEMENT THROUGH THE PIP Did we make a difference, and will it have an ongoing impact? In this step, the MHP/DMC-ODS identifies the likelihood that significant and sustained improvement occurred as a result of the PIP. This includes an assessment of the overall validity and reliability of the PIP methods and findings to determine whether there is confidence in the results. An important component of a PIP is to determine if the reported change is real change or the result of an environmental or unintended consequence or random chance. It is also essential to demonstrate sustained improvement. To do so requires repeated measurements to be conducted over the course of the PIP and demonstration of the degree of change in performance relative to baseline measurement. Behavioral Health Concepts, Inc.

WORKSHEET 9: LIKELIHOOD OF SIGNIFICANT AND SUSTAINED IMPROVEMENT THROUGH THE PIP Did we make a difference, and will it have an ongoing impact? Provisional Findings, if applicable: (For PIPs that are in process at the time of submission, or that do not yet have any remeasurement data, please briefly provide preliminary results or impressions to date) Click or tap here to enter text. 9.1 What is the conclusion of the PIP? Click or tap here to enter text. 9.2 Do changes appear to be the results of the PIP interventions? Please explain. Click or tap here to enter text. 9.3 Does statistical evidence support that the change represents a real improvement or difference? Click or tap here to enter text. 9.4 Did any factors affect the methodology of the study or the validity of the results? If so, what were they? Click or tap here to enter text. 9.5 Was the improvement sustained through repeated measurements over comparable time periods? (If this is a new PIP, what is the plan for monitoring and sustaining improvement?) Click or tap here to enter text. 9.6 How were untoward results addressed? Click or tap here to enter text. 9.7 What is the MHP/DMC-ODS s plan for continuation or follow-up? Click or tap here to enter text. Behavioral Health Concepts, Inc.

What is the reviewer looking for when validating the likelihood of significant and sustained improvement? Question Yes 9.1 Was the same methodology used for baseline and repeat measurements? 9.2 Was there any quantitative evidence of improvement in processes or outcomes of care? 9.3 Was the reported improvement in performance likely to be a result of the selected intervention? 9.4 Is there statistical evidence (e.g., significance tests) that any observed improvement is the result of the intervention? No N/A Comments 9.5 Was sustained improvement demonstrated through repeated measurements over time? 9.6 Overall assessment: In the comments section, note any recommendations for improving the significance and sustainability of improvement as a result of the PIP. TOTAL of 5 items Behavioral Health Concepts, Inc.

Overall Validation and Reporting of PIP Results PIP Validation Information Yes No Was the PIP validated? Validated means that the EQRO reviewed all relevant part of each PIP and made a determination as to its validity. In many cases, this will involve calculating a score for each relevant stage of the PIP and providing feedback and recommendations.) Validation phase (check all that apply): PIP submitted for approval Planning phase Implementation phase Baseline year First remeasurement Second remeasurement Other (specify): Validation rating: High confidence Moderate confidence Low confidence No confidence Validation rating refers to the EQRO s overall confidence that the PIP adhered to acceptable methodology for all phases of design and data collection, conducted accurate data analysis and interpretation of PIP results, and produced significant evidence of improvement. EQRO recommendations for improvement of PIP: Behavioral Health Concepts, Inc.