eCQM Testing: Making the Most of Limited Data
This session covers topics such as eCQM refresher, technical specification, best practices, challenges in testing with limited data, and more. Learn about tools like VSAC, MAT, and Bonnie to ensure proper electronic specification of measures.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
MACRA Info Session #9 CMS Measure Development Education & Outreach eCQM Testing: Making the Most of Limited Data Presenters: Jeff Geppert, Battelle Brenna Rabel, Battelle June 12, 2019 2:00-3:00pm ET
Upcoming Public Webinars Planned Upcoming Webinars: June 25, 2019 and June 27, 2019 Measuring Quality to Improve Quality: Strengths and Challenges of Clinical Quality Measurement July 24, 2019 and July 25, 2019 Patient- Centered Quality Measurement: What It Is and How to Get Involved 1
Vision and Goals: MACRA Info Sessions An ongoing process to engage the public in quality measure development. Elicit feedback that will help CMS design resources that can help all of those interested in healthcare quality improvement better understand the goals of quality measurement. Education Outreach Dedicated Websites Measure Development Roadmaps Listserv opportunities 2
Agenda 1. eCQM Refresher 2. Technical Specification & Testing 3. Best Practices 4. Challenges in eCQM Testing with limited data 5. Specification Testing 6. Scientific Acceptability Testing 7. Questions and Discussion 3
eCQM Refresher Just a reminder that the lower case e in eCQM means electronic! Historically, CQMs were typically paper-based with a time-consuming manual process Electronic specification means creating a machine- readable set of measure files using agreed upon standards Potential advantages include time savings, ease of access to clinical EHR data, consistency and uniformity, and error reduction 4
Technical Specification & Testing These advantages can only be realized if the measure is properly electronically specified. How do we ensure this? Testing! There are tools provided by the measure community to facilitate this process: Specification Value Set Authority Center (VSAC), Measure Authoring Tool (MAT) Logic Testing Bonnie Developer Collaboration JIRA 5
VSAC Value sets are used in the specification of eCQMs to represent coded versions of data elements, the objects we want to talk about in the measure NLM s VSAC is the only authoritative tool to author value sets for eCQMs From a testing perspective, it provides validation tools for creating and revising value sets Using existing VSAC value sets, especially official versions used in CMS programs, increases harmonization and the reliability of your measure 6
MAT 1. eCQM Refresher CMS requires that all eCQMs be created utilizing their MAT, a publicly-available web application that is API-integrated into the VSAC for value set access The MAT has been constructed to produce the electronic files in the agreed-upon standards that represent the measure elements and logic See the References slide for links to more information on how to use the MAT and a thorough explanation of the output files in the measure package the MAT provides for export 7
Bonnie 1. eCQM Refresher The MAT allows measure developers to electronically specify anything they imagine, but given potential logic complexity, it s not guaranteed a measure says what a developer intends Bonnie is a testing tool that enables verification of measure logic behavior by allowing developers to create a synthetic patient test deck for a measure MAT measure packages can be imported into Bonnie Bonnie s interface allows for efficient test case creation and immediate validation of each individual logic section 8
Best Practices 1. eCQM Refresher Conduct specification and testing together, because this process is iterative and testing should be ongoing during specification Export from the MAT and import to Bonnie frequently, adding test cases as you develop the specification 9
Best Practices 1. eCQM Refresher Test cases should include fringe patients to confirm logic down to the finest detail Basic Example: Logic states the patient must be 18 years old at the start of the measurement period (2019) Create a test case with a patient born on 12/31/2000 at 23:59 Create a test case with a patient born on 1/1/2001 at 00:00 10
Best Practices 1. eCQM Refresher Basic Example: Logic states the patient must be 18 years old at the start of the measurement period (2019) Create a test case with a patient born on 12/31/2000 at 23:59 Create a test case with a patient born on 1/1/2001 at 00:00 This also serves to illustrate that you should create both positive and negative test cases Above, we ve created a patient which we expect to fail the measure based on age, and a patient we expect to pass 11
Developer Collaboration ONC hosts an instance of JIRA, a collaboration platform used for information sharing for eCQMs JIRA projects use a ticketing process, where developers submit questions or issues in a ticket We recommend utilizing this site for taking advantage of the collective specification and testing knowledge of the community 12
References CMS MMS Blueprint https://www.cms.gov/Medicare/Quality-Initiatives-Patient- Assessment-Instruments/MMS/Downloads/Blueprint.pdf CMS Measure Authoring Tool https://www.emeasuretool.cms.gov/web/guest/mat-home NLM Value Set Authority Center https://vsac.nlm.nih.gov/ Bonnie https://bonnie.healthit.gov/ ONC Jira Issue Tracking https://oncprojectracking.healthit.gov/support/projects/CQM/sum mary 13
Working with Limited Testing Data
Overview Challenges in eCQM testing with limited data Specification testing Tools Best practices Scientific acceptability testing Reliability Validity Questions and discussion 15
Challenges in eCQM testing with limited data 1. eCQM Refresher eCQM testing requirements are minimal due to challenges in obtaining testing data from multiple sites As a result, eCQM testing tends to focus on feasibility, specification reliability and validity, and data element reliability and validity Performance score reliably and validity is rarely assessed (outside face validity) Generally, data from at least 30 sites is needed for reliability and validity testing of the performance score The absence of performance score testing makes it difficult to determine whether eCQM implementation will be impactful 16
Challenges in eCQM testing with limited data 1. eCQM Refresher CMS Blueprint for the CMS Measures Management System [1] 24.2.5 Testing Multiple Sites and Multiple EHRs Testing multiple sites for feasibility, validity, and reliability is important to address potential variability in reporting based on differences in local workflow process. Even multiple sites using the same EHR vendor product may show different results since the local workflow may vary and data may not be consistently entered into the fields expected by the vendor. Variances in results from such testing at multiple sites should be evaluated to determine whether changes are needed in the measure logic or definition. Testing must encompass at least two EHR products. 17
Specification Testing 1. eCQM Refresher In the context of limited data, rigorous specification testing is even more important Tools Best practices 18
Scientific Acceptability Testing Reliability Validity 1. eCQM Refresher Evaluate the measure for clarity, logic ambiguity, and data element alignment with standard specifications that support consistent implementations Specification Evaluate measure logic to confirm whether the correct patients are included in or excluded from the numerator, denominator, and other relevant populations Data element Evaluate reproducibility of data acquisition or abstraction based on interrater-agreement (kappa statistic) Evaluate agreement of acquired or abstracted data with an authoritative source based on sensitivity, specificity, and negative and positive predictive value Instrument Evaluate stability of measures (test- retest), internal consistency of measurement instruments, and interrater reliability of instrument scores Evaluate agreement of instrument score with an abstract concept (theoretical construct) Performance score Signal-to-noise Hypothesis testing 19
Performance Score Reliability Definition of reliability: a metric is reliable inasmuch as it constantly provides the same result when applied to the same phenomena. [2] performance score - the phenomena are the quality construct. Definition of quality construct: The degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge. [3] Reliability is about the process of measurement (the how, not the what). Deficiencies in reliability require modifications to the process of measurement. 20
Performance Score Reliability Reliability is evaluated at the measured entity level (not the measure level). Measure level reliability metrics are aggregates of measured entity level reliability metrics (mean, median, or distribution). (between / between + within) is not the definition of the reliability, but rather an assumption about or evidence for the reliability of an individual measured entity. Evaluating reliability is associated with usability: is reliability sufficient to purpose, intended use, or functionality? 21
Performance Score Reliability Reliability testing of the performance score with limited data A simple reliability evaluation might be the percentage of measured entities that are below a threshold or above a benchmark at some probability. E.g., 50% of measured entities have an 30% probability of generating an outcome above the benchmark E.g. the median measured entity has an 20% probability of generating an outcome below the threshold We then show how each proposed measure would demonstrate reliability to that purpose. That would offer an apples-to-apples comparison of the various methods of estimating reliability. 22
Performance Score Reliability Reliability testing of the performance score with limited data Rule of Five: There is a 93.75% chance that the median of a population is between the smallest and largest values in any random sample of five from the population Estimate the signal variance for each measured entity Identify patient pathways Use process evidence from the literature or data Select a performance benchmark and/or threshold Based on the signal variance for the smallest and largest values, calculate the probability that the measured entity is above or below the benchmark or threshold 23
Performance Score Reliability Reliability testing of the performance score with limited data Example: Among identical patients, variation in process of care pathways yields variation in outcome, which is what we mean by the signal variance of an individual measured entity. 24
Performance Score Validity Validity is an overall evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy of appropriate interpretations and actions on the basis of [the measure] [4]. Performance Score Validity A finding of an empirical association between: the performance score and the quality construct the quality construct and the outcome of interest the performance score and the outcome of interest 25
Performance Score Validity Validity relates to the entitled inferences and interpretations for the use of the measure Validity is not a characteristic of the measure but, like reliability, depends on the purpose, intended use, or function Quality improvement, comparative reporting, value-based purchasing Validity is a unitary concept and an evaluative judgment Sources of evidence do not represent different types of validity, but rather different aspects of validity Validity is different than validation, which refers to the process for obtaining evidence with which to support measure use 26
Performance Score Validity Validity testing of the performance score with limited data Example: Conceptual model Interpretation and comprehension, information retrieval, judgement, and reporting (communicating) Direct methods Scenarios, focus groups, and interviews Indirect methods Time complexity of operations involved and the time to complete Observation 27
Performance Score Validity A process is a series of steps that transform inputs to outputs [5]; Inputs to a process include materials, methods, information, people, equipment, the work environment; Outputs of a process are products and services. 28
Conclusions and Takeaways eCQMs should undergo specification testing (ie., in Bonnie) and full feasibility and scientific acceptability testing including performance score reliability and validity Scientific acceptability testing can be challenging given limited data availability from test sites, but there are alternative methods or sources of evidence available Reliability: Signal variance Validity: Observational methods 29
References [1] Blueprint for the CMS Measures Management System (Version 4.1; February 2019). https://www.cms.gov/Medicare/Quality-Initiatives-Patient-Assessment- Instruments/MMS/Downloads/Blueprint.pdf [2] EAGLES (Expert advisory group for language engineering standards). 1995b. The EAGLES extension to ISO 9126. http://www.issco.unige.ch/ewg95/node15.html [3] Medicine, Institute of. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academies Press, 2001. [4] Messick S. Validity. In Linn RL (Ed.): Educational measurement. New York: Macmillan; 1989: 13-103. [5] Office of Research Services, National Institutes of Health. Process Mapping. https://www.ors.od.nih.gov/OD/OQM/training/Pages/training.aspx. [6] AERA. Standards for educational and psychological testing. AERA, 1999. 30
Upcoming MACRA Info Sessions Planned Upcoming Webinars: July 17, 2019 CMS National Impact Assessment Overview August 2019 TBD To suggest topics for upcoming Info Sessions, email: MMSsupport@battelle.org 32
Upcoming Public Webinars Planned Upcoming Webinars: June 25, 2019 and June 27, 2019 Measuring Quality to Improve Quality: Strengths and Challenges of Clinical Quality Measurement July 24, 2019 and July 25, 2019 Patient- Centered Quality Measurement: What It Is and How to Get Involved 33
Contact Information Battelle Measures Management System Contract Holder Contact: MMSsupport@Battelle.org CMS Kimberly Rawlings: Kimberly.Rawlings@cms.hhs.gov 34
