Challenges in Translation of High Acuity Pediatric Quality Indicators
This presentation delves into the challenges faced in translating knowledge into performance measurement for high acuity pediatric conditions. Topics include quality of care, quality indicators, and the use of measures to improve healthcare outcomes.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
The Development of Quality Indicators for High Acuity Pediatric Conditions: Challenges in the Translation of Knowledge into Performance Measurement Antonia Stang MDCM MBA MSc Assistant Professor University of Calgary Departments of Pediatrics and Community Health Sciences
Disclosure I do not have an affiliation (financial or otherwise) with any commercial organization that may have a direct or indirect connection to the content of my presentation.
If we want healthy citizens-as opposed to citizens who have ready access to sickness care-we need a profound philosophical shift in what we should expect from medical professionals. We need to reward and incent quality, not quantity Andre Picard, Globe and Mail, March 20, 2012
Background Quality of Care: the degree to which health services for individuals increases the likelihood of desired health outcomes and are consistent with current professional knowledge (Institute of Medicine)
Background Quality Indicators: explicitly defined and measurable items pertaining to the structures, processes or outcomes of care Structures: staff, equipment, physical layout of the department, laboratory and diagnostic imaging resources Processes: interactions between professionals and patients Outcomes: mortality, morbidity, patient satisfaction, quality of life
Quality Indicator Uses Improve health care and outcomes Benchmark performance Set minimum standards of care Improve efficiency Accountability Transparency Research Pay-for-Performance
What Makes a Good Measure? Impact, Opportunity, Evidence Important to Measure and Report Reliability and Validity Scientific Acceptability of Measure Properties Usability Feasibility National Quality Forum Measure Evaluation Criteria http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx
Objectives To review methods for involving stakeholders in the indicator development process. To discuss the application of GRADE (the Grading of Recommendations Assessment, Development and Evaluation) in indicator development and selection. To describe the challenges in developing and testing broadly applicable performance measures for high impact, relatively low frequency, conditions.
Study Objective to use a systematic process involving multiple stakeholders to develop evidence based quality of care indicators for pediatric conditions requiring high acuity ED care.
Research Team Antonia Stang MD MBA MSC, Principal Investigator Astrid Guttmann MD MSc, Co-Investigator David Johnson MD, Co-Investigator Sharon Straus MD MSc, Co-Investigator Lisa Hartling MSc PhD, Collaborator/methodological expert Francois Belanger MD, Collaborator/decision maker Angelo Mikrogianakis MD, Collaborator/decision maker Jen Crotts RN, Research Assistant Janie Williamson RN, Pediatric Emergency Research Team Coordinator
Funding Funded by an operating grant from the Canadian Institutes of Health Research (CIHR)
Rationale Lack of research on quality indicators specific to the pediatric population. Quality measures that are part of pediatric emergency medicine practice have not been systematically validated. Performance measures specific to pediatrics and pediatric emergency medicine have been identified as a research priority.
Phase 1: Condition Selection 6 conditions selected for indicator development Phase 2: Systematic Review 47 existing indicators 51 guideline and evidence based Phase 3: Expert Panel Process 17 new indicators 114 considered 62 selected Phase 4: Data Collection
Stakeholder Involvement Goals: to represent different stakeholder perspectives to incorporate scientific evidence and expert opinion
Stakeholder Involvement Systematic methods to combine expert opinion and medical evidence Consensus development conferences Guideline based Delphi technique Nominal group technique RAND/UCLA appropriateness method
Phase 1 Condition Selection 32 Member advisory panel Data on the main diagnosis for high acuity (resuscitation and emergent at triage) pediatric patients (age 0-19 yrs) seen in all EDs in Ontario and Alberta. Criteria for Condition Selection; importance (morbidity or mortality) impact (potential to address gap between current and best practice) validity (adequacy of scientific evidence linking performance of care to patient outcome)
Phase 1: Condition Selection Table 1: High Acuity (Resuscitation and Emergent ) Pediatric visits in 2006/2007 and 2007/2008 for all EDs in Ontario and Alberta for Selected Conditions Condition Number of ED Visits 1138 489 1334 439 941 240 Diabetic ketoacidosis Status asthmaticus Anaphylaxis Status epilepticus Severe head injury Sepsis/septic shock
Phase 2: Indicator Development Systematic Review of the Literature for each condition Existing Indicators High quality national and international guidelines (AGREE), Systematic Reviews (AMSTAR), Randomized Controlled Trials Criteria for Indicator Development High quality evidence linking care structure or process to patient outcome Strongly recommended Consistency across guidelines
Literature Review Condition Search Results Full Text Reviewed Articles 3 8 4 3 5 3 Guidelines 3 5 3 5 5 2 Diabetic ketoacidosis Status asthmaticus Anaphylaxis Status epilepticus Severe head injury Sepsis/septic shock 2480 4564 5889 870 4789 3866 374 172 276 28 95 78
GRADE Grading of Recommendations Assessment, Development and Evaluation 1-Very low quality: Any estimate of effect is very uncertain 2-Low quality: Further research is very likely to have an important impact on our confidence in the estimate of effect and is likely to change the estimate 3-Moderate Quality: Further research is likely to have an important impact on our confidence in the estimate of effect and may change the estimate 4-High Quality: Further research is very unlikely to change our confidence in the estimate of effect
GRADE Rating by Condition 18 Number of Indicators 16 14 12 GRADE 10 8 1 6 4 2 2 0 3 4 Condition
GRADE Inter-rater Reliability Condition Kappa* 0.74 0.73 1.00 0.15 0.66 0.55 1.00 0.68 Agreement 0.83 0.81 1.00 0.50 0.85 0.73 1.00 0.78 Diabetic Ketoacidosis Status Asthmaticus Anaphylaxis Status Epilepticus Severe Head Injury Severe Sepsis/Septic Shock General Measures Overall *Cohen s Kappa unweighted
Challenges using GRADE Subjective Lack of concordance between guidelines Time consuming Requires substantial knowledge of clinical condition and research methodology Variable inter-rater reliability
Phase 3: Indicator Selection Expert Panel Process: 2 rounds of a web-based survey and a face-to-face meeting Indicators were selected based on two criteria rated on a scale of 1(strongly agree) to 9 (strongly disagree): Relevance Impact Indicators rated 7 on both criteria by 70% of panelists were retained
Indicator Type Source GRADE Numerator Denominator % of patients with anaphylaxis with documentation of specialist referral P 25, 26, 28-31 2 Number or patients with anaphylaxis with documentation of specialist referral including primary MD follow-up for referral, or documentation of existing specialist relationship Total number of patients with anaphylaxis (based on ICD- 10 codes) Relevance Disagree strongly Disagree Disagree moderately Disagree somewhat Neutral Agree somewhat Agree moderately Agree Agree strongly 1 2 3 4 5 6 7 8 9 Impact Disagree strongly Disagree Disagree moderately Disagree somewhat Neutral Agree somewhat Agree moderately Agree Agree strongly 1 2 3 4 5 6 7 8 9
Type of Indicator 60 50 40 30 20 10 0 Structure Process Outcome
Challenges in Indicator Development Lack of high quality evidence on the link between treatment/processes and outcomes, particularly in the pediatric setting Difficulty in identifying performance measures applicable to all settings
"The only man I know who behaves sensibly is my tailor; he takes my measurements anew each time he sees me. The rest go on with their old measurements and expect me to fit them." George Bernard Shaw
Phase 4:Data Collection Condition Number of ED Visits* 112 852 269 133 108 79 Diabetic ketoacidosis Status asthmaticus/severe asthma Anaphylaxis Status epilepticus Severe head injury Sepsis/septic shock *Based on ICD-10 code
Diabetic Ketoacidosis Indicator Results GRADE 1 Kappa N/A (single site) % of Emergency Departments (EDs) with guidelines % of ED visits with: Intravenous (IV) fluids within 60 minutes of ED arrival* Isotonic solution as initial IV fluid* IV insulin given Appropriate initial insulin dose and route* Potassium replacement Bicarbonate given Time (median minutes with interquartile range) from: Triage to initiation of IV fluids* Arrival to insulin Arrival to expert consultation 36 (17/47) 72 (34/47) 88 (57/65) 88 (52/59) 91 (59/65) 0 (0/61) 1.00 1.00 1.00 1.00 1.00 N/C 1 2 4 4 4 3 78 (45,114) 115 (60,148) 161 (130,201) 1 1 1 *Includes only visits with fluids or insulin started in study ED
Anaphylaxis Indicator % of EDs with: Results GRADE Kappa Clinical guidelines for the treatment of anaphylaxis % of patient visits with: Epinephrine given in ED (or in 3 hours prior to ED visit) Epinephrine given in ED by the appropriate route Documentation of epinephrine auto-injector at discharge Documentation of discharge instructions to avoid offending allergen* Documentation of instruction for epinephrine self-administration Documentation of specialist referral N/A (single site) 2 68 (144/211) 94 (77/82) 85 (180/211) 17 (29/173) 38 (81/211) 56 (119/211) 3 3 2 2 2 2 0.89 0.68 0.02 0.41 0.39 1.00 * for food and insect sting induced reactions
Feasibility and Reliability Condition ED Visits* Missing Chart/Visit 112 269 133 Meets Criteria Kappa 65/78 (83%) 211/250 (84%) 85/127 (67%) Diabetic ketoacidosis Anaphylaxis Status epilepticus 34 19 6 1.00 1.00 0.67 *Number of ED visits based on ICD-10 code
Challenges in Indicator Testing Feasibility and Reliability Retrospective Proper diagnosis is in itself a quality issue Accuracy of ICD -10 codes Cost/effort of data collection Small and variable sample size
Lessons Learned Allow ample time for systematic review and evidence grading Composition of expert panel is key Need an experienced moderator Formal qualitative analysis of expert panel meeting Include patient/care-giver perspective
Next Step Multicentre data collection on select high acuity indicators combined with existing pediatric and emergency department performance measures Reliability and feasibility testing Process to outcome link
Quality Improvement and Indicator Development Institute of Medicine. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academy Press. 2001: Washington, DC. Campbell, S.M., et al., Research methods used in developing and applying quality indicators in primary care. BMJ, 2003. 326(7393): p. 816-9. Chassin MR et al. Accountability Measures-Using Measurement to Promote Quality Improvement. N Eng J Med 363:7. Donabedian A. The quality of care. How can it be assessed? JAMA. Sep 23-30 1988;260(12):1743-1748. http://www.qualityforum.org/docs/measure_evaluation_criteria.aspx Center for Health Policy/Center for Primary Care and Outcomes Research & Battelle Memorial Institute. Quality Indicator Measure Development, Implementation, Maintenance, and Retirement (Prepared by Battelle, under Contract No. 290-04-0020). Rockville, MD: Agency for Healthcare Research and Quality. May 2011.
Pediatric and Emergency Indicators Alessandrini E, Gorelick MH, Shaw K, Kennebeck S. Using Performance Measures to Drive Improvement in Pediatric Emergency Care 2010; http://webcast.hrsa.gov/postevents/archivedWebcastDetail.asp?aeid =534 Bardach NS, Chien AT, Dudley A. Small Numbers Limit the Use of the Inpatient Pediatric Quality Indicators for Hospital Comparison. Academic Pediatrics, 2010. 10(4). Guttmann A, Razzaq A, Lindsay P, Zagorski B, Anderson GM. Development of measures of the quality of emergency department care for children using a structured panel process. Pediatrics, 2006. 118(1): p. 114-23. Schull MJ, Guttmann A, Leaver CA, Vermeulen M, Hatcher CM, Rowe BH, Zwarenstein M, Anderson GM. Prioritizing performance measurement for emergency department care: consensus on evidence based quality of care indicators. CJEM 2011. 13(3):300- 309.
Rating the Quality of Evidence http://www.gradeworkinggroup.org/index.htm (accessed February 23, 2012) Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Sch nemann HJ; GRADE Working Group. Rating quality of evidence and strength of recommendations: What is "quality of evidence" and why is it important to clinicians? BMJ. 2008 May 3;336(7651):995-8. Guyatt GH, Oxman AD, Vist G, Kunz R, Falck-Ytter Y, Alonso-Coello P, Sch nemann HJ, for the GRADE Working Group. Rating quality of evidence and strength of recommendations GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008;336:924-926. Sch nemann HJ, Oxman AD, Brozek J, Glasziou P, Jaeschke R, Vist GE, Williams JW Jr, Kunz R, Craig J, Montori VM, Bossuyt P, Guyatt GH; GRADE Working Group. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ. 2008 May 17;336(7653):1106-10. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009AGREE Next Steps Consortium (2009). The AGREE II Instrument [Electronic version]. Retrieved March 23 2012, from http://www.agreetrust.org .
