Clinical Decision Support and Predictive Analytics in Emergency Medicine

Clinical Decision Support and Predictive Analytics 11/4/2018 Brian Patterson MD MPH Medical Director, Clinical Informatics BerbeeWalsh Department of Emergency Medicine

Goals Define CDS Discuss practical aspects CDS design, implementation, and govenance Contrast Machine Learning models for driving predictive analytics with more traditional statistical methods Discuss Principles for adopting CDS based on Predictive Analytic/Machine Learning Models

What is CDS Any software designed to directly aid in clinical decision making in which characteristics of individual patients are matched to a computerized knowledge base for the purpose of generating patient-specific assessments or recommendations that are then presented to clinicians for consideration. Garg AX, Adhikari NK, McDonald H, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. Jama. 2005;293:1223-1238. Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. Jama. 1998;280:1339-1346.

What is CDS Not Just Popups/BPAs Can include Order sets Documentation tools Trackboards

Why Use It? Decrease Practice Variance Improve Quality Shorten time to diffuse new care patterns Because we have to

Sources of CDS EHR vendors Hospitals Increasingly, 3rd party systems Drug-drug interactions Rise of Predictive Analytics/Machine Learning/AI

CDS Terms Modal(Interruptive) vs. Non modal(Passive) vs. On Demand Trigger Conditions Hard Stops

CDS In Emergency Medicine Not a ton of literature A few areas of clear impact Modal Alerts At time of Radiology Ordering Can strengthen guideline compliance for Head CT, CTPE Changing CPOE to improve prescribing patterns with alerts or order sets Geriatrics, Peds Diagnosis/Treatment/Disposition pathways PNA, Asthma, Sepsis

CDS In Emergency Medicine Mostly Positive Results Mostly Before-After Studies Mostly Focused on Process Outcomes Generally Fail to Measure Negative Consequences

CDS Design Principles 5 rights of decision support the right information to the right person (physicians, nurses, residents?) in the right format (mode and delivery format) through the right channel (i.e. not necessarily GUI of an EHR) at the right time in the workflow (Triggers vs. On demand) https://healthit.ahrq.gov/ahrq-funded-projects/current-health-it-priorities/clinical- decision-support-cds/chapter-1-approaching-clinical-decision/section-2-overview- cds-five-rights

10 Commandments for Effective CDS 1. Speed Is Everything. 2. Anticipate Needs and Deliver in Real Time. 3. Fit into the User's Workflow. 4. Little Things Can Make a Big Difference. (Do usability testing) 5. Recognize that Physicians Will Strongly Resist Stopping. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(6):523-30.

10 Commandments for Effective CDS 6. Changing Direction Is Easier than Stopping. 7. Simple Interventions Work Best. 8. Ask for Additional Information Only When You Really Need It. 9. Monitor Impact, Get Feedback, and Respond. 10. Manage and Maintain Your Knowledge-based Systems. Bates DW, Kuperman GJ, Wang S, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc. 2003;10(6):523-30.

CDS Design Bad CDS is often worse than none at all

CDS Governance Ensure all stakeholders in CDS are represented before implementation Avoids cardiology wants a popup to stop ED docs from doing x scenario Allows leveraging of institutional experience Ensure those proposing CDS have a pre-implementation bar to meet Approved clinical workflow (i.e. does a transfusion threshold of Hb 7 work everywhere?) Firing rate, expected effect Analysis by dry-firing Clinical Workflow effects

CDS Governance Ensure total burden of CDS is managed/coordinated Ensure post-implementation monitoring/evaluation occurs Sundown old or ineffective CDS

Cool New Stuff Predictive Analytics vs. Artificial Intelligence vs. Machine Learning vs. Neural Networks vs. Deep Learning

Predictive Analytics Predictive Analytics: any time we use an algorithm to try to predict future risk based on past data Machine Learning: Allowing a computer algorithm to do more than simply solve a regression equation (often much more). Generally in our context we re looking at using these to risk stratify or classify patients Neural Networks: ML methods wherein we train the computer with a neural net Deep Learning: A specific subset of machine learning algorithms which everyone is particularly excited about

Machine Learning Traditional Statistical Modeling I select variables (or at least a subset for selection) Beyond prediction, we re generating a mental model We can learn something by looking at a regression model vs. Machine Learning Solely focused on prediction Finds a thumbprint signature from a number of variables

Well Known Pitfalls of ML Models Overfitting Minimized by cross validation Minimized by performance in holdout test set Generalizability Needs validation in your population Health Equity Is there bias in training data towards those with more access to care Ex. Facial Recognition Chen JH, Asch SM. Machine Learning and Prediction in Medicine - Beyond the Peak of Inflated Expectations. N Engl J Med. 2017;376(26):2507-2509.

Common Machine Learning Methods Regression: Lasso Regression, Ridge Regression, Elastic Nets Decision Trees Random Forests, Gradient Boosted Trees Neural Networks Bayesian models Support Vector Machines (SVM s)

Model Descriptions Interpretable vs. Black Box Models Human interpretability in computer science Understandable vs. Opaque to clinicians Causal vs. Associative Proprietary vs. Open Source

Potential Impact Predicting impact Sensitivity/Specificity But also PPV/NPV In your data At a minimum, how many times will this fire (per day, per clinician) Ideally create 2x2 tables and consider patients in each box

Guiding Principles for Predictive Analytics at UW Predictive model (purchased or custom built) evaluation includes validation of performance on our local data Model evaluation includes statistical measures and relevant operational metrics (e.g. sensitivity, specificity, PPV + BPAs, interventions) Model output follows the 5 rights of CDS and is associated with interventions whenever possible Model monitoring (pilot or scale-out) includes statistical measures, operational metrics, relevant outcomes, and reevaluation criteria

Thank you