Innovative Educational Evaluation and Scholarship Insights

Evaluation of Educational Innovations as Scholarship Presented by: Andrea Berry and Magda Pasarica Developed by: Medical Education Scholarship Research & Evaluation (MESRE) Janet Riddle, Anna Cianciolo, Andrea Berry, Maria Blanco, Nicole Borges, Christy Boscardin, Kathryn Huggett, Steven Rougas, Sebastian Uijtdehaage, Anne Vo A section of the AAMC Group on Educational Affairs www.aamc.org/members/gea/gea_sections/mesre/

OBJECTIVES Through analysis of the educational innovation case studies provided, workshop participants will be able to: Utilizing a logic model, select program outcomes for their own innovations that will be valued by key stakeholders Describe program processes that are likely to contribute to the selected outcomes Describe a theory-in-action or rationale for how program processes lead to program outcomes Identify features within the context in which the innovation is implemented that are likely to impact or generalize program outcomes

What is an innovation? New process or event Solutions to problems Assessment tools Curricula New application or perspective Kanter SL. Toward better descriptions of innovations. Acad Med 2008 Aug;83(8):703-4.

Evaluation: defined Systematic determination of the merit or worth of an object (educational intervention). (Scriven, 1967) Utilization-focused program evaluation: done for and with specific intended primary users for specific, intended uses. (Patton, 2008) What? So what? Now what? (Patton, 2008) Numerous additional definitions.

Evaluation of Educational Innovations Understanding how and why an innovation works Diagnose and document program strengths and weaknesses Inform the allocation of resources Determine that educational objectives met To determine next steps/make decisions To document success Dissemination of best practices

Going Beyond Did it work?* Planned Did it work? Emergent What else happened? How is it happening? How did it work? *Haji F, Morin M-P, Parker K. Rethinking programme evaluation in health professions education: beyond did it work? Med Educ 2013;47:342-351

Evaluation Theory Logic Model Goal Oriented/Objective-Based (Tyler) Goals-free Evaluation (Scriven) Kirkpatrick s 4-level model CIPP (Stufflebeam)

www.uwex.edu/ces/pande/evaluation/evallogicmodel.html

Outputs Outcomes Impact Short Term Long Term Inputs Activities Participation Residents have positive perceptions of and accept the curriculum Impact on: Medical errors Resident workflow Verbal and written miscommunications I-Pass study group education executive committee 267 I-PASS faculty champions recruited Team-building strategies Educational frameworks and guiding principles Support from Initiative for Innovation in Pediatric Education 855 residents trained Examining how: Hospital-level factors Patient-level factors I-PASS handoff process spreads within study sites Development of curricular components 888 faculty observations of resident handoffs Support from institutional leadership I-PASS handoff process adopted by other disciplines and provider types 48 academic presentations Modify quality of I- PASS Handoff Bundle to be determined Partnership with TeamSTEPPS leadership 1007 requests for curricular materials Grant funding PMID: 24871238 Assumptions External Factors Evaluation

SMALL GROUP EXERCISE

Now Its Your Turn! Use the small group case vignette to fill out the blank logic model Consider: How does the innovation work ? problem inputs activities outputs outcomes Why does (or might) this innovation work ? context assumptions theory or rationale

The Associate Dean for Undergraduate Medical Education at your institution has charged all clerkship directors with developing a Wellness Curriculum for third-year medical students. You are the Clerkship Director for Family Medicine and have been appointed chair of the wellness curriculum development committee. To guide your efforts, the committee will use the Logic model for both conducting an initial needs assessment and planning for the eventual program evaluation. Medical students at your institution have favorable views of the clinical learning environment. Even so, most of students report increased fatigue and concerns about making a specialty choice during their clerkships. The members of the curriculum development committee are willing to work hard on this project and they expect that this curriculum development effort will result in multiple works of peer-reviewed scholarship.

Small Group Debrief What did you gain from working through the logic model? What did you think about the checklist? What was most challenging?

INNOVATIONS AS SCHOLARSHIP

https://www.mededportal.org/submit/instructions/

Why YOU, as an Educator, Are PERFECT for Scholarly Program Evaluation Careful, systematic documentation (e.g., via logic modeling) of your educational journey will help you generate very practical theory. That is, to conduct SCHOLARSHIP! YOU have first-hand exposure to: Learners psychological experiences Contextual factors YOU are the expert on: Subject matter of the intervention Opportunities for and constraints on teaching and learning Scholarly Program Evaluation is a form of participatory action research * Knowledge is built through a sequence of data collection, reflection, and action Educators participate as researchers Inquiry is context-bound *Baum F, MacDougall C, Smith D. Participatory action research. J Epidemiol Community Health 2006; 60(10): 854-7.

Common Limitations of Medical Education Program Evaluations* 1. Focus on Outcomes Only No characterization of the psychological experience/educational process Limited description of context Limited description of other potential solutions 2. Insufficient Outcomes No behavioral data (e.g., reactions, satisfaction, attitudes only) Self-reported Superficial measures of learning (e.g., knowledge quizzes) 3. Lack of a Comparison Group 4. Other Methodological Flaws Small, non-representative sample Non-validated outcome measures No generalizability/sustainability *1. Cook, DA. If you teach them, they will learn: why medical education needs comparative effectiveness research. Adv Health Sci Educ 2012; 17: 305-10. 2. Cook DA, Beckman TJ. Reflections on experimental research in medical education. Adv Health Sci Educ 2010; 15: 455-64. 3. Kanter SL. Toward better descriptions of innovations. Acad Med 2008 Aug;83(8):703-4.

How Can We Make Program Evaluation Count as Scholarship? Include Detailed Description of the Educational Program Theory, Resources Used, Similar Programs Include Descriptions of the Process Outcomes How was the Program Implemented & Resources Used Use multiple outcomes data (e.g. subject scores, faculty evaluations, observations, social networks) Collaborate across multiples sites or multiple programs Contextualize differential outcomes based on trainee characteristics or program implementation status Deliberate case studies Review and include validity information on the tools Conduct validation study whenever possible Use literature review to support your validity information

PUBLISHING INNOVATIONS

Tips for Finding the Right Journal 1 Read the journal description & scope 2 Read the instructions for authors 3 Review a few examples from that journal category 4 If you are unsure, email the editor or editorial team

Reflection or Advice Curriculum Innovation JGME Perspectives On teaching/on learning Medical Teacher Twelve tips Personal view Medical Education When I say Academic Medicine Last page Teaching and learning moments Journal of Interprofessional Care IPEP guides Teaching and Learning in Medicine Observations MedEdPortal MedEdWorld JGME Educational innovation Brief report Medical Teacher Short communications Medical Education Really good stuff Academic Medicine Innovation reports Journal of Interprofessional Care Short reports Teaching and Learning in Medicine Educational case reports Medical Science Educator Innovation Short communications Adapted from: Blanchard RD, Nagler A, Artino AR Jr. Harvest the Low-Hanging Fruit: Strategies for Submitting Educational Innovations for Publication. J Grad Med Educ. 2015 Sep;7(3):318-22.

MESRE Annotated Bibliography https://www.aamc.org/download/456646/data/annotated-bibliography-of-journals-march- 2016.pdf

THANK YOU!