Response Bias in Evaluation Processes in Graduate Medical Education

Faculty Development Session on Evaluations Graduate Medical Education Ann Dohn, MA, Designated Institutional Official (DIO) Nancy Piro, PhD Program Manger/Education Specialist Kim Walker, PhD Program Manger/Education Specialist 1 Department of Graduate Medical Education (GME)

ACGME Requirements: Seven Domains I. Institution II. Program Personnel & Resources III. Resident Appointments IV. Educational Program V. Evaluation I. Resident II. Faculty III. Program VI. Resident Duty Hours VII. Experimentation & Innovation 2 Department of Graduate Medical Education (GME)

Faculty Development Session EVALUATIONS Our focus today: How do we eliminate unintended bias from our evaluation process? 3 Department of Graduate Medical Education (GME)

What is cognitive bias Cognitive bias is distortion in the way we perceive reality / information. Response bias is a type of cognitive bias which can affect the results of an evaluation if evaluators answer questions in the way they think they are designed to be answered, or with a positive or negative bias toward the fellow being evaluated 4 Department of Graduate Medical Education (GME)

Where does response bias occur? Response bias most occurs most often in the wording of the question. Response bias is present when a question contains a leading phrase or words. Response bias can also occur in rating scales. Response bias can be in the raters themselves Central Tendency Halo Effect Similarity Effect 5 Department of Graduate Medical Education (GME)

Examples of Question Bias Example 1: "I can always talk to my Program Director about residency related problems." Problem: Terms such as "always" and "never" will bias the response in the opposite direction. Result: Data will be skewed. 6 Department of Graduate Medical Education (GME)

Examples of Question Bias Example 2: Career planning resources are available to me and my program director supports my professional aspirations." Problem: Double-barreled ---resources and aspirations Respondents may agree with one and not the other. Evaluator cannot make assumptions about which part of the question respondents were rating. Result: Data is useless. 7 Department of Graduate Medical Education (GME)

Examples of Question Bias Example 3: "Communication in my program is good." Problem: Question is too broad. If score is less than 100% positive, researcher/evaluator still does not know what aspect of communication needs improvement. Result: Data is of little or no usefulness. 8 Department of Graduate Medical Education (GME)

Rating Scale Bias Competence and knowledge in general medicine. Poor Fair Good Very Good Excellent The data will be artificially skewed in the positive direction with this scale because there are far more (4:1) positive than negative rating options. 9 Department of Graduate Medical Education (GME)

Rater/Evaluator Bias Response bias can be in the evaluators themselves Central Tendency Similarity Effect Halo Effect 10 Department of Graduate Medical Education (GME)

Beware the Halo Effect The halo effect refers to a cognitive bias whereby the perception of a particular behavior or trait is influenced by the perception of the former traits in a sequence of interpretations. Thorndike (1920) was the first to support the halo effect with empirical research. People seem not to think of other individuals in mixed terms; instead we seem to see each person as roughly good or roughly bad across all categories of measurement. 11 Department of Graduate Medical Education (GME)

The Halo Effect and Expectations The halo effect is involved in Kelley's implicit personality theory the first traits we recognize in other people influence our interpretation and perception of later ones because of our expectations. 12 Department of Graduate Medical Education (GME)

Halo Effect Extends to Products / Marketing The iPod has had positive effects on perceptions of Apple s other products 13 Department of Graduate Medical Education (GME)

Could this impact our evaluations here? Empirical evidence from our HouseStaff . A question from the most recent GME HouseStaff survey: The general feeling in my program is that your ability will be labeled based on your initial performance. Overall SHC-LPCH Overall Peds Fellows 71.8% 10.3% 17.9% 54.7% 15.6% 24.4% 14 Department of Graduate Medical Education (GME)

Reverse Halo Effect A corollary to the halo effect is the reverse halo effect (devil effect) individuals, brands or other things judged to have a single undesirable trait are subsequently judged to have many poor traits, allowing a single weak point or negative trait to influence others' perception of the person, brand or other thing in general. 15 Department of Graduate Medical Education (GME)

Blind Spots In the 1970s, the social psychologist Richard Nisbett demonstrated that we may have no awareness of when the halo effect influences us (Nisbett, R.E. and Wilson, T.D., 1977) The problem with Blind Spots is that we are blind to them 16 Department of Graduate Medical Education (GME)

Know the ACGME Core Competencies and How You are Evaluating Them Medical Systems based Practice Professionalism Interpersonal Communication Patient Care Practice based Learning & & Improvement 17 Department of Graduate Medical Education (GME)

Questions? 18 Department of Graduate Medical Education (GME)