Understanding Rater Reliability in Behavioral Health Concerns
Explore the concept of rater reliability in the context of common behavioral health concerns. Discover the types of reliability that matter most, the impact of variation by raters, and the significance of inter-rater reliability in research.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Rater Reliability: Science and Practice Nate Israel, PhD Chapin Hall for Children at the University of Chicago
Goals Understand what Reliability means to participants Review scientific literature on inter-rater reliability of common behavioral health concerns Describe the difference between pre- and post- rating triangulation Walk through scenarios for making sure our CANS / ANSA ratings are reliable
Reliability When someone in our profession says that something is reliable or not, what does that mean? When we are using the CANS / ANSA, what kinds of reliability matter the most to us? Are there times when we actually expect our data to be unreliable in some way?
Science, Psychometrics, and Reliability Common types of reliability: Cross-time Cross-item Cross-rater What does each of these tell us, clinically?
Science, Psychometrics, and Reliability Research on inter-rater reliability of common behavioral health needs Raters commonly include: Professional (often a psychiatrist if diagnoses are involved) Parent Youth sometimes an additional party, such as a teacher
Reliability: The Data Research findings are (generally) consistent across a variety of studies What would you guess that inter-rater reliability of common behavioral health concerns would be? .9 is exceptionally reliable .8 is highly reliable .7 is often the minimum threshold for an instrument s reliability (for research purposes) .6 is seen as somewhat unreliable
Reliability: AAAAAAAAhhhhhhhhhhh!!!! Science agrees: raters disagree Disagreement is substantial. Disagreement is consistent What do we do with disagreement??
Reliability: A Pragmatic Approach This disagreement is discovered with most behavioral health measures after the measure is filled out The CANS and ANSA are designed to be completed differently The goal of the collaborative process underlying the CANS and ANSA is to surface and address disagreement before a rating is made
Reliability: Building Collaboration This approach is sometimes referred to as pre- rating triangulation This means that you get important information from all relevant / available sources before making a rating You make the rating in consultation with the client
Reliability: Moments of Truth In making a rating this way you can surface differences of opinion and address them This is the heart of therapy: building and acting on a common understanding It s probably part of why collaborative assessment processes are associated with both better engagement and a small TREATMENT effect
Reliability: The Tough Parts True Collaboration often means dealing with initially strong differences of opinion about needs and strengths The CANS is designed to help people get through the toughest, most stigmatizing part of disagreement: the Why It does this by allowing you to build a sense of Why something is happening together with the client
Example: Different Perspectives Johnny is a 16-year old male youth His Spanish teacher reports that he has been skipping class and thinks that he is hanging out with the wrong crowd and probably getting into trouble and wants these issues addressed Johnny reports that he sometimes misses Spanish class because it s right after lunch and denies being involved in any delinquent behavior
Example: Different Perspectives Jenna is a 36-year old female adult Her CPS caseworker reports that she recently failed a drug screen (heroin), and that as long as she uses, her child s permanency plan will not include her as the preferred placement outcome for the child Jenna strongly denies using heroin and loudly protests being rated as needing to address Substance Use concerns. She wants her child back in her custody.
Example: Different Perspectives How can you make this about the What? What kind of treatment goal(s) would be consistent with Jenna s perspective and desire, and the case worker s needs?
Reliability: A CANS/ANSA Perspective Reliability is not separate from our process of relating to, and working with our child, family, and adult clients The more we make the ratings about their action implications, the more useful, and reliable, they become
Professional-on-Professional Doubt We may have questions about how a professional arrived at a CANS / ANSA rating The CANS / ANSA are designed to be transparent When there is a question about a rating, it should be asked
Professional-on-Professional Doubt We have a dirty little secret in some places of our profession; we sometimes institutionalize unreliability This may seem protective of clients, the system, etc Yet with the CANS and ANSA accuracy is advocacy For us to get better we have to be honest and collaborative with our clients, selves, supervisors, and managers
Additional Thoughts / Questions Thanks for spending this morning together. If there s any other way I can be helpful, please let me know. I hope to see you at the CANS conference in November!
