Drive Trauma-Informed Change Through Data: A Guide to Measuring Care

Measuring Trauma-Informed Care: Using Data to Drive Trauma- Informed Change Steve Brown, Psy.D. Traumatic Stress Institute Klingberg Family Centers www.traumaticstressinstitute.org 413-218-3293

Logic Model for Trauma-Informed Change Process

Inputs Resources go in TIC change Activities performed Measurable shorter term outcomes Longer term measurable outcomes Leadership buy-in Money Staff time Training & consultation about trauma and TIC Continuing education Organizational focus on TIC change Formation TIC Task Force and/or RC Trainer Group Mandated RC trauma training RA training and implementation (if applicable) Trauma screening RC refreshers Integrate RC principles in supervision Revision of crisis prevention forms Application of Trauma Framework to individual clients Creation of forums to address VT Etc . STAFF LEVEL OUTCOMES Increased knowledge (RC knowledge scale) Increase in staff attitudes favorable to TIC (ARTIC) Increase in staff behaviors indicative of TIC (Indicators of TIC in Tx Programs) More favorable professional quality of life (PROQOL) STAFF LEVEL OUTCOMES Reduced turnover Reduced staff injuries (client related) Reduce staff call-outs/sick time CLIENT LEVEL OUTCOMES Reduced restraint and seclusion Reduced serious incident reports Reduced calls/visits from police Increase in positive discharges Reduced ER assessments and visits Reduced AWOLs CLIENT LEVEL OUTCOMES Decrease in PTSD Sxs (UCLA PTSD Index) Decrease in common trauma sxs (Trauma Sx Checklist) Decrease in overall problem behavior (CBCL; Ohio Scales) SYSTEM OUTCOMES Higher score on org assessment tool (TICOMETER and others) SYSTEM LEVEL OUTCOME Cost savings

Organizational Self-Assessment What have been the inputs at your organization? (Listed in chart or others) What have been primary activities performed at your organization (Listed in chart or others) What shorter term or longer term metrics have you used (if any)? Which of the longer term staff and client level outcomes do you routinely collect? (if any)

Design Your Own Outcome Evaluation 1. What is your capacity to collect data, enter it into database system, analyze? How will you do so? Are there ways to increase this capacity over time? 2. What specifically do you want to learn? 3. Who do you want to learn about? 4. At what time points do you want to collect data? 5. What measures will you use? 6. How will you collect the data? 7. How will you ensure client or staff feel safe to answer honestly? 8. How will you give feedback to participants? 9. How will you use the information in decision making?

What is your capacity to do outcome evaluation? What data do you collect already on a regular basis? Is it stored in a database system? Who would administer measures? enter into database? analyze? (In house? Grad student? Outside evaluator?) How could you org build capacity for using data?

What specifically do you want to learn? (short term outcomes) Examples: We are intensifying TIC implementation, what is the outcome? What org job roles display most TI attitudes Did staff knowledge increase as a result of RC training? Did staff attitudes favorable to TIC increase as result of TIC? To what extent have we sustained TIC over time?

What specifically do you want to learn? (long term outcomes) Examples: Did TIC implementation impact staff-related metrics (turnover, injuries, call-out time, etc.) Did TIC impact client-related metrics? (restraint and seclusion, incident reports, etc) Did TIC implementation impact client symptoms/problem behavior?

CASE EXAMPLE 80% trained in RC; 20% not. New effort to sustain TIC: Intense booster RC refreshers and 3-Day Basics require all programs to do Trauma Frameworks on 10% of clients within 2 months All programs must design and use VT/self-care ritual in their program every meeting

CASE EXAMPLE - What want to learn? Does the initiative increase overall attitudes favorable to TIC from baseline to 4 months out? Which programs and which job roles increase the most or the least?

Who do you want to learn about? Examples The system as a whole Staff in the whole organization Staff in particular job roles Staff in particular programs Clients

CASE EXAMPLE: Who do we want to learn about? Staff as whole Staff in different programs Staff in different job roles

At what time points do you want to collect data? Pre- and post- RC Basic? Baseline and 6 months later? Pre- and post- RC Basic AND 6 months later? Etc, etc etc

CASE EXAMPLE At what time points do you want to collect data? Baseline at beginning of RC Refreshers and RC Basics. 4 months post conclusion of Refreshers and Basics.

What measure(s) will you use? (shorter term) What is measure reliability and validity? Staff-level metrics RC Knowledge Assessment Attitudes Related to Trauma-Informed Care (ARTIC) Scale Professional Quality of Life (ProQOL) Scale System metrics TICOMETER

The ARTIC Scale

Why the need for the ARTIC? Trauma-informed care movement rapidly expanding TIC has become a buzzword; operationalization of TIC vague and varied Literature on TIC is rich with theories, principles, and implementation ideas; BUT there is currently very little empirical evidence for TIC Lack of objective tools to measure TIC Research tools and designs expensive and impractical for practice settings

How does the ARTIC fill the gap? Reflects and synthesizes the current theoretical and empirical knowledge of TIC Psychometrically validated Spur TIC field to drill down to what is, and is not, trauma-informed Push TIC field to more data-driven decision making Easy and inexpensive to administer

Why measure staff attitudes? In overarching system that fosters and supports TIC Staff attitudes Moment-to-moment staff behavior with clients/students Positive outcomes for clients/students, staff, system

Results: ARTIC Subscales Approach: Item analyses and confirmatory factor analysis (CFA) 5 Main Subscales Attitudes about: The underlying cause of problem behavior/symptoms (Subscale 1) Staff responses to problem behavior (Subscale 2) Staff on-the-job behavior (Subscale 3) Staff feeling of self-efficacy at work (Subscale 4) Staff reactions to the work (Subscale 5) 2 Supplementary Subscales Attitudes about: Staff members personal support of TIC (Subscale 6) System-wide support for TIC (Subscale 7)

Versions of the ARTIC Human Services ARTIC-45 (Orgs that have begun TIC implementation) ARTIC-35 (Orgs that have not implemented TIC) ARTIC-10 (Short form; fine for orgs that have or have not implemented TIC) Education ARTIC-45 (Schools that have begun TIC implementation) ARTIC-35 (Schools that have not implemented TIC) ARTIC-10 (Short form; fine for orgs that have or have not implemented TIC)

How to obtain the ARTIC? Cost: $450 one-time fee per organization (introductory offer). $150 for RC agencies. Can use as much as like, now and in future, within the organization. Includes: All versions of ARTIC Information about how to use and score ARTIC Scoring Spreadsheet (calculates subscale and total scores for you) See Traumatic Stress Institute website for details about purchasing. Email: artic@klingberg.com Fees from the ARTIC fund further research on TIC and other TSI educational projects.

Professional Quality of Life (ProQOL) Scale Psychometric measure Measures the positive and negative aspects of doing one s job that influence one s professional quality of life. 3 subscales Compassion Satisfaction (Pleasure derived from being able to work well) Burnout (Feelings of exhaustion, frustration, anger, depression, hopelessness at work) Secondary Trauma (Work related secondary exposure to people experiencing traumatic events. Includes sleep difficulties, intrusive images, or avoiding reminders of the person s trauma)

TICOMETER Psychometric properties Measures perception of staff about trauma-informed care at an organizational level in five domains 1. Build TI knowledge and skills 2. Establish trusting relationships 3. Respect service users 4. Foster TI service delivery 5. Promote TI procedures and policies https://www.ticometer.com/ (Bassuk, Unick, Paquette, & Richard, 2016)

CASE EXAMPLE: What measure(s) will you use? ARTIC-45 for Human Services

What measure(s) will you use? (longer term) Client Level Outcomes Restraint and seclusion Serious incident reports Calls/visits from law enforcement Nature of discharges ER assessments and visits Reduced AWOLS Issues to Consider The more data you have prior to baseline and after intervention, the more trustworthy your results. Many factors other than TI implementation can impact these metrics Interrupted time series research design is option (need good researcher)

What measures will you use? (longer term) Client level outcomes Decrease in symptoms, problem behaviors UCLA PTSD Index Trauma Symptom Inventory (TSI) (age 17+); Trauma Symptom Checklist for Children (TSCC) (age 7-17) Trauma Symptom Checklist for Young Children (TSCYC) (age under 7) Child Behavior Checklist (CBCL) Ohio Scales

How to ensure staff feel safe to answer honestly? Informed consent Ensure that demographic data doesn t compromise anonymity Use of coding system to match the measure at different time points Use of outside evaluator increases safety Place measure in sealed envelope.

CASE EXAMPLE: How to ensure staff feel safe to answer honestly? Create a consent form about the outcome evaluation that staff sign. Staff write code (last 4 digits of cell phone and month born) on packet which use to match with one complete at 4 months. No names except signature on consent form. When complete measure, place in blank envelop and seal. Sealed measures get mailed directly to graduate student research assistant.

How to give feedback to participants? Feedback to staff is an intervention itself Help people feel like study not waste of their time Used as a way to solicit input about next steps

CASE EXAMPLE: How to give feedback to participants? Present to staff in an all-staff forum and discuss implications and future steps to be taken.

How will you use information in decision making? Data should inform decisions about next steps Data can help justify your work and leverage additional funding

Inputs Resources go in TIC change Activities performed Measurable shorter term outcomes Measurable longer term outcomes Leadership buy-in Funding Staff time Organizational research about TIC change Link to organizational mission Priority as organizational strategic goal Building organizational readiness Formation TIC Task Force Organizational assessment Choosing TIC model and/or consultants to change process Mandated staff trauma training Trauma screening Integration into policy and procedure Improve capacity for use of data Implement trauma- specific treatments Change behavior management system (if applicable) Revise staff screening, hiring, and supervision Staff supports for secondary trauma STAFF LEVEL OUTCOMES Increased knowledge Increase in staff attitudes favorable to TIC (Attitudes Related to Trauma- Informed Care (ARTIC) Scale, Baker et. al.) Increase in staff behaviors indicative of TIC (Indicators of TIC in Tx Programs) Increase professional quality of life (Professional Quality of Life Scale, Stamm) STAFF LEVEL OUTCOMES Reduced turnover Reduced client-related staff injuries Reduce staff call-outs/sick time EXAMPLES OF CLIENT LEVEL OUTCOME METRICS Decreased frequency of visits Increased behavioral health referrals Reduced restraint and seclusion Reduced serious incident reports Reduced calls/visits from police Increase in positive discharges EXAMPLES OF CLIENT LEVEL OUTCOME VIA MEASURES Decrease in PTSD Sxs (UCLA PTSD Index, Pynoos) Decrease in common trauma sxs (Trauma Symptom Inventory (TSI), Trauma Symptom Checklist for Children (TSCC), Briere) Decrease in overall problem behavior (Child Behavior Checklist (CBCL), Achenbach) SYSTEM OUTCOMES Increased integration of TIC within organization (ex. TICOMETER) Improved client experience of services (client survey) EXAMPLES OF SYSTEM LEVEL OUTCOME