Innovative Approaches to Program Evaluation and Replication

[G]overnment should be seeking out creative, results- oriented programs like the ones here today and helping them replicate their efforts across America. President Barack Obama, 6/30/2009 http://www.nationalservice.gov/about/ newsroom/ statements_detail.asp?tbl_pr_id=1828 Abt Associates | pg #

When Is a Program Ready for Rigorous Impact Analysis? Diana Epstein (CAP/Center for American Progress) and Jacob Alex Klerman (Abt Associates) APPAM/HSE Conference Improving the Quality of Public Services , Moscow, June 2011 Abt Associates | pg #

Outline The Basic Argument On Logic Models Some Examples Some Broader Implications Discussion Abt Associates | pg #

The Goal Identify program ideas that can successfully address pressing social problems Program Idea Roll them out nationally Broad Rollout Abt Associates | pg #

Require Rigorous Impact Evaluation Many apparently plausible programs don t work Program Idea Many that work in one site, don t work in another site So, require Impact Evaluation tollgate Usually random assignment Saving money Efficacy Trial Replication This is the New Orthodoxy Coalition for Effective Policy OMB (2009) Effectiveness Trial Broad Rollout Abt Associates | pg #

Many Programs Fail RA, So Pilot We argue that a rush to random assignment evaluation has two problems 1. Some programs clearly will not pass the Impact Evaluation tollgate 2. Some of those programs, would pass the Impact Evaluation with more development Program Idea Pilot Efficacy Trial A pilot would help with both problems i.e., run the program for a while Then, if the program is promising Start the Impact Analysis Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

Formative Evaluation/Process Evaluation Formative Evaluation to improve the program Program Idea Process Evaluation to screen out programs that are unlikely to show impact Formative Evaluation Process Evaluation Efficacy Trial Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

Formative Evaluation/Process Evaluation Formative Evaluation to improve the program Program Idea Process Evaluation to screen out programs that are unlikely to show impact Formative Evaluation Process Evaluation But, how do you do that? New Orthodoxy: Only random assignment can reliably detect impact So, how can a Process Evaluation screen? Efficacy Trial Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

Outline The Basic Argument On Logic Models Some Examples Some Broader Implications Discussion Abt Associates | pg #

Falsifiable Logic Models Can Screen Program Idea Require Falsifiable Logic Model Formative Evaluation Process Evaluation Efficacy Trial Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

Falsifiable Logic Models Can Screen Program Idea Require Falsifiable Logic Model Formative Evaluation Revise program and Falsifiable Logic Model Process Evaluation Efficacy Trial Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

Falsifiable Logic Models Can Screen Program Idea Require Falsifiable Logic Model Formative Evaluation Revise program and Falsifiable Logic Model Process Evaluation Only proceed if program satisfies it s own Falsifiable Logic Model Efficacy Trial Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

Why Might this Work? Logic Models explicate the path from resources to impacts Program Idea All but the impact step occur In the treatment group During (or at the end of) treatment Formative Evaluation Process Evaluation Efficacy Trial Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

Why Might this Work? Logic Models explicate the path from resources to impacts Program Idea All but the impact step occur In the treatment group During (or at the end of) treatment Formative Evaluation Process Evaluation Efficacy Trial So, verifying the logic model does not require Random assignment Or even a control group Long program follow-up And expensive post-program survey tracking efforts Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

Outline The Basic Argument On Logic Models Some Examples Some Broader Implications Discussion Abt Associates | pg #

But Will this Screen? Need Examples of Intermediate benchmarks Resources/inputs, Activities, Outputs, Outcomes that were (should have been/could have been) specified in a Falsifiable Logic Model And, that could be detected Using only the treatment group Without an expensive follow-up survey Before (or perhaps shortly after) the end of treatment Here goes Abt Associates | pg #

Forms of Logic Model Failures: 1-3 1. Acquire Resources: Form partnerships, acquire and retain staff (with target qualifications) Salem ERA: Very high staff turnover 2. Recruit Cases: Fill the program Portland ERA: Recruited only a third of target enrollees 3. Sustain Participation Rural WTW Strategies Evaluation; SC Moving Up ERA; Cleveland Achieve ERA Abt Associates | pg #

Forms of Logic Model Failures: 4-5 4. Implement with Fidelity Mathematica Supplemental Reading Evaluation; Abt Mentoring Evaluation 5. Pre/Post Progress MDRC NEWWS HCD program s academic testing (but see GED) Abt Associates | pg #

Outline The Basic Argument On Logic Models Some Examples Some Broader Implications Discussion Abt Associates | pg #

Inducing Truth Telling Currently, program developers have an incentive to over- promise More likely to be funded But, underpowered Impact Evaluations and null results Abt Associates | pg #

Inducing Truth Telling Currently, program developers have an incentive to over- promise More likely to be funded But, underpowered Impact Evaluations and null results Process Evaluation tollgate gives an incentive to under- promise More likely to pass the Process Evaluation tollgate, but Less likely to fund Pilot And, less likely to fund Impact Evaluation Abt Associates | pg #

Inducing Truth Telling Currently, program developers have an incentive to over- promise More likely to be funded But, underpowered Impact Evaluations and null results Process Evaluation tollgate gives an incentive to under- promise More likely to pass the Process Evaluation tollgate, but Less likely to fund Pilot And, less likely to fund Impact Evaluation And if developing a Falsifiable Logic Model forces program developers to more thoroughly and realistically think through their program models, that s good too! Abt Associates | pg #

Key Innovation: Separate Contracts For Program Operator Otherwise, an implicit expectation of proceeding E.g., ED i3, CNCS SIF, Orszag (2009) For Evaluator And probably different contractors Otherwise, an implicit expectation of proceeding And contractual considerations lean towards doing so Abt Associates | pg #

Key Innovation: Separate Contracts For Program Operator Otherwise, an implicit expectation of proceeding E.g., ED i3, CNCS SIF, Orszag (2009) For Evaluator And probably different contractors Otherwise, an implicit expectation of proceeding And contractual considerations lean towards doing so Current practice often runs Process Evaluation simultaneously with Impact Evaluation Abt Associates | pg #

Outline The Basic Argument On Logic Models Some Examples Some Broader Implications Discussion Abt Associates | pg #

Approach Seems Infeasible: Timeline Evaluation timelines are already long Inconsistent with Pressing problems Short-term attention to (and funding for) specific problems This approach would make evaluation timelines much longer Additional piloting Additional contracting between the steps Abt Associates | pg #

Approach Seems Infeasible: Willingness Implicit Assumption: Programs are willing to subject themselves to: Long and burdensome evaluation Possibility (likelihood) of failure Plausible if: Program s goal is broad scale rollout Rigorous evaluation is the only way to get there Programs are confident of passing Some positive examples (Nurse-Family Partnership; Teen Pregnancy Prevention Program; Orszag, 2009) But they are the exception, rather than the rules Abt Associates | pg #

Outline The Basic Argument On Logic Models Some Examples Some Broader Implications Discussion Abt Associates | pg #

When Is a Program Ready for Rigorous Impact Analysis? Diana Epstein (CAP/Center for American Progress) and Jacob Alex Klerman (Abt Associates) When Is a Program Ready for Rigorous Impact Evaluation? APPAM/HSE Conference Improving the quality of Public Services , Moscow, June 2011 Abt Associates | pg #

The Need for Impact Evaluation Many apparently plausible programs don t work Program Idea So, require Impact Evaluation tollgate Usually random assignment Saving money Random Assignment Trial Broad Rollout Abt Associates | pg #

Efficacy Trial/Replication/Effectiveness Trial Some programs that work in one site, don t work in other sites Program Idea So: Efficacy Evaluation (small trial, ideal conditions) Replicate to other (and more) sites Effectiveness trial at the replicated sites (larger trial, real world conditions) Efficacy Trial Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

This Is Hardly New It s the New Orthodoxy Coalition for Effective Policy OMB (2009) Program Idea Efficacy Trial Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

This Is Hardly New It s the New Orthodoxy Coalition for Effective Policy OMB (2009) Program Idea And, we think that s a problem Efficacy Trial Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

Random Assignment Has Lots of Problems Random assignment fits Winston Churchill s description of democracy The worst form of government [evaluation], except for all the others that have been tried from time to time. Program Idea Random assignment is Expensive Long time lines Subjects people to programs that don t work Efficacy Trial Replication Effectiveness Trial Broad Rollout Abt Associates | pg #

Random Assignment Has Lots of Problems Random assignment fits Winston Churchill s description of democracy The worst form of government [evaluation], except for all the others that have been tried from time to time. Program Idea Random assignment is Expensive Long time lines Subjects people to programs that don t work Efficacy Trial Replication Effectiveness Trial Can we do better? Avoid evaluating programs with no impact Improve programs so that they will have impact Broad Rollout Abt Associates | pg #

Thus, Tollgate Is Implementable Falsifiable and specifiable in Logic Model 1. Acquire Resources 2. Recruit Cases 3. Sustain Participation 4. Implement 5. with Fidelity 6. Pre/Post Progress Measured in Treatment Group only No expensive follow-up survey needed Occurs during or shortly after program activities Abt Associates | pg #

Thus, Tollgate Is Implementable Falsifiable and specifiable in Logic Model 1. Acquire Resources 2. Recruit Cases 3. Sustain Participation 4. Implement 5. with Fidelity 6. Pre/Post Progress Measured in Treatment Group only No expensive follow-up survey needed Occurs during or shortly after program activities with a Pilot Implementation and a Process Evaluation Abt Associates | pg #

When Is a Program Ready for Rigorous Impact Analysis? Diana Epstein (CAP/Center for American Progress) and Jacob Alex Klerman (Abt Associates) When Is a Program Ready for Rigorous Impact Evaluation? APPAM/HSE Conference Improving the quality of Public Services , Moscow, June 2011 Abt Associates | pg #

Logic Model Definition The program logic model is defined as a picture of how your organization does its work the theory and assumptions underlying the program. A program logic model links outcomes (both short- and long-term) with program activities/processes and the theoretical assumptions/principles of the program. Source: W.K. Kellogg Foundation Logic Model Guide http://www.wkkf.org/~/media/6E35F79692704AA0ADCC8C3017200208.ashx Abt Associates | pg #

Logic Model: Your Planned Work YOUR PLANNED WORK describes what resources you think you need to implementyour program and what you intend to do. 1. Resources include the human, financial, organizational, and community resources a program has available to direct toward doing the work. Sometimes this component is referred to as Inputs. 2. Program Activities are what the program does with the resources. Activities are the processes, tools, events, technology, and actions that are an intentional part of the program implementation. These interventions are used to bring about the intended program changes or results. YOUR INTENDED RESULTS include all of the program s desired results (outputs, outcomes, and impact). Abt Associates | pg #

Logic Model: Your Intended Results YOUR PLANNED WORK describes what resources you think you need to implementyour program and what you intend to do. YOUR INTENDED RESULTS include all of the program s desired results (outputs, outcomes, and impact). 3. Outputs are the direct products of program activities and may include types, levels and targets of services to be delivered by the program. 4. Outcomes are the specific changes in program participants behavior, knowledge, skills, status and level of functioning. Short-term outcomes should be attainable within 1 to 3 years, while longer-term outcomes should be achievable within a 4 to 6 year timeframe. 5. Impact is the fundamental intended or unintended change occurring in organizations, communities or systems as a result of program activities within 7 to 10 years. Abt Associates | pg #

If you dont know where youre going, how are you gonna know when you get there? Yogi Berra, New York Yankees Player and Manger, 1925- Happy families are all alike; every unhappy family is unhappy in its own way. Anna Karenina, Chapter 1, first line Leo Tolstoy, Russian mystic & novelist (1828 1910) Abt Associates | pg #

Paper Status Paper is nights and weekends work In reaction to evaluation experience positive and negative Has been presented and read internally (Abt JASG) and externally (N. Campbell/ACF, Burt Barnow, Demetra Nightingale; we hope soon B. Kelly/ACF) Probably going to try to present at ACF Your comments on presentation, on paper, and on ideas much appreciated In particular, more and better examples (We hope) to a journal soon Abt Associates | pg #

Goal: Effective Programs Efficacy Evaluation Program Idea Broad-scale Rollout Replication Effectiveness Evaluation Abt Associates | pg #

Question: How to Get There? Efficacy Evaluation Replication ? Program Idea Broad-scale Rollout Effectiveness Evaluation Abt Associates | pg #

Random Assignment is Necessary Most rigorously evaluated programs fail Even programs that pass initial efficacy trial often fail follow-on effectiveness trial And the more rigorous the evaluation, the more likely is failure => Evaluate before roll-out Otherwise implement ineffective programs Abt Associates | pg #

Suggests a Random Assignment Tollgate Efficacy Evaluation Program Idea Broad-scale Rollout Replication Effectiveness Evaluation Abt Associates | pg #

Random Assignment is Necessary, but Expensive Most rigorously evaluated programs fail Even programs that pass initial efficacy trial, often fail follow-on effectiveness trial In dollars In calendar time In the lives of clients/participants who waste time in programs that don t work And the more rigorous the evaluation, the more likely is failure => Evaluate before roll-out Otherwise implement ineffective programs Abt Associates | pg #

Random Assignment is Necessary, but Expensive Most rigorously evaluated programs fail Even programs that pass initial efficacy trial, often fail follow- on effectiveness trial In dollars In calendar time In the lives of clients/participants who waste time in programs that don t work And the more rigorous the evaluation, the more likely is failure => Evaluate before roll-out Otherwise implement ineffective programs => Don t evaluate programs that will fail. Duh! Abt Associates | pg #

Random Assignment is Necessary, but Expensive Most rigorously evaluated programs fail Even programs that pass initial efficacy trial, often fail follow- on effectiveness trial In dollars In calendar time In the lives of clients/participants who waste time in programs that don t work And the more rigorous the evaluation, the more likely is failure => Evaluate before roll-out Otherwise implement ineffective programs => Don t evaluate programs that will fail. Duh! Abt Associates | pg #