Program Effectiveness and Implementation Research
This research delves into understanding the influence of program implementation on mandatory welfare-to-work programs. It explores how variations in services, implementation, context, clients, and evaluation affect program effectiveness. The study design incorporates data from multi-site randomized trials, analyzing the impact of program implementation on client earnings. Key findings highlight factors that enhance program effects, such as employment messaging, staff attention, caseload sizes, and educational services.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Learning from Variation in Program Effectiveness and Implementation Howard S. Bloom MDRC Presented at a federal inter-agency meeting on Methods for Implementation Research held in Washington DC on September 20-21, 2010.
My Focus What can be learned from research on the determinants of variation in program effects (with an emphasis on program implementation) How this research can be conducted within multi-site randomized field trials, What is required for such research How current federal scale-up projects can provide an effective platform for such research Some major analytic issues that confront such research
My Conceptual Framework Variation in observed program effects can reflect differences in its: Services Implementation Context Clients Evaluation All of this must be taken into account.
My Related Research1 Research Question: How does implementation influence the effectiveness of mandatory welfare-to-work programs? Study Design: A pooled analysis of primary data from three multi-site randomized trials (GAIN, NEWWS and PI) with integrated data collection on characteristics of sites program implementation, context and clients. Study Sample: 59 sites (local welfare offices) with an average per site of: 1176 randomized sample members with individual earnings data from state unemployment insurance records 21 staff survey responses about local program operation/implementation 258 survey responses about service receipt for a random sub-sample of treatment- and control-group members Statistical Model: A two-level random-effects model in which: Level One (for individual clients) specifies mean annual earnings for the first two years after random assignment as a function of clients baseline characteristics and treatment/control status Level Two (for sites) specifies estimated program effects on mean client earnings as a function of sites program implementation, services and economic context Bloom, Howard S., Carolyn J. Hill and James A. Riccio (2003) Linking Program Implementation and Effectiveness: Lessons from a Pooled Sample of Welfare-to-Work Experiments, Journal of Policy Analysis and Management, 22(4): 551 575.
My Related Research (cont.) Key Findings: A mean program-induced earnings gain of $879 per year or 18 percent Other things being equal, program effects increased with: ($720 per )1 ($428 per ) ($268 per ) ($208 per ) ($291 per ) A strong employment message to clients Greater staff emphasis on personal client attention Smaller staff caseloads Less reliance on basic education services Lower unemployment rates Other things being equal, program effects did not vary substantially with differences in client characteristics 1 is the standard deviation of each factor.
What Does Such Research Require? 1. Enough sites (perhaps more than 50)1 2. Large enough sites (with perhaps as few as 100 sample members each)1 3. Consistent, comprehensive, valid and reliable measurement 4. Substantial variation across sites in program effects 5. Substantial variation across sites in program implementation Greenberg, David, Robert Meyer, Charles Michalopoulos and Michael Wiseman (2003) Explaining Variation in the Effects of Welfare-to-Work Programs, Evaluation Review, 27(4): 359 394.
What Are Some Current Opportunities for Conducting Such Research? Federal initiatives to launch and evaluate interventions at scale The White House Social Innovation Fund (SIF) The Department of Education s program of Investment In Innovation (I3) The Department of Health and Human Services program to replicate evidence-based innovations in home visiting, teen pregnancy and fatherhood support
What Are Some Key Analytic Issues for Such Research? How to: Account for site variation in program compliance or dosage (ITT vs. LATE) Develop random effects models for instrumental variables analysis Account for random error in measures of program implementation and context (attenuation bias) Use Generalizability Theory to properly assess measurement reliability Account for this reliability in one s impact analyses Account for treatment- and control- group differences in the amount and quality of services received (their treatment contrast) Test for sensitivity to outliers and specification error
