Importance of Evidence-Based Evaluation in Government Programs

Federal Evidence and Evaluation Agenda Diana Epstein Office of Management and Budget

Why does evaluation matter? Limited resources For training (and all government functions) We want our programs to deliver their intended impact This is true for training (and all other programs) We want to use our limited resources in the smartest way possible to achieve our intended impact and drive continuous improvement What strategies work, for whom, and under what conditions? What is the most cost-effective strategy to achieve that impact in a given circumstance? Is the strategy being implemented as intended? Evidence-based policy and evidence-based budgeting

What is evidence? Evidence can be many things. For example: Impact evaluations, including randomized control trials and high-quality quasi-experimental designs can answer questions about a program s impact relative to a counterfactual (no intervention or a different intervention) Process or implementation evaluations can answer questions about whether a program is being implemented as designed or whether the fundamental structure of a program is realistic Program measures can answer questions about program efficiency, outputs and outcomes, but do not provide information about impact Statistics and other forms of research and analysis can provide insight into trends, strategies, and underlying principles, but are less specific to particular program operations Collectively we refer to the above as a portfolio of evidence, each piece of which provides information about a different aspect of the problem

Goals for evidence and evaluation Evaluation is supported throughout the federal budget and more broadly by other partners and stakeholders Use strong evidence of what works to improve performance Evaluation is a mechanism to show us what works and what is not working Use this information to learn and improve Build evidence when we need it We have much to learn about what works, for whom, and under what circumstances Develop capacity to make it easier to build and use evidence in the future Improve agency capacity to conduct evaluations and become critical consumers Utilize new tools and methods

How can evaluation help us? Homelessness, especially chronic homelessness, was long considered an intractable problem Historically approached with a great deal of compassion but little analytical rigor In 2010, the Administration released Opening Doors, the nation s first-ever strategic plan to prevent and end homelessness Among other things, plan set forward an aggressive strategy to end Veteran homelessness A broad body of research, including rigorous evaluations, demonstrated that permanent supportive housing is more cost-efficient and effective than other approaches Various studies have shown that permanent supportive housing not only improves housing stability but can also reduce costs for emergency department and inpatient services

Evidence-based results Since 2010, working in partnership with communities across the country, we ve reduced: Chronic homelessness by 22 percent, Family homelessness by 19 percent, Overall veterans homelessness by 36 percent, and The number of unsheltered homeless veterans by nearly half The State of Virginia and communities across the country (including New Orleans, Las Vegas, Houston, and Philadelphia) have put systems in place to end veteran homelessness

Evaluations and program administration Improving Retirement Savings Among Service Members: Servicemembers were under-enrolling in retirement savings; just 42 percent of active duty Servicemembers were enrolled in Thrift Savings Plans (TSP), compared to 87 percent of civilian Federal employees Evidence: Evidence from a range of applications suggested that clarifying and streamlining enrollment processes (such as that for TSP) can promote participation in programs Evaluation: The Social and Behavioral Sciences Team (SBST) collaborated with DOD on a RCT to test the effect of sending email messages crafted using behavioral insights on Servicemember enrollment in TSP Messages informed by behavioral insights (relative to business-as-usual) led to roughly 4,930 additional enrollments and $1.3 million in additional savings in the first month alone The most effective message nearly doubled the rate at which Servicemembers enrolled Result: DOD has committed to scaling-up this practice by sending three emails per year (with effective messages) to the entire population of Servicemembers who are not enrolled in TSP Message testing is ongoing, learn and improve

Evaluations and program administration Simplifying the Free Application for Federal Student Aid (FAFSA): The Administration wanted to increases college enrollment, particularly among low-income and other disadvantaged populations Evaluation: Rigorous evaluations have found that making it easier for students to apply for federal financial aid can increase college enrollment, and it would be feasible to do Action: On the basis of that evidence the Department of Education took steps over the last several years to simplify the FAFSA, including revamping the online form for all families so they can skip questions that are not relevant to them and automatically retrieve needed tax information when filling out the FAFSA Result: The time required to complete the FAFSA was reduced by two- thirds to about 20 minutes More than 6 million students and parents took advantage of the ability to electronically retrieve their income information from the IRS when completing their 2014-2015 FAFSA, an innovation that improves both speed and accuracy Other evaluations are testing the impacts of various FAFSA awareness strategies

Key ideas Promote high quality evaluation, regardless of type Rigorous, well-designed, and well-implemented Use the most rigorous methods that are appropriate to the evaluation questions and feasible within budget and other constraints Let the question drive the method Apply the type of evaluation, or other evidence tool, best suited for the particular question Develop credible evidence Five themes: Rigor, Relevance, Independence, Transparency, and Ethics Use evaluation to learn and improve, rather than as an up or down vote

Best practices Learning agenda Build and use a portfolio of evidence Match the most appropriate tool to answer each question Utilize new methods/opportunities when appropriate Rapid-cycle iterative evaluation designs Use of administrative data Applying behavioral insights Evidence Commission: White Papers Once you have the evidence, use it!

Need to build staff capacity for evaluation Limited staff capacity to generate and use evidence is a barrier Important to identify high-quality staff training opportunities in evaluation Some agencies have well-developed internal trainings, and other agencies have a wealth of resources posted online USAID Learning Lab Evaluation Toolkit: https://usaidlearninglab.org/evaluation EPA Logic Model Training: https://www3.epa.gov/evaluate/lm-training/ CNCS Evaluation Resources: http://www.nationalservice.gov/resources/evaluation CDC Evaluation Resources: http://www.cdc.gov/eval/resources/index.htm External training providers (American Evaluation Association, Evaluators Institute, etc.)

Current efforts to build staff capacity Principles and Practices Interagency Council on Evaluation Policy Best Practices in Contracting for Evaluation-related Technical Assistance Evaluation-specific COR training Evaluation Hiring Tips Efforts to include evidence-based decision making in SES trainings

Additional Resources FedEval (Program Evaluation Federal Community): ShipmanS@gao.gov Social and Behavioral Sciences Team: https://sbst.gov/ FY 2017 Analytical Perspectives Chapter 7: Building the Capacity to Produce and Use Evidence OMB Evidence Page: https://www.whitehouse.gov/omb/evidence

Questions? Contact: Diana Epstein depstein@omb.eop.gov 202-395-4621