Evidence Base and Tiers for Program Evaluation

Evidence Base Andrew MacDonald Michael Long November 15, 2018 Volunteer West Virginia Training Charleston, WV

Learning Objectives By the end of this presentation, you will understand: How points for the Evidence Base section are awarded for the 2019 competition What information you need to include about your program to get as many points as possible 2

Evidence Base Evidence base refers to how much evidence you have that your program will work that is, that your program will be effective at producing the intended outcomes in your logic model Your evidence could be from: a. An evaluation you conducted of your own program; b. An external evaluation someone else did of your program; or c. Evaluation studies from other organizations of the same intervention you plan to implement. 3

Evidence Base You can earn up to 16 points for your evidence base: 8 points are awarded based on which evidence tier you are in 8 points are awarded for the quality of the evidence that you provide 4

Evidence Tiers Presentation Title 3/18/2025 5 ICF proprietary and confidential. Do not copy, distribute, or disclose. ICF proprietary and confidential. Do not copy, distribute, or disclose. ICF proprietary and confidential. Do not copy, distribute, or disclose.

Three Changes Related to Evidence Tiers in 2019 NOFO There are now only four evidence tiers; the No Evidence tier no longer exists Up to 8 points are awarded based only on evidence tier; presumably applicants in higher tiers will receive more points The new tiers allow you to submit studies of programs other than your own as evidence However, you need to be sure you show in detail how your program is the same as the interventions that were studied 6

Quick Refresher: Types of Evaluations A process evaluation examines how a program works, by studying its inputs, activities, and outputs For example, a process evaluation might look at whether a program is reaching its intended beneficiaries; whether program activities are being carried out as intended; or whether participants are satisfied with the services they are getting An outcome evaluation focuses more on whether a program works that is, whether it brings about the intended outcomes in its logic model For example, an outcome evaluation might look at changes in knowledge, attitudes, or behavior among program participants Outcome evaluations are generally quantitative (i.e., measure something in terms of numbers or percentages) 7

Quick Refresher: Impact Evaluations An impact evaluation is a specific type of outcome evaluation that includes a comparison group, and is considered a stronger way of showing that your program caused outcomes. There are two main types of impact evaluation designs: A randomized controlled trial (RCT) randomly assigns some people to participate in the program, and other people to be in a control group that does not participate in the program. A quasi-experimental design (QED) study compares results between program participants and another group of similar people that did not receive the program (but does not randomly assign people between the two groups). 8

Evidence Tiers Pre-Preliminary Incorporates research from other evidence- based programs into program design; Preliminary Shows positive findings from an outcome evaluation of the same intervention as proposed. Moderate Shows positive findings from an impact evaluation of the same intervention as proposed. Strong Shows positive findings from an impact evaluation of the same intervention as proposed. and/or May be internal or external evaluation. Cannot be an internal evaluation. Cannot be an internal evaluation. Demonstrates collection of performance measure data on proposed intervention May be a single site evaluation. Must be multi-site and/or with different populations. 9

Evidence Tiers of 2018 Competitively Funded Applicants Strong, 9% Moderate, 5% Pre-preliminary, 46% Preliminary, 40% 10

Pop Quiz: What Tier Am I? What evidence tier could the following evidence qualify you for? 1) A study conducted by an external evaluation showing that participants in your program had better outcomes that those in a comparison group 2) A study of a program that was similar to yours in some ways that showed some positive results 3) A process evaluation that showed that your program activities were being implemented according to your model 4) A study you conducted of your program that showed that scores on a post-test were higher than those on a pre-test 5) A high-quality internal RCT impact study you conducted of your program that showed positive and significant findings 11

Applicants with Multiple Proposed Interventions An intervention is your program s activity that is, what you propose to use AmeriCorps members to do. Many AmeriCorps members propose multiple interventions in their applications. If you are proposing to use AmeriCorps members to do multiple things, you do not need to provide evidence for all of those different interventions you can qualify for a tier based on evidence for just one intervention Therefore, focus on finding the best evidence for one of your proposed activities, rather than finding some evidence for all of your proposed activities 12

Evidence Quality Presentation Title 3/18/2025 13 ICF proprietary and confidential. Do not copy, distribute, or disclose. ICF proprietary and confidential. Do not copy, distribute, or disclose. ICF proprietary and confidential. Do not copy, distribute, or disclose.

Evidence Quality After the reviewers assign your application to a specific evidence tier, they will assess the quality of the evidence that you provided The criteria for how they will judge evidence quality depend on the tier you are in the criteria are tougher for higher evidence tiers If you think you will qualify for the Preliminary, Moderate, or Strong tiers, you should also address the criteria for the Pre- preliminary tier as well just in case that is how you are assessed In other words, if you miss out on one of the higher tiers, you could get bumped down to Pre-preliminary and if you haven t addressed those criteria you could get 0 points for Evidence Quality 14

Evidence Quality (Pre-preliminary) For applicants rated pre-preliminary, evidence quality is based only on the narrative in the Evidence Base section you cannot submit any other studies. For pre-preliminary applicants, the goal is to show how your program design is evidence-informed that is, how you have used data or evaluation results to develop your logic model You can show this by (a) describing evidence from other studies of similar programs and/or (b) describing data that you have collected about your own program (e.g., performance data), and specifically showing how you have used that evidence to inform your logic model 15

Evidence Quality (Pre-preliminary) For Pre-preliminary applicants, the reviewers will assess whether the evidence: Is relevant to their proposed program design, i.e., aligned with their logic model Is recent (preferably within the past 6 years) Shows meaningful positive effects on program participants 16

Evidence Quality (Pre-preliminary) Examples of how to connect evidence or data to your logic model: Researcher X found that tutoring worked best when children received at least two hours of tutoring a week. Therefore, we designed our program to ensure that AmeriCorps members will provide each child with at least two hours of tutoring per week. (Connecting research to program activities) In our last grant cycle our performance measurement data showed that we were able to reach many men with our program, but not as many women as we had hoped. Therefore, we have adjusted our recruitment activities to do ______ in order to reach more women. (Connecting performance data to program activities) 17

Evidence Quality (Pre-preliminary) Another example of how to connect evidence or data to your logic model: Researcher Y found that providing services to the homeless had a positive effect on their emotional well-being, and Researcher Z found that emotional well-being was related to people s ability to find permanent housing. Based on these two studies, we believe that the services we provide to the homeless will help them find permanent housing. 18

Evidence Quality (Preliminary, Moderate, Strong) When reviewing evidence quality for applicants rated Preliminary, Moderate, or Strong, reviewers will look at both the narrative in the Evidence Base section and up to two studies submitted by the applicant (plus their own evaluation report, if applicable) In the Evidence Base section, be sure to summarize the studies that you are providing, their findings, and how they connect with your own logic model Be sure that the interventions that the studies describe are the same as your own (see later slide), and be sure that the positive outcomes described in the studies are the same as those in your logic model 19

Evidence Quality (Preliminary, Moderate, Strong) The reviewers will assess whether the study or studies: Are recent, preferably within the past 6 years Show a meaningful and significant positive effect on program participants Are of satisfactory methodological quality and rigor 20

Evidence Quality (Preliminary, Moderate, Strong) The criteria for whether the studies have satisfactory methodological quality and rigor are likely to be strict and technical in nature. For example, reviewers will look at: Sample size and the statistical power of the study to detect effect size, as demonstrated by a power analysis Internal and external validity, meaning data instruments and processes are described and justified Construction and composition of comparison groups (for impact evaluations), and the extent to which they have baseline equivalency The statistical analyses that are conducted on the data If possible, it could be helpful to have someone with research experience review the studies to make sure they meet these criteria 21

Using Studies of Other Organizations Programs One advantage in 2019 is that you do not necessarily need to provide evidence from your own program to reach a higher evidence tier you can also provide evidence from studies of other programs that are implementing the same intervention. For example, if you can find an RCT impact study of a tutoring program identical to yours, that evidence can potentially count just as much as if you had done your own experimental study. However, if you submit research from other organizations as evidence, you will need to show that you are implementing the exact same intervention as the one described in your research 22

What Counts as the Same Intervention? To use studies of another intervention as evidence, you must explicitly explain how your own program is the same in terms of: Characteristics of the beneficiary population (e.g., age, income, geography, urbanicity) Characteristics of the population delivering the intervention (e.g., age, education level, national service) Dosage and design of the intervention (e.g., frequency, duration, and type of activities) Context in which the intervention is delivered (e.g., where, when, among whom) Intended outcomes (e.g., changes in knowledge, attitudes, behavior, condition) 23

Where Can I Find Evidence about Other Similar Interventions? CNCS Evidence Exchange Other focus area-specific research clearinghouses, such as the U.S. Department of Education s What Works Clearinghouse Talking to an external evaluator or university professor Talking to other AmeriCorps grantees with similar programs 24

Case Studies Presentation Title 3/18/2025 25 ICF proprietary and confidential. Do not copy, distribute, or disclose. ICF proprietary and confidential. Do not copy, distribute, or disclose. ICF proprietary and confidential. Do not copy, distribute, or disclose.

Applicant A Applicant A uses AmeriCorps members to run both a tutoring program and a summer camp for low-income children in a rural area. They have found what they think is a high-quality RCT study of a similar tutoring program in a rural community in Oklahoma, but haven t been able to find or collect any evidence about the effectiveness of summer camps. What is the highest evidence tier that this applicant could reach? What do they need to provide in their application to reach this tier? 26

Applicant B Applicant B is applying for their second round of AmeriCorps funding, and they are proposing to use AmeriCorps members for an innovative program to serve food to the homeless. They have collected performance data for three years, and based on those data they have made significant changes to their program model. However, they have not conducted an evaluation of their program, and don t know of any similar programs anywhere else. What is the highest evidence tier that this applicant could reach? What do they need to provide in their application to reach this tier? 27

Applicant C Applicant C, which uses AmeriCorps members to remove invasive species from state parks, is planning to submit a report describing an evaluation that they conducted themselves. In their program logic model, the main intended outcome of their program is increased plant diversity in the parks where they work. Their evaluation measured plant diversity before and after the AmeriCorps members did their work, and found that it did increase. What is the highest evidence tier that this applicant could reach? What do they need to provide in their application to reach this tier? 28

Questions to Think About What evidence tier do you think you will qualify for? Is there any way you could find more evidence for any of your program s proposed interventions? Given your evidence tier, what information will you need to include in your application to get maximum points for evidence quality? What are likely to be the biggest challenges as you are writing the Evidence Base section of your application, and how can you address them? 29