Program Evaluation Webinar #1 - August 27th, 2024
This webinar focuses on DESE's upcoming Federal Grant Evaluation support, Title grant program evaluation requirements, common evaluation challenges, and DESE's draft Title Grants Program Evaluation Workbook. It emphasizes the importance of annually evaluating Title grant programs as mandated by Federal law. Additionally, it highlights the upcoming webinars planned to provide targeted support for clarifying program evaluation requirements and assisting with FY24-25 evaluations.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Program Evaluation Webinar #1 August 27th, 2024 1
2 Introductions Leif Jacobsen DESE, Federal Programs Team Kendra Winner DESE, Research and Evaluation Coordinator 2
Objectives By the end of today s convening, participants will... 1. Be familiar with DESE s upcoming Federal Grant Evaluation support 2. Understand Federal Title IA, IIA, and IVA program evaluation requirements 3. Know about common evaluation challenges 4. Be familiar with DESE s draft Title Grants Program Evaluation Workbook, Steps 1-3 3
Why is this important? Federal law mandates that as a condition of accepting Title grant funding, a district must annually evaluate its Title grant programs. 4
Your Feedback and Upcoming DESE Support Your Feedback Our Response At the Federal Grants Conference, participants shared that: As a result, we are planning five webinars in the upcoming year: Most districts have some familiarity with program evaluation but are far from experts. Today: Overview of Program and Evaluation Goals, Evaluation Teams, and Data Collection Plans November: Theory of Action & Logic Models December: Metrics and Collecting and Analyzing Data February: Preparing Your Analysis May: Communication Plan Targeted support would be helpful to a) clarify the program evaluation requirements within the Title Grants, and b) assist with the development of FY24-25 evaluations. 5
Federal Title Grant Requirements - Title IA Two requirements: 1. Title IA (Submission 13): Title I Program Evaluation Procedure Learn more on the DESE Federal Grants website. 2. Title IA (Submission 14): Title I Program Evaluation Summary 6
Federal Title Grant Requirements Title IIA and IVA Title IIA Title IVA One requirement: Activities Evaluation One requirement: Activities Evaluation 7
Monitoring Follow-Up Required Action Only required for districts who received Program Evaluation findings during 2023-24 monitoring Two required work products: 1) written summary of your 2023-24 Title I Program Evaluation, and 2) Plan for 2024-25 Program Evaluation Deadline: October 11th, 2024 8
Evaluation 1. Common evaluation challenges 2. DESE s draft Title Grants Program Evaluation Workbook 9 9
Evaluation Challenges 1) Evaluation anxiety 2) Support from the top 3) Allocating time and energy 4) Going it alone 5) Trying to evaluate the unevaluable 6) You can t answer every question 10
Common Evaluation Challenges 1) Evaluation anxiety 2) Support from the top 3) Allocating time and energy 4) Establishing a team for the evaluation 5) Trying to evaluate the unevaluable 6) You can t answer every question 12
Title Grants Program Evaluation Workbook An outline and checklist of five steps to take The key questions each step is meant to answer Tools to support completing the steps 13
Evaluation Checklist Step Question Checklist 1: Define program and evaluation questions What did we design our program to do? What do we want to find out? How would we know? What is measurable and attainable? What skill(s) do we need on our team and who has those skills? 2: Identify members of the team How and when do we collect and analyze the data? 3: Create a data collection plan How do we analyze our findings? 4. Analyze data and summarize findings How do we report our findings and what do we do as a result of our findings? 5. Report, discuss, and apply evaluation findings 14
Step 1: Program Goals, Evaluation Questions and Measures What did you design your program to do? What do you what to know? How will you know? 15
Draft Evaluation Outline Program Goal(s): What did we design our program to do? Evaluation Question(s): What do we want to know? Outcome Measure(s): How will we know? We want our programming to improve literacy skills in Elementary Students We want to know if the literacy interventions and resources provided through Title I funding are working. We would know if there was an increase in the % of students meeting or exceeding grade-level reading benchmarks on standardized reading assessments. 16
Step 1: Define Your Title I Program Goals Column 1 Draft Evaluation Outline Program Goal(s): What did we design our program to do? We want our programming to improve literacy skills in Elementary students Discuss and answer these questions about your FY25 Title I grant: oYour district designed, chose programming, etc. for a reason, what was that reason? oWhat did you design your Title I program to do or impact? oWhat are one or two main goals of the grant funds? 17
Step 1: Evaluation Questions Column 2 Draft Evaluation Outline Evaluation Questions(s): What do we want to know? What we want to know if the literacy interventions and resources provided through Title I funding are working? Discuss and answer whether your evaluation question(s) help address the following: oIs our programming achieving our planned outcomes? oWhat are areas for improvement? oWhat decisions about the program can we make to moving forward to improve our planned outcomes? 18
Step 1: Identify Potential Measures Column 3 Draft Evaluation Outline Outcome Measure(s): How will we know? We would know if there was an increase in the % of students meeting or exceeding grade-level benchmarks on standard reading assessments Discuss and answer the following about potential measures What would you measure? Can you measure what you would like to measure? If you can t, what can you measure? Does your measure capture what it looks like when you achieve your goal? 19
Step 2: Identify Members of the Team Ensure you have the right leadership, administrative support Clarify what skills your team or other colleagues already have Identify skills to prioritize developing or recruiting 20
Step 2: Identify Members of the Team Name Position Relevant Responsibilities Skills --Understand the federal grants evaluation requirement, the skills, and time necessary to meet that requirement Ali Washington District Leader -- Ability to assign responsibilities and co- manage project with Grants Coordinator --Support the recruitment of staff with necessary skills to participate in the evaluation process --Ability to provide encouragement --Ensure participating staff have adequate time to contribute to the evaluation process --Ability to support other staff to participate in evaluation activities (e.g., participate in or administer surveys) --Participate in updates as needed 21
Step 3: Create a Data Collection Plan What data do we need to collect? How and when do we collect the data? 22
Step 3: Existing and New Data Existing Data: focus on existing data, because: There is a lot of existing data Creating and/or finding tools to conduct surveys, focus groups, or interviews that are valid and reliable is extremely challenging It can be difficult to get candid feedback when you re gathering information about work you have a stake in and when there are power differentials New Data: the workbook contains survey and focus group information, resources, and links 23
Step 3: Existing DESE Data Edwin Analytics: reporting and data analysis tool that integrates a host of data and includes longitudinal data. Getting Started in Edwin: includes directions to help you contact your district s Edwin Analytics contact for additional information at the top of the Getting Access and Support guide. Examples of data: Student outcome data (MCAS and ACCESS) Student Demographics Student growth Student Views on Climate and Learning, school climate and bullying information (VOCAL) 24
Step 3: Existing DESE Data District Analysis Review Tools (DARTs) DART for District and Schools: data trends for enrollment, assessment, finance and management, student support, leadership and governance and human resources DART Detail: English Learners: consolidates state data on English learners into a single tool DART Detail: Success After High School: data to provide and indication of the overall condition or district or school efforts to ensure all student are ready for their next steps after graduation. DART Detail: Staffing and Finance: district staff (e.g., teachers, special education staff), finance and expenditures by source of funds, per pupil expenditures, out of district costs, revenues 25
Upcoming DESE Data Trainings Tues September 3rd EWIS 101 9 10:30 AM This session is the very basics of the Early Warning Indicator System. If you are an experienced user, and don't want a refresher, this session is not for you. If you are a newer user, or want a refresher, feel free to jump aboard! Wed September 4th Using VOCAL Data and Holding Data Meetings 9 10:30 AM This session will focus on the use of VOCAL data with other data that you may have in district and their potential use in data meetings. Thurs September 5th Using EWIS with SEL in Mind 9 10:30 AM A favorite of many of our EWIS fans with John Crocker from Methuen Public Schools, considers using EWIS with mental health and other student data to move students towards academic and social success! 26
Step 3: School and District Data Examples: Student Characteristics Characteristics Characteristics Reading ability Subject-specific ability Specific special needs Specific ELL status Time in program Intensity of program Learning speed Student engagement School skills Extracurricular involvement Data Needed Years behind grade level on reading Entering grade level on the specific subject Type of disability; Severity of disability Level of English proficiency Semesters or years in the program Minutes per week spent in the program Growth rate in previous year (fall to spring growth) Homework completion rates Number of course fails Number of extracurricular groups; Participation rate in extracurricular groups Timing of grade level promotion Grade retention/promotion history For a dropout prevention program: (E.g. has a mentor with the Big Brother/Big Sister Mentorship Program), For a reading program: Student received support tutor Age relative to expected age for grade level Participation in related programs Age Attendance Behavior Parent engagement Transience/mobility Attendance rate, Tardy rate Detentions, Merits/Demerits, Has an IEP goal related to behavior Parent attendance at parent-teacher meetings, PTA, etc Number of times moved between schools or districts in the last year 27
Step 3: Data Collection Plan When and how to we collect data? The Data Collection Plan Tool walks you through: 1) Data Source(s) 2) Is this data available? 3) If yes, where is it housed? If no, how will you collect it? 4) Who collects the data? Who has access to the data? OR Who will collect and store the data? 5) When will is/will the data collected? 6) Who will analyze the data? 7) When will the data be analyzed? 8) Potential challenges/questions for the team to discuss 28
Step 3: Data Collection Plan Tool Timeline/Calendar Steps Metric 1 Metric 2 Data Source (or Metric) DIBELS Survey of Staff Is this data available? Yes No, we must collect If yes, where is it housed? If not, how will you collect it? Individual Schools have their DIBELS data, typically in Excel files Identify a survey tool and make sure it has a report function Who collects and has access to the data? OR Who will collect and store the data? District Literacy Coordinator Assistant Superintendent When is/will the data be collected? September 2024, December 2024, and May 2025 29
More Evaluation Support Coming! Webinars November: Theory of Action and Logic Models December: Metrics and Collecting and Analyzing Data February: Preparing your Analysis May: Communication Plan Workbook Additions and Revisions Please send us your feedback, questions, etc. 30
Next Steps For DESE Scheduling: November and Beyond Revising Evaluation Guidance For Grantees October 11th: monitoring required actions due Here s what you should do over the next couple of months to stay on track We re looking for partners to build out this work. If you re interested in getting some support email leif.jacobsen@mass.gov! 31
