Program Planning and Evaluation for Migrant Education Directors

Module 8: Program Planning Migrant Education Program Evaluation New Directors Orientation Tutorial

Table of Contents Section Section Slide # Slide # Getting Started 3 What is Required 9 Performance Results to be Included in the Evaluation 22 Program Evaluation as Part of the Continuous Improvement Cycle 30 Setting the Context for the Migrant Education Program Evaluation 33 The Program Evaluation Plan 41 Conducting the Evaluation 58 The Written Evaluation Report 75 Wrapping Up 84 2

Getting Started In This Section In This Section Tutorial Objectives How to Use the Tutorial Icons to Guide You Key Readings and Resources 3

Tutorial Objectives Module 8 will enable state directors to 1. understand the legislative and regulatory requirements for the Migrant Education Program (MEP) Evaluation, 2. understand the role of program evaluation as part of a continuous improvement cycle, 3. develop a program evaluation plan that aligns with Measurable Program Outcomes (MPOs) and state performance targets/Annual Measurable Objectives (AMOs), 4. ensure that Priority for Services students (PFS) are featured, 5. develop implications for program improvement from evaluation results, and 6. develop an action plan for the next MEP Evaluation. 4

How to Use the Tutorial For optimal benefit from the tutorial, you should allow sufficient time to read the slides, reflect on the information, and complete all activities on the slides or on the Quick Resource and Reflection Sheets (QRRS) that can be downloaded as worksheets; read each slide as well as the information referenced in the slides; engage with the What Do You Think? slides to facilitate interaction with the information (Answers will be provided directly following each of these slides.); 5

How to Use the Tutorial For optimal benefit from the tutorial, you should (continued) pause to reflect on your state program at the Check-in slides (A QRRS document will typically accompany these.); complete the Pop Quiz! slides to reinforce key concepts; review your state s MEP documents and reports as directed; develop an action plan using the worksheets provided; add actionable items to your MEP planning calendar (See QRRS 14.2); and contact your OME Program Officer for follow-up questions. 6

Icons to Guide You The following icons will guide you in making the best use of this tutorial: What Do You Think? Check-in Pop Quiz! Quick Reference and Reflection Sheet (QRRS) Action Planning Calendar Item 7

Key Readings and Resources You should have these documents readily available while completing the module, as the module will refer to these documents for more complete information on various topics. MEP Guidance on the Education of Migratory Children under Title I, Part C of the Elementary and Secondary Education Act of 1965, Chapter VIII Migrant Education Program Evaluation Toolkit developed by the Office of Migrant Education (OME) Your state s MEP Service Delivery Plan (which should include the program evaluation plan) The most recent state MEP Evaluation Report 8

What is Required In This Section In This Section Basic Requirements What is Measured Involvement of Local Operating Agencies (LOAs) in the Evaluation Connection to Program Improvement 9

Basic Requirements States must plan, implement, and evaluate ensure that the state and local operating agencies (LOAs) address the special educational needs of migratory children, including preschool migratory children. evaluate programs and projects that Section 1304(b)(1) of the ESEA, as amended 10

Basic Requirements The state must develop and update a written comprehensive state plan that, at a minimum, has the following components: 1. Performance targets, 2. Needs assessment, 3. Measurable program outcomes, 4. Service delivery, and 5. 5. Evaluation Evaluation. 34 CFR 200.83 11

Basic Requirements Each state education agency (SEA) must determine the effectiveness of the MEP through a written evaluation of the program that measures the implementation and results of the program against the State's performance targets, particularly for those students who have priority for service (PFS). 34 CFR 200.84 12

Basic Requirements PFS children are those: 1. Who are failing, or most at risk of failing, to meet the states challenging state academic content and student achievement standards; and 2. Whose education has been interrupted during the regular school year. Section1304(d) of the ESEA 13

What is Measured SEAs are required to evaluate the effectiveness of the MEP in terms of: 1. Program implementation and 2. Program results. 34 CFR 200.84 14

What is Measured Implementation and results of the MEP are measured against performance targets/annual measurable objectives (AMOs) that the state has established For all children in reading and mathematics achievement, high school graduation, the number of school dropouts, and school readiness (if any) , and Any other performance targets that the state has identified for migrant children. 34 CFR 200.84 15

What is Measured Thus, implementation and results of the MEP should be measured against the: Measurable program outcomes (MPOs) the state has established for the MEP as part of its comprehensive Service Delivery Plan (SDP). Those MPOs are what the state MEP will produce to meet the needs of migratory children and help them achieve the state s performance targets/AMOs. 34 CFR 200.83(a)(3) 16

Involvement of Local Operating Agencies in the Evaluation In evaluating the program, the SEA will need LOAs to collect data on migratory children who receive services from the program (implementation component) and the outcomes of these services (using the program s measurable outcomes, as specified in the SDP). Evaluations of the program at the local level will also assist the state in its subgranting process (see Module 4). 17

Involvement of Local Operating Agencies in the Evaluation The SEA must ensure that the LOA conducts the local evaluation properly, and The SEA should inform its LOAs in advance of any specific data that it will need to evaluate the statewide program, and how the LOAs should collect the data. 18

Connection to Program Improvement SEAs and LOAs must use the results of the MEP evaluation to improve services for migrant children. 34 CFR 200.84 and 200.85 19

Pop Quiz! Instructions: Note True or False for each of the following statements related to the requirements for the MEP evaluation. Statement T F 1. The MEP evaluation collects and analyzes state-level data only. 2. The focus of the evaluation is on program results, not implementation. 3. The MEP evaluates instructional services and educational support services. 4. The MEP evaluation should include the evaluation of services for preschool migratory children. 20

Pop Quiz! - Response Number 1 is FALSE local and state levels. FALSE. . The MEP evaluation includes data from both the Number 2 is FALSE implementation of the MEP. FALSE. . The evaluation focuses on the results and Number 3 is TRUE educational support services that enable migrant children to participate effectively in school. TRUE. . The MEP evaluates both instructional services and Number 4 is TRUE special educational needs of migratory children, including preschool migratory children. TRUE. . MEP programs and projects must address the 21

Performance Results to be Included in the Evaluation In This Section In This Section Performance Goals, Indicators, and Targets Government Performance and Results Act (GPRA) Measures 22

Performance Goals, Indicators, and Targets Familiarity with the following key terms will assist you in understanding MEP evaluation requirements for reporting performance results: Performance goals, Performance indicators, Performance targets, and Measurable program outcomes (MPOs). See QRRS 8.1 Reviewing Key Terms for Performance Assessment 23

Performance Goals, Indicators, and Targets For purposes of program design and evaluation, the MEP will focus on ESEA Performance Goals 1 and 5, along with the indicators for each goal. MEP Guidance, Chapter VIII, B3 24

State Performance Goals, Indicators, and Targets Performance Goals and Performance Indicators that Performance Goals and Performance Indicators that the State s MEP Evaluation Address Address the State s MEP EvaluationMust Must Performance Goal 1 Performance Goal 1: By 2013-2014, all students will reach high standards, at a minimum attaining proficiency or better, in reading/language arts and math. 1.1 Performance indicator: The percentage of students, in the aggregate and for each subgroup, who are at or above the proficient level in reading/language arts on the state s assessment 1.2 Performance indicator: The percentage of students, in the aggregate and in each subgroup, who are at or above the proficient level in math or the state s assessment Performance Goal 5: Performance Goal 5: All students will graduate from high school. 5.1 Performance Indicator: The percentage of students who graduate from high school each year with a regular diploma 5.2 Performance Indicator: The percentage of students who drop out of school 25

Performance Goals, Indicators, and Targets For Performance Goals 1 and 5, the U.S. Department of Education required all SEAs to submit performance targets in their consolidated state applications for each performance indicator and baseline data for the targets. States are not required to resubmit their performance targets during the period for which the ESEA, as currently enacted is authorized unless the state makes a significant change in one or more of them. (See Module 3). o Several states have revised their performance targets via application for a waiver of certain ESEA requirements (also known as ESEA Flexibility ). These revised targets are also referred to as revised Annual Measurable Objectives (AMOs). 26

Check-in Review your most recent MEP Evaluation report and respond to the following questions: To what extent are migrant students reaching the state s performance targets under ESEA Goals 1 and 5, or any other state performance targets included in the MEP evaluation? Does the evaluation show the performance of priority for services (PFS) students relative to these targets? See QRRS 8.2 State Performance Targets for ESEA Goals 1 and 5 27

Government Performance and Results Act (GPRA) Measures In compliance with the Government Performance and Results Act (GPRA), ED/OME has adopted four GPRA measures for monitoring progress and maintaining accountability for the MEP on a national level. The GPRA measures present a national picture of the program, to which each state contributes. Therefore, GPRA measures should be part of the MEP evaluation. 28

Government Performance and Results Act (GPRA) Measures GPRA Measures: 1. Percentage of MEP students who scored at or above proficient on their state s annual reading/language arts assessments in grades 3- 8 and high school. 2. Percentage of MEP students who scored at or above proficient on their state s annual mathematics assessments in grades 3-8 and high school. 3. Percentage of MEP students who were enrolled in grades 7-12 and graduated or were promoted to the next grade level. 4. Percentage of MEP students who entered 11th grade who had received full credit for Algebra I or a higher mathematics course. 29

Program Evaluation as Part of the Continuous Program Improvement Cycle In This Section In This Section Continuous Improvement Cycle Making Connections in the Planning Process 30

Continuous Improvement Cycle 31

Making Connections in the Planning Process Program planning involves a continuous cycle of needs assessment, planning services, implementation, and evaluation. The MEP evaluation determines the degree to which the services identified in the Service Delivery Plan (SDP) (1) are implemented as planned, (2) truly meet the needs identified in the needs assessment, and (3) result in improved performance of migrant students as measured against the state s performance targets/annual measurable objectives (AMOs) and the GPRA measures. The evaluation will inform updates of the needs assessment and changes in service delivery to improve the state MEP and services it provides to migrant children. 32

Setting the Context for the MEP Evaluation In This Section In This Section Purpose of the MEP Evaluation Evaluation for Implementation and Results Responsibilities of Local Operating Agencies for the Migrant Education Program Evaluation Frequency of Evaluation 33

Purpose of the Migrant Education Program Evaluation Program evaluation involves systematically collecting information about a program, or some aspect of a program, in order to determine the effectiveness of the program and to improve it. MEP Guidance, Chapter VIII, A1 34

Purpose of the Migrant Education Program Evaluation The MEP evaluation allows SEAs and LOAs to: 1. Determine whether the program is effective and document impact on migrant children, 2. Improve program planning, 3. Determine the degree to which services are implemented as planned and identify implementation problems, and 4. Identify areas in which migrant children may need different MEP services. MEP Guidance, Chapter VIII 35

Evaluation for Implementation and Results States should develop methods of disaggregating state assessment data and data on measurable outcomes in order to determine the impact of the MEP on all migrant children and in particular those who have a priority for services (PFS). Data Sets Data on all migrant students Data on priority for services (PFS) students MEP Guidance, Chapter VII, C8 36

Responsibilities of Local Operating Agencies for the Migrant Education Program Evaluation So that its evaluation of the MEP is statewide, each SEA must ensure that its LOAs, as needed, conduct a local project evaluation that measures the implementation of the project and student performance against measurable outcomes (which are aligned to the state MEP s measurable outcomes, as specified in the service delivery plan). MEP Guidance, Chapter VIII, C3 37

Frequency of Evaluation Evaluation of Overall Implementation of the State MEP Evaluation of Overall Implementation of the State MEP An SEA should conduct an evaluation of the implementation of MEP services on a 2-3 year cycle to determine whether any improvements are needed. Evaluation of Program Results Evaluation of Program Results SEAs and LOAs should evaluate the results of the program (e.g., the degree to which the program has met the measurable state and local outcomes) on an annual basis. MEP Guidance, Chapter VIII, C5 38

What Do You Think? Why do you think there is a difference in the recommended time frame for evaluating program implementation and program results? 39

What Do You Think? - Reflection Did your response address the following points? Overall Program Implementation Evaluation 2-3 year cycle o Implementation of new programs or strategies usually takes two to three years to get staff up to speed, adapt to the context, and make mid-course adjustments for full implementation. Evaluation of Program Results annually o A results-based evaluation is necessary for monitoring student progress toward established goals; information that is collected annually will help determine if academic progress is occurring and will lead to exploration of interventions to foster progress in the cycle of continuous program improvement. 40

The Program Evaluation Plan In This Section In This Section Measurable Program Outcomes (MPOs) Evaluation Questions The Data Collection Plan The Data Collection Task and Timeline The Evaluation Matrix 41

Measurable Program Outcomes (MPOs) The MEP evaluation builds on the MPOs developed in the process of developing the SDP. (See Module 7.) SDP SDP MPOs are the backbone of the SDP and MEP evaluation. MPOs MPOs Evaluation Evaluation 42

Measurable Program Outcomes Strong MPOs define: What services will be provided, What is expected to happen as a result of the MEP services, Which students will directly benefit from the services, and Time frame for the service. 43

Pop Quiz! To what extent to you think the following is a strong MPO? Migrant children in grades 3-5, whose parents attend two reading and homework support workshops during SY 2012-2013, will increase their performance on the state reading assessment by at least 10%. 44

Pop Quiz! - Response You probably noticed that the sample MPO included each of the components of a strong MPO: Migrant children in grades 3-5 [which students], two reading and homework support workshops [what services will be provided] provided] during SY 2012-2013 [time frame], [time frame], will increase their performance on the state reading assessment by at least 10% [what will happen as a result of the services] will happen as a result of the services]. [which students], whose parents attend [what services will be [what 45

Evaluation Questions Strong MPOs lead to evaluation questions that can: o Measure the effectiveness of the strategies included in the SDP and o Maintain alignment with state performance goals, indicators, and targets. A Program Alignment Chart is a useful tool to show the connection between all parts of the service delivery plan, culminating in evaluation questions that will show the effectiveness of the program. See QRRS 8.3 Example of a Program Alignment Chart, also included in the Service Delivery Plan Toolkit, Section D.5. 46

Evaluation Questions Review the Alignment Chart (QRRS 8.3). Identify the following: 1. The state performance target related to reading and language proficiency, 2. The concern regarding the barrier to migrant students meeting the state performance target, 3. Strategies selected to address this concern, 4. MPOs for the strategies and, 5. Evaluation questions. 47

Evaluation Questions In reviewing the Alignment Chart in QRRS 8.3, note: The evaluation questions directly relate to the MPOs; they are designed to determine to what extent each MPO s depiction of program success was achieved. The MPOs relate to the proposed strategies and are measurable ways to determine if the strategies produced the desired results and/or were implemented as described. The strategies address the needs identified in the Need Statements, which were developed based on data to support the concern about barriers to migrant students attaining the state performance target in reading/language arts. 48

What Do You Think? The Alignment Chart in QRRS 8.3 contains several evaluation questions for each of the MPOs. What are three benefits of having several evaluation questions related to each MPO? 1. 2. 3. 49

What Do You Think? - Reflection Did your responses include any of the following reasons for multiple evaluation questions for each MPO? Can capture possible impacts on results, such as student attrition, or skill level upon entering the program. Can capture possible impacts on implementation, such as teacher level of experience, type of instruction, or frequency of supplemental instruction. Can triangulate results for greater validity. Can raise questions for further examination when discrepancies in data exist. 50