Early-Phase Selection Criteria and Scoring for Education Innovation and Research (EIR)

EDUCATION INNOVATION AND RESEARCH (EIR) EARLY-PHASE SELECTION CRITERIA AND SCORING

Selection Criteria and Competitive Preference Priority Absolute Priority 2: STEM Criterion Points A. Quality of Project Design 40 B. Adequacy of Resources and Quality of the Management Plan 35 C. Quality of the Project Evaluation 25 Competitive Preference Priority 1: Computer Science 5 2

A. QUALITY OF THE PROJECT DESIGN (40 POINTS) EARLY-PHASE: STEM SLIDE 1 OF 2 1.The extent to which the goals, objectives, and outcomes to be achieved by the proposed project are clearly specified and measurable. (10 points) 2.The extent to which the design of the proposed project is appropriate to, and will successfully address, the needs of the target population or other identified needs. (10 points) 3

A. QUALITY OF THE PROJECT DESIGN (40 POINTS) EARLY-PHASE: STEM SLIDE 2 OF 2 3.The extent to which the design of the proposed project reflects up-to-date knowledge from research and effective practice. (10 points) 4.The potential contribution of the proposed project to increased knowledge or understanding of educational problems, issues, or effective strategies. (10 points) 4

B. ADEQUACY OF RESOURCES AND QUALITY OF THE MANAGEMENT PLAN (35 POINTS) EARLY-PHASE: STEM SLIDE 1 OF 2 1. The adequacy of the management plan to achieve the objectives of the proposed project on time and within budget, including clearly defined responsibilities, timelines, and milestones for accomplishing project tasks. (10 points) 2. The extent to which the costs are reasonable in relation to the objectives, design, and potential significance of the proposed project. (5 points) 5

B. ADEQUACY OF RESOURCES AND QUALITY OF THE MANAGEMENT PLAN (35 POINTS) EARLY-PHASE: STEM SLIDE 2 OF 2 3. The qualifications, including relevant training and experience, of key project personnel. (5 points) 4. The adequacy of procedures for ensuring feedback and continuous improvement in the operation of the proposed project. (10 points) 5. The extent to which the results of the proposed project are to be disseminated in ways that will enable others to use the information or strategies. (5 points) 6

MANAGEMENT PLAN ELEMENTS TO CONSIDER 1. Goal(s): A broad statement(s) of what the project intends to accomplish. What do you hope to accomplish by implementing your project? 2. Objective(s): A concrete attainment that can be achieved by following a number of steps. What is your project doing to support the overall program goal(s)? Are your objectives SMART (Specific, Measurable, Achievable, Relevant, and Time-bound)? 3. Performance Measures: A measurable or observable indicator to assess how well objectives are being met. How will you measure the success of your project? 4. Activities: Day to day pieces that must be completed to signal that the grant is on track. 5. Timeline (Start/End Dates): Provide some timeline that will allow task monitoring. 6. Responsible Personnel: Who will be carrying out those activities? 7

MANAGEMENT PLAN EXAMPLE Goals Objectives Measures Activities Start Date End Date Responsible Status Notes Personnel Goal 1: Increase involvement of Smith Elementary School families in their students education. Objective 1.1: Logins on the Smith Elementary School Online Parent Training System will increase 25% from baseline to the end of the grant. Performance Measure 1.1a: Parents reporting in an annual survey knowing about the Online Parent Training System. Activity 1.1.1: Administer parent survey to get baseline data. Activity 1.1.2: Create a pamphlet for parents that describes how to access and use the Parent Portal. Activity 1.1.3: Distribute pamphlet during school-wide events and parent-teacher conferences. Activity 1.1.4: Design a training for parents on using the Parent Portal Activity 1.1.5: Organize a focus group on the Parent Portal to gather parent feedback. 9/1/2016 9/15/2016 Evaluation Team Not Begun 3/1/2016 3/15/2016 Project Director Completed Performance Measure 1.1b: Number of logins per year. 9/15/2016 12/1/2016 Project Coordinator Not Begun 2/1/2016 3/1/2016 Project Director Completed 11/15/2016 12/1/2016 Evaluation Team Not Begun Activity 1.1.6: Deliver Parent Portal trainings. 9/15/2016 12/1/2016 Project Director & Project Coordinator In Progress Scheduled for 10/1 and 11/1. Activity 1.1.7: Administer parent survey. 5/1/2017 6/1/2017 Evaluation Team Not Begun Activity 1.1.8: Collect monthly reports on parent logins. 10/1/2016 12/1/2016 Data Director In Progress Objective 1.2: The percentage of students with parents regularly engaging with the school will increase by 5% every school year. Performance Measure 1.2a: Percentage of students that have at least 1 parent/guardian attend 1 parent-teacher conference per school year. Performance Measure 1.2b: Average parent attendance at school-wide events every school year. Activity 1.2.1 Activity 1.2.2 Activity 1.2.3 8

C. QUALITY OF THE PROJECT EVALUATION (25 POINTS) EARLY-PHASE: STEM 1. The extent to which the methods of evaluation will, if well implemented, produce evidence about the project's effectiveness that would meet the What Works Clearinghouse standards with or without reservations as described in the What Works Clearinghouse Handbook (as defined in this notice). (15 points) 2. The extent to which the evaluation plan clearly articulates the key project components, mediators, and outcomes, as well as a measurable threshold for acceptable implementation. (5 points) 3. The extent to which the methods of evaluation will provide valid and reliable performance data on relevant outcomes. (5 points) 9

Selection Criteria and Competitive Preference Priority Absolute Priority 3: Teacher-Directed Professional Learning Criterion A. Quality of Project Design Points 45 B. Adequacy of Resources and Quality of the Management Plan 30 C. Quality of the Project Evaluation 25 Competitive Preference Priority 2: SEA Partnership 5 10

A. QUALITY OF THE PROJECT DESIGN (45 POINTS) EARLY-PHASE: TEACHER-DIRECTED PROFESSIONAL LEARNING SLIDE 1 OF 2 1. The extent to which professional learning funded through the stipend will replace existing mandatory professional development for participating teachers at the following levels: (i) Replacing less than 20 percent of required professional learning. (0 points) (ii) Replacing 20 percent of required professional learning. (5 points) (iii) Replacing 40 percent of required professional learning. (10 points) (iv) Replacing 60 percent of required professional learning. (15 points) (v) Replacing 80 percent of required professional learning. (20 points) (vi) Replacing 100 percent of required professional learning. (25 points) 11

A. QUALITY OF THE PROJECT DESIGN (45 POINTS) EARLY-PHASE: TEACHER-DIRECTED PROFESSIONAL LEARNING SLIDE 2 OF 2 2. The adequacy of plans to ensure that stipends are appropriately used for high-quality professional learning. (5 points) 3. The extent to which the proposed project will offer teachers flexibility and autonomy regarding the extent of the choice teachers have in selecting their professional learning. (5 points) 4. The likelihood that the procedures and resources for teachers result in a simple process to select or request professional learning based on their professional learning needs and those identified needs of high-need students. (5 points) 5. The extent to which the goals, objectives, and outcomes to be achieved by the proposed project are clearly specified and measurable. (5 points) 12

B. ADEQUACY OF RESOURCES AND QUALITY OF THE MANAGEMENT PLAN (30 POINTS) EARLY-PHASE: TEACHER-DIRECTED PROFESSIONAL LEARNING SLIDE 1 OF 2 1. The sufficiency of the stipend amount to enable professional learning funded through the stipend to replace a significant portion of existing mandatory professional development for participating teachers. (5 points) 2. The extent to which the costs are reasonable in relation to the objectives, design, and potential significance of the proposed project. (5 points) 3. The extent to which the proposed payment structure will enable teachers to have an opportunity to apply for and use the stipend with minimal burden. (5 points) 13

B. ADEQUACY OF RESOURCES AND QUALITY OF THE MANAGEMENT PLAN (30 POINTS) EARLY-PHASE: TEACHER-DIRECTED PROFESSIONAL LEARNING SLIDE 2 OF 2 4. The qualifications, including relevant training and experience, of key project personnel. (5 points) 5. The adequacy of the management plan to achieve the objectives of the proposed project on time and within budget, including clearly defined responsibilities, timelines, and milestones for accomplishing project tasks. (5 points) 6. The adequacy of procedures for leveraging the stipend program to inform continuous improvement and systematic changes to professional learning. (5 points) 14

C. QUALITY OF THE PROJECT EVALUATION (25 POINTS) EARLY-PHASE: TEACHER-DIRECTED PROFESSIONAL LEARNING 1. The extent to which the methods of evaluation will, if well implemented, produce evidence about the project's effectiveness that would meet the What Works Clearinghouse standards with or without reservations as described in the What Works Clearinghouse Handbook (as defined in 34 CFR 77.1(c)). (15 points) 2. The extent to which the evaluation plan clearly articulates the key project components, mediators, and outcomes, as well as a measurable threshold for acceptable implementation. (5 points) 3. The extent to which the methods of evaluation will provide performance feedback and permit periodic assessment of progress toward achieving intended outcomes. (5 points) 15

EVALUATION EXPECTATIONS EARLY-PHASE Must be an independent evaluation. (EIR Program Requirement) Design is encouraged to have the potential to meet moderate evidence (as defined in the notice). First years of grant are encouraged to focus on developing and iterating the practice in a subset of schools. Later years are encouraged to include an efficacy study of the practice with full-scale implementation. 16

TECHNICAL ASSISTANCE RESOURCES ON EVALUATION What Works Clearinghouse Procedures and Standards Handbooks Technical Assistance Materials for Conducting Rigorous Impact Evaluations Institute of Education Sciences /National Center for Education Evaluation Technical Methods papers Webinar: Strategies for designing and executing well- designed quasi-experimental design studies Webinar: Strategies for designing and executing experimental studies that meet What Works Clearinghouse evidence standards without reservations 17

SUGGESTIONS FOR SELECTING AN EVALUATOR SLIDE 1 OF 2 Applicants might consider evaluators with the following Knowledge of What Works Clearinghouse standards Experience conducting evaluations using a variety of designs and methodologies Published journal articles Staff capacity (i.e., team of qualified individuals) Experience recruiting control sites Experience working with districts to gain appropriate consents and data sharing agreements 18

SUGGESTIONS FOR SELECTING AN EVALUATOR SLIDE 2 OF 2 Experience managing data records and protecting privacy Familiarity with the literature related to your project Shared understanding of the goals of the evaluation Contingency plans for adjusting the design and/or the budget if there are implementation challenges Ability to agree to the terms of the independent evaluation (e.g., the clearly defined responsibilities and deliverables) 19

AGENDA Introductions Early-phase Priorities Application Requirements Selection Criteria Additional Information 20

OVERVIEW OF EARLY-PHASE REVIEW PROCESS Applications sorted and placed in panels by Absolute Priority 2 (STEM) or Absolute Priority 3 (Teacher- Directed Professional Learning) Each application is scored against selection criteria Panel Review Reviewers draft comments and scores Panel discusses comments/scores Final scores are averaged There will be two separate rankings: one for Absolute Priority 2 and another for Absolute Priority 3 21

SCORING THE COMPETITIVE PREFERENCE PRIORITY Peer reviewers will determine how well the applicant addresses the competitive preference priority and will use their own judgment to determine how many of the 5 possible competitive preference points will be assigned. Peer reviewers will write a justification of their scoring on the technical review form. Applicants should make clear in their application abstract, narrative, and eligibility checklist whether they are addressing a competitive preference priority. Applicants who do not address the competitive preference priority are not eligible to receive any additional points. 22

TIPS FOR ORGANIZING YOUR APPLICATION We recommend that you organize and sequence your project narrative using the selection criteria. Within each criterion, make sure that you include a direct response to each factor. When appropriate, use language from the selection criteria to help guide reviewers For entities applying under Absolute Priority 3, refer to the crosswalk in the Application Instructions for suggestions of how to both address application requirements as well as selection criteria. 23

EDUCATION INNOVATION AND RESEARCH (EIR) EARLY-PHASE SELECTION CRITERIA AND SCORING