Analysis of State School Accountability System in KY

Analysis of State School Accountability System In accordance with SB175 (2019 KY General Assembly) Dr. Wayne Lewis, Commissioner Rhonda Sims, Associate Commissioner November 20, 2019 1

What does SB 175 (2019) require? Requires the Commissioner of Education to convene a committee each year that shall analyze state assessment results and examine and consider the expected impacts, unintended consequences, and potential for all schools to reach the highest ratings in the state accountability system. The Kentucky Department of Education shall report to the Interim Joint Committee on Education by December 1, 2019 and by December 1, 2020 regarding the findings of each committee. 2 KDE: OSAA: IJCE NOVEMBER 2019

Statewide Representation 3 KDE: OSAA: IJCE NOVEMBER 2019

# # 1 1 2 2 Name Name Nick Carter Cornelius Faulkner Role Role Superintendent Superintendent Location Location Breckinridge County Caverna Independent Committee Members 3 3 4 4 5 5 6 6 7 7 8 8 Deanna Miller Jonathan Jett Erica Thompson Patty Johnson Kimberly White Tharon Hurley Superintendent Superintendent District Assessment Coordinator District Assessment Coordinator District Assessment Coordinator District Assessment Coordinator Fulton Independent Perry County Jefferson County Pike County Henderson County Laurel County 9 9 Shelli Wilson Monica Heavrin DeeAnna Crump Todd Neace Catrina Mcdermott Amy Tracy Kaysin Higgins District Assessment Coordinator Director of Special Education Director of English Learners Principal, Bath County Middle School K-12 Principal, Pineville Independent Principal, Lincoln County ATC Teacher Campbell County Grayson County Warren County Bath County Pineville Independent Lincoln County Calloway County 10 10 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18 Ron Jones Susan Readnower Jim Flynn Teacher Teacher Executive Director, Kentucky Association of School Superintendents Executive Director, Kentucky Association for Career and Technical Education Director of Political Affairs, Kentucky Chamber of Commerce Data and Policy Analyst, Prichard Vice President of Academic Affairs and Student Success, Council on Postsecondary Education Dean, School of Education, Campbellsville University President, Kentucky Community and Technical College System Chair and Professor, Department of Foundational and Graduate Studies, Morehead State University Jefferson County Harrodsburg ATC Statewide 19 19 Mike Stone Statewide 20 20 Travis Burton Statewide 21 21 22 22 Susan Weston Melissa Bell Statewide Statewide 23 23 Lisa Allen Statewide 24 24 Jay Box Statewide 25 25 Timothy Simpson Statewide 26 26 27 27 28 28 Charles Aull Autumn Neagle Adrian Wallace Public Policy Manager, Greater Louisville Inc. Parent, Kentucky PTA Parent, Kentucky PTA Jefferson County Statewide Statewide 4 KDE: OSAA: IJCE NOVEMBER 2019

Committee to Analyze and Review Assessment Results and Accountability System A diverse stakeholder group met October 22 in Frankfort. Analyze state assessment results and examine and consider the: Expected Impacts; Unintended Consequences; and Potential for All Schools to Reach the Highest Ratings in the State Accountability System. 5 KDE: OSAA: IJCE NOVEMBER 2019

Survey of Committee Members Committee members were surveyed prior to meeting. Role on committee (i.e., superintendent, district assessment coordinator, teacher, post-secondary representative, etc.) What are the most positive aspects of the 5-star accountability system? What component(s) of the accountability system would be more effective with revision? Do you believe implementation of the accountability system has resulted in unintended consequences? If you answered yes to the previous question, please describe the unintended consequences. 6 KDE: OSAA: IJCE NOVEMBER 2019

Committee to Analyze and Review Assessment Results and Accountability System Topics were identified for discussion at the meeting: Overall Accountability System Equity: Reporting Equity: How Measured and Evaluated Growth of English Learner (EL) Students Student Growth in Reading and Mathematics Transition Readiness Committee prioritized potential changes and made recommendations. 7 KDE:OSAA: SB 175 IJCE 11.20.19

Overview of the Current System 8 KDE: OSAA: IJCE NOVEMBER 2019

Indicators Elementary and Middle Schools Indicator Proficiency Measures Reading and mathematics tests Metric (Elementary and Middle Schools) Index score (0-125) N=0, A=.50, P=1.0, D=1.25 Each score from reading and mathematics weighted equally Index score (0-125) N=0, A=.50, P=1.0, D=1.25 Each score from science, social studies and writing weighted equally Growth score (0 - 300) Value table that assigns points for individual student growth Each score from reading (including ELP) and mathematics weighted equally Separate Academic Science, social studies and writing tests Growth Reading and mathematics tests, English language proficiency (ELP) tests 9 KDE: OSAA: IJCE NOVEMBER 2019

High Schools Indicator Proficiency Measures Metric Reading and mathematics sections from ACT (weighted equally) Index score (0-125) N=0, A=.50, P=1.0, D=1.25 (ACT levels) Separate Academic Transition Readiness Science and writing tests (weighted equally) Academic readiness, (ACT, college placement, dual credit, AP, IB, CAI, ELP) career readiness (industry cert., CTE EOP exam, exceptional work experience, apprenticeship) ELP attainment (ACCESS) Index score (0-125) N=0, A=.50, P=1.0, D=1.25 Transition readiness rate (0-125) Percentage of graduates who have a high school diploma and also demonstrate academic or career readiness ELs anytime throughout high school Graduation Four- and five-year graduation rates Percentage of grade 9 students (adjusted) who graduated in four or five years Average of four- and five-year rates (0-100) 10 KDE: OSAA: IJCE NOVEMBER 2019

Overall Accountability Weights Proficiency Proficiency (Reading and (Reading and Mathematics) Mathematics) Separate Separate Academic Academic (Science, (Science, Social Social Studies Studies and and Writing) Writing) Growth Growth Quality of Quality of School School Climate Climate and and Safety* Safety* Transition Transition Readiness Readiness (including (including ELs) ELs) Graduation Graduation (Four (Four- - and Five Five- -Year Year Cohort) Cohort) (including (including ELs) ELs) and Elementary/ Elementary/ Middle Middle Schools Schools 35 26 35 4 -- -- High High Schools Schools 45 15 -- 4 30 6 *Quality of School Climate and Safety is scheduled to enter the accountability system in 2019-2020. 11 KDE:OSAA: SB 175 IJCE 11.20.19

Overview of Assessment and Accountability Results 12 KDE: OSAA: IJCE NOVEMBER 2019

2018-2019 Reporting Schools, Districts and State received: Star Ratings (1 to 5 stars) Overall Accountability Score Indicator Scores and Labels of Very High to Very Low for Proficiency, Growth (EL/MS), Separate Academic (EL/MS/HS), Transition Readiness (HS) and Graduation (HS) Federal Classifications (schools only) 13 KDE: OSAA: IJCE NOVEMBER 2019

2018-2019 School Ratings 14 KDE:OSAA: SB 175 IJCE 11.20.19

Federal Classifications for Kentucky 2018-2019 Comprehensive Support and Improvement (CSI) 50 schools designated Performance at the bottom 5% by level Additional Targeted Support and Improvement (ATSI) 11 schools identified Initially identified for TSI-Tier II last year and failed to met exit criteria this year Focus on low student group performance 15 KDE: OSAA: IJCE NOVEMBER 2019

Updated Data Review After the public release of accountability data, there is a 10-day period for districts to request individual student changes. Based on individual change requests and allegations, there are changes to the star ratings and federal classification: 2 schools increased a star rating ; 2 districts increased a star rating; 1 school and 1 district decreased a star rating ; and 1 school removed from ATSI identification . 16 KDE: OSAA: IJCE NOVEMBER 2019

Prior State Law on Targeted Support and Improvement Previously, KRS 160.346(2) required a school to be identified for Targeted Support and Improvement (TSI) if the school had: Tier II: One or more subgroups performing as poorly as all students in any of the lowest performing five percent of schools by level (identified annually beginning in 2018-2019) or Tier I: One or more subgroups performing as poorly as all students in any of the lowest performing 10 percent of schools by level for two consecutive years (identified annually beginning in 2020-2021). 418 schools were identified in the fall of 2018 for Tier II TSI. During the 2019 legislative session, the Kentucky General Assembly passed Senate Bill 175 to further refine Kentucky s implementation of ESSA, particularly as it relates to the development of standards and assessments, postsecondary readiness, and the identification of schools for TSI.

Terminology Changes in Identifying TSI Schools SB 175 (2019) also introduced new terminology that better aligns with ESSA: Additional Targeted Support and Improvement (ATSI). Prior to the 2019 legislative session, ATSI was not used in state law, which instead offered two different tiers of the TSI label. All 418 schools identified TSI in the fall of 2018 met the requirements and expectations under the Every Student Succeeds Act (ESSA) for ATSI and were reported to ED as ATSI schools classified pursuant to ESSA Section 1111(d)(2)(C)-(D). Beginning with reporting this fall, KDE will adopt this terminology in the School Report Card. Use of the terminology ATSI is a technical change to align with federal and state law and is not, nor should it be construed as, a substantive change to any school s designation. A change to the definition of TSI in the statute is needed.

Amendments to Kentuckys Consolidated State Plan On May 31, 2019, the changes SB175 (2019) made to TSI identification criteria were submitted for approval to the United States Department of Education (ED) as part of Kentucky s Consolidated State Plan (State Plan). On September 6th, feedback from ED revealed that the State Plan would not receive federal approval if TSI identification continued to follow the guidelines established in SB 175 (2019). Negotiating ED s approval of the State Plan was paramount to the department. Failing to have an approved State Plan puts federal funding Kentucky school districts receive under ESSA in jeopardy, which is a cost far too high for KDE to risk. On September 23th, KDE received notice that Kentucky s State Plan had been approved by ED.

Discussion of Committee and Findings 20 KDE: OSAA: IJCE NOVEMBER 2019

Expected Impact The assessment and accountability systems were reported as negotiated via Kentucky s State Plan. Administration and reporting were completed well without any major incidents. Schools performance on multiple measures were reported, and overall performance reported in a simple 5-star rating. There was a range of scores, with most in the middle (3- star). Schools and districts received detailed data to help them identify areas of strength and where they could improve. 21 KDE: OSAA: IJCE NOVEMBER 2019

Expected Impact (continued) Lower-performing schools were identified to receive state and district support. Statewide analysis showed that in general: Some schools performed well, even with challenging circumstances. Achievement is lower than desired and has not improved much on most indicators. There are large achievement gaps between groups. 22 KDE: OSAA: IJCE NOVEMBER 2019

Unintended Consequences The committee agreed that it was very important to monitor for unintended consequences, both positive and negative. One concern is the complexity of the accountability system and some people may misinterpret the results. The committee spent the majority of its time discussing how some definitions and rules in the current accountability system might not produce the expected results, and therefore might need to be modified or otherwise improved. 23 KDE: OSAA: IJCE NOVEMBER 2019

Transition Readiness In negotiation with USED, the ESSA State Plan will need to be amended for the 2019-2020 school accountability. The calculation will need to include 12th grade non-graduates. The amendment will be revised as follows: Transition readiness shall be calculated by dividing the number of high school graduates plus 12th grade non-graduates who have met measures of transition readiness plus the number of English learners who have achieved English language proficiency by the total number of graduates plus 12th grade non-graduates plus the number of graduates who have received English language services during high school. 24 KDE: OSAA: IJCE NOVEMBER 2019

Growth of English Learner (EL) Students Much discussion occurred on the inclusion of English Learners (ELs) into state accountability. ELs are included in the growth and transition readiness indicators twice. The including ELs twice is a requirement of ESSA. Another option would be to remove EL progress from Transition Readiness and include a separate EL indicator to measure English Language proficiency. The committee was not receptive to having a separate indicator. Including ELs in growth will remain unchanged. KDE: OSAA: IJCE NOVEMBER 2019 25

Potential for All Schools to Reach the Highest Rating The committee agreed that the accountability system was designed so that logically there was the potential that all schools could reach the highest rating (5-stars). One question raised by the committee was whether schools with very high percentages of their students scoring Proficient or Distinguished could achieve a 5-star rating. 26 KDE: OSAA: IJCE NOVEMBER 2019

Potential for All Schools to Reach the Highest Rating (continued) The Growth Value Table was discussed at length. Schools cannot earn as many points in Growth as can schools whose students are growing from lower levels of Novice or Apprentice. Mathematically if there were an elementary or middle school with all its students Proficient on average, and all the students maintained Proficiency from year to year, that school would earn a 5-star rating. KDE does not recommend changes to the current Growth Value Table. 27 KDE: OSAA: IJCE NOVEMBER 2019

Potential for All Schools to Reach the Highest Rating (continued) Achievement gap identification was discussed extensively. Specifically, the rule that if a school had a statistically significant achievement gap for at least one student group in the school, that school would be lowered one-star rating if it otherwise would have been designated a 5- or 4-star rating. Some committee members thought more schools should be identified as having an achievement gap significant. Some committee members thought fewer schools should be identified as having an achievement gap significant. 28 KDE: OSAA: IJCE NOVEMBER 2019

Achievement Gaps Achievement gaps are determined through statistical and practical differences. A Cohen s d is used to determine statistical and practical significance. Cohen s d provides a measure of effect size for comparisons of groups with differing sizes and variability as seen in student groups across the state. This statistical test of significance is used to determine if achievement gaps are significant or not. 29 KDE: OSAA: IJCE NOVEMBER 2019

Achievement Gaps (continued) Cohen recommended the effect size and corresponding d value shown below. To ensure gaps were statistically significant, the 1.0 level was used to determine if the achievement gaps to reduce a star rating and reporting. 30 KDE: OSAA: IJCE NOVEMBER 2019

Achievement Gaps (continued) An examination of the data for the 2018-2019 rating showed that over 75% of the schools at 4- or 5-star levels did not have a significant achievement gap. Further analysis showed that the schools achieved high ratings while being accountable for student groups of students with IEPs, English learners, economically disadvantaged students, and every racial/ethnic student group. KDE will enhance reporting of achievement gaps. 31 KDE: OSAA: IJCE NOVEMBER 2019

Next Steps Advisory Committee Feedback School Curriculum, Assessment and Accountability Council, Local Superintendent Advisory Committee, other Advisory Committees Amendments to 703 KAR 5:270 Regulatory Process Proposed changes discussed with the Kentucky Board of Education First Reading-December Second Reading-February Public Comment Period (60 days) Statement of Consideration (Late Spring/Early Summer) Legislative Committees (Summer 2020) 32 KDE: OSAA: IJCE NOVEMBER 2019