Training Session on Assessment Reporting and Accountability in Education

Assessment Reporting (Editing and Checking Assessment Data, Third Period Cumulative ADM, and English Learners) May 16, 2024 Training - ODE District site (to participate, get handouts, and view recorded webinars) Welcome! Oregon Department of Education 1

Contacts Regional ESD Partners Accountability Reporting Team Jon Wiens, Manager 503-931-7682 Cindy Barrick, Research Analyst 971-208-0480 Bianca Llamas, Research Analyst 971-208-0437 Elyse Bean, Research Analyst 971-720-8659 Oregon Department of Education 2

Todays Agenda 1 Assessments and Accountability Assessment Data Reviewing and Editing Oregon Department of Education 3

Assessments and Accountability 1 ESSA, School Profiles, and Other Assessment Accountability Reports Timelines Links for Information Oregon Department of Education 4

ESSA and School Profiles Every Student Succeeds Act (ESSA) ESSA maintains annual testing in English language arts and mathematics (grades 3-8 and 11), and science (grades 5, 8, and 11). ESSA maintains report cards and requires the inclusion of English Learner progress and at least one measure of school quality/student success as indicators to display on report cards. ESSA requires that the denominator for the achievement calculations includes at least 95% of students enrolled at the school, or the number of students participating, whichever is higher. ESSA Amendment 2023 Oregon Department of Education 5

School and District At-A-Glance Profiles Displays: Information on students and staff Class Size and Regular Attenders Statewide assessments Ninth Grade On-Track, Graduation, and Completers Local information on school programs, parent/community engagement TENTATIVE availability date: July 25, 2024 (subject to change) More details in the webinar in July Oregon Department of Education 6

Accountability Details Elementary & Middle Indicator Levels for all students and 11 student groups ELA Achievement, Average Gap Score Change, and Participation Math Achievement, Average Gap Score Change, and Participation Regular Attenders On Track to English Language Proficiency (ELP) TENTATIVE availability date: July 18, 2024 (subject to change) More details in the June webinar Oregon Department of Education 7

Accountability Details High School Indicator Levels for all students and 11 student groups ELA Achievement and Participation Math Achievement and Participation Regular Attenders On Track to English Language Proficiency (ELP) Ninth Grade On-Track Four-year Cohort Graduation Rate Five-year Completers Rate Oregon Department of Education 8

Other Accountability Reports Other Reports using Assessment Data (outside of ESSA) Public Assessment Group Reports for current and previous years Title III English Learners (formerly AMAO) Special Education At-A-Glance Profiles for Districts Federal EDFacts Reports Assessment Participation (C185, C188, C189) Academic Achievement (C175, C178, C179) Title III Former LEP (C126) ELPA Participation and Proficiency (C138, C139, C205) Indicator Status (C200, C201) Oregon Department of Education 9

Assessments and Accountability 2 ESSA, School Profiles, and Other Assessment Accountability Reports Timelines Links for Information Oregon Department of Education 10

Timelines for Reports Student and Classroom Reports (in Secure Assessment Reports 2.0 application) ELA, Math, Science available since February 8, 2024 ELPA Summative Batch 1 available March 11; Batch 2 available April 3; Batch 3 available May 7; Batch 4 (including remote administration) will be available May 31 Oregon Extended Assessments will be available June 28 Public Test Results (ELA, Math, Science) Validation Window (in ADI) August 15-23, 2024 Final 2023-24 Test Results Posted mid-September Oregon Department of Education 11

Timelines for Reports contd At-A-Glance School and District Profiles, Accountability Details Accountability Details Validation Window: July 18 September 30, 2024 At-A-Glance Profiles Validation Window: July 25 September 30, 2024 Final reports posted for district preview: October 3, 2024 Final reports posted on the public website: TBA Oregon Department of Education 12

Timelines for Data Sources Institutions for Accountability Reporting Validation Window in ADI: April 11 May 31, 2024 Boundary Calculators for 2023-24 were due to ODE: May 1, 2024 (Institution changes for 2024-25 due to ODE: September 15, 2024) Third Period Cumulative ADM Collection Data Collection Window: April 25 May 31, 2024 Audit Review Window: June 6 - 14, 2024 Validation Window in ADI: June 6 August 23, 2024 Title III English Learners Collection Spring Data Collection Window: April 11 May 24, 2024 Review Window: June 6 - 21, 2024 Oregon Department of Education 13

Timeline for Assessments See slides 30-32 for Assessments Load Schedule Non-participation (virtual) records generated based on Spring Membership starting June 4 (editable in the ARUA) Demographic and Program fields filled from Third Period Cumulative ADM (Spring Membership) after May 31 Refreshed and available weekly through the summer (same schedule as for Accountability Details refresh) Oregon Department of Education 14

Timelines for Final Edits Last date for edits to enrollment and assessment records: August 23, 2024 The ARUA closes August 23 at 5 PM Third Period Cumulative ADM closes May 31 at 11:59 PM; reopened during audit window (June 6-14) and upon request to Regional ESD Partner or Amanda Leopard; final editing closes August 23 @ 11:59 PM Oregon Department of Education 15

Assessments and Accountability 3 ESSA, School Profiles, and Other Assessment Accountability Reports Timelines Links for Information Oregon Department of Education 16

Links for Webpages Quick Links to Accountability Information Assessment & Accountability Checklist Every Student Succeeds Act (ESSA) Training (on the ODE District secure website) Oregon Department of Education 17

Links for Assessment Help Documents, tools, guidance Quick Links to Accountability Information OSAS Online User Guides Contacts Regional ESD Partners Application assistance: ODE Helpdesk Accountability Reporting: Elyse Bean Oregon Department of Education 18

Questions? 1 Oregon Department of Education 19

Todays Agenda 2 Assessments and Accountability Assessment Data Reviewing and Editing Oregon Department of Education 20

Assessment Data Reviewing and Editing 1 Changes to Applications and Assessments for 2023-24 Things that Did Not Change for 2023-24 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 21

Changes for 2023-24 Accountability Warehouse Extract (AWE) Assessment Student tab Select Student(s) search screen Gender section add checkbox for Non-Binary Male Female Non-Binary Oregon Department of Education 22

Changes for 2023-24 (contd) Secure Assessment Reports (SAR) Individual Student Report (ISR) or Combined ISR Download files will be zipped with one separate PDF file for each student report. Class Roster Option to hide SSID numbers and/or student names on Class Rosters before printing them Asserted Name on ISR and Combined ISR See Using a Student s Asserted Name on Assessment Score Reports guidance document for instructions Oregon Department of Education 23

Reporting Changes for 2023-24 Extended Assessments Student test scores from Extended assessments count as meeting the alternate standards on Public Assessment Group Reports (PAGR). We have been using Performance Level for Benchmark (PLB) for ELA and Math Achievement on At-A-Glance Profiles and Accountability Details. Starting in 2023-24, we will also be using Performance Level for Benchmark (PLB) for ELA, and Math, and Science performance on Public Assessment Group Reports (PAGR). The likely effect will be increases of 0.2% to 0.5% for state level performance, depending on the grade. District Special Education Students Due to SB 923, we are removing the exclusion of District Special Education students from school level reports. Oregon Department of Education 24

Assessment Data Reviewing and Editing 2 Changes to Applications and Assessments for 2023-24 Things that Did Not Change for 2023-24 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 25

Not Changed in 2023-24 Achievement Standards for ELA, Mathematics, and Science assessments remain the same as for 2022-23 Performance Standards for ELPA Summative remain the same as for 2022-23 Banked test records from this year s 11th graders who met the high school standard in a prior year Identify previous years records by test year (field R in Production Download) or by test date (field S in Production Download) Oregon Department of Education 26

Not Changed in 2023-24 contd Merged SSIDs If test record is already posted, then merged SSID automatically changes assessment record If test record is not yet posted, then loaded with error, and district can confirm SSID (and save record to post) Student demographics and District/School filled with information from SSID record Be sure SSID System has correct demographics at least two days prior to start of student testing; can get help from Regional ESD Partners How to Edit Demographic and Program Fields document EngLrnrCd filled from spring Title III English Learners Collection (calculated by ODE) See Inclusion Rules Manual, Table 8, for list of codes Oregon Department of Education 27

Not Changed in Applications Transactional (ARUA) Student & Institution info (Institution IDs, Student name, grade, etc.) Reporting (AWE) Student & Institution info (Institution IDs, Student name, grade, etc.) Test Identification info (Subject, PADMID, Test Type, Benchmark, Total Test Score & Perf Levels) Test Identification info (Subject, PADMID, Test Type, Benchmark, Total Test Score & Perf Levels) Administration & Accommodation Codes (editable fields) Administration & Accommodation Codes Claim Scores & Perf Levels Reporting Flags Remote Flag Oregon Department of Education 28

Assessment Data Reviewing and Editing 3 Changes to Applications and Assessments for 2023-24 Things that Did Not Change for 2023-24 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 29

Assessments Load Schedule for 2023-24 By May 31 (first closure of the ARUA for processing): ELA & Math tests completed through mid-May Scoring turnaround time for PT is about two weeks Scores are sent to ODE as soon as both parts are completed & scored All ELPA Summative IN-PERSON and REMOTE tests loaded in batches 1-4 High School Domain Proficiency scores will be finalized by June 7 Science tests completed by May 30 Oregon Department of Education 30

Assessments Load Schedule for 2023-24 (contd) By June 11 (reopening of the ARUA after second closure for processing): More ELA & Math tests completed toward the end of May Science tests completed by June 6 (loaded on June 7) Non-Participation ( Virtual ) Records First refresh available to districts on June 11 by 9 AM Refreshed and available weekly through summer (except week of July 4) Student Group Flags updated from Third Period Cumulative ADM & other student collections Same schedule as Virtual records Oregon Department of Education 31

Assessments Load Schedule (contd) Remaining Loads in June and July 2024: Oregon Extended Assessments in ELA, Mathematics, and Science Loaded in one batch: tests data-entered by June 14 will be loaded by June 28 Science Partials Loaded by June 28 ELA and Mathematics completed tests Final load by July 1 ELA and Mathematics Partials Loaded by July 12 Oregon Department of Education 32

Assessment Data Reviewing and Editing 4 Changes to Applications and Assessments for 2023-24 Things that Did Not Change for 2023-24 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 33

Accessing Data Different Applications for Different Functions OSAS Portal (vendor for ELA, Math, Science, and ELPA) Student, Class Roster, and Summary Reports BRT s website (Oregon Extended Assessments vendor) Tablet administration, Data Entry, Reports (raw scores) ODE District Secure Login Applications Accountability Warehouse Extract (AWE) Consolidated Collections ARUA for editing test records & correcting errors Secure Assessment Reports for Official Student, Class Roster, and Class Summary Reports Oregon Department of Education 34

Accessing Data - TIDE Cambium s Test Information Distribution Engine (TIDE) on the OSAS Portal Participation Reports Advantages: immediate status reports of tests in progress and completed Limitations: does not include Oregon Extended Assessments Accountability Limitations: students are not identified as accountable; school summary reports may have more students included than are accountable Oregon Department of Education 35

Accessing Data - CRS Cambium s Centralized Reporting System (CRS) on the OSAS Portal Student, Class Roster, and Summary Reports Advantages: immediate Science scores and performance levels; easy to access Limitations: does not include Oregon Extended Assessments Accountability Limitations: students are not identified as accountable; school summary reports may have more students included than are accountable Oregon Department of Education 36

Accessing Data CRS (contd) Cambium s Centralized Reporting System (CRS) CRS User Guide Training Videos (under OSAS Portal ) Oregon Department of Education 37

Accessing Data ODE Applications ODE District Secure Login Applications Oregon Department of Education 38

Accessing Data - AWE Accountability Warehouse Extract (AWE) Download Test Records in Bulk Files or by Selection Criteria Options: Student Search, Bulk Download, or Regional Warehouse Advantages: final scale scores and performance levels; complete test records with all fields; download as spreadsheet Limitations: one-day lag after records loaded or edited in the ARUA, to show up in AWE; cannot edit any data fields Accountability Limitations: none; all accountability report inclusion flags are available for download Oregon Department of Education 39

Accessing Data - SAR Secure Assessment Reports 2.0 (SAR) Individual Student, Class Roster, and Class Summary Reports Advantages: final scale scores and performance levels for total test and all reported score reporting categories (where available) Limitations: one-day lag after records loaded or edited in the ARUA, to show up in SAR Accountability Limitations: students are not identified as accountable; class summary reports may have more students included than are accountable Oregon Department of Education 40

Accessing Data - ARUA Consolidated Collections ARUA View or Edit Test Records, Fix Errors, Download/Upload Files Advantages: district can edit some data fields; displays final scores and performance levels Limitations: one-day lag after scores posted in CRS, to show up in ARUA; subset of fields from test record (no score reporting category / claim scores or performance levels) Accountability Limitations: accountability report inclusion flags not available for download Oregon Department of Education 41

Accessing Data ARUA Closures ARUA Temporary Closures: Closed May 31 (@ 5 PM) through June 3 Reporting & student group flags will be set Virtual records will be created Reopen June 4 by 9 AM Closed June 7 (@ 5 pm) through June 10 Reporting & student group flags will be set Virtual records will be created Reopen June 11 by 9 AM; will stay open until August 23 @ 5 PM 2023-24 Subjects will not be available from Secure Assessment Reports during these closures. Oregon Department of Education 42

Accessing Data ARUA Errors ARUA Error Management Review District Errors (from district-uploaded files) Review Vendor Errors (from vendor files) If none listed, then no errors to fix! ARUA User Guide Oregon Department of Education 43

Accessing Data ARUA Participation ARUA Participation Report All Records column all Best Scores Not in Denominator column not counted in accountability Participation denominator Participation Denominator column records counted in accountability Participation denominator Participants column records counted in Participation numerator Non-Participants column records counted in Participation denominator as non-participants Oregon Department of Education 44

Accessing Data ARUA Participation Report NOTE: All counts will be in the Not in Denominator column until Participation flags are filled (after May 31, 2024). Oregon Department of Education 45

Accessing Data ARUA Test Lookup Report ARUA Test Lookup Report Look up students at any school to see what subjects they ve tested in No scores are displayed on this report, but there are student names and SSIDs, so please keep secure. Oregon Department of Education 46

Assessment Data Reviewing and Editing 5 Changes to Applications and Assessments for 2023-24 Things that Did Not Change for 2023-24 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 47

Quality Checking Did all grades 3 8 and 11 students enrolled on May 1, 2024 participate in the required assessments? Last day to test students is June 14, 2024 This year s 11thgraders who met the high school standard in a prior year have banked test records. Did all the Oregon Extended Assessments get entered? Last day for ORExt data entry is June 14, 2024 Contact: Mason Rivers Oregon Department of Education 48

Quality Checking (contd) Are these my students? (Does Resident School = May 1 School?) Am I missing students? Review AWE records for accountability reporting flags Are my Non-Participants accurate? Log into AWE, Assessment Student Extract tab Choose the year and subject (i.e., 2023-2024 English Language Arts) Choose Non-Participation under Test Type Open Record Resolution Attributes section, set desired Record Resolution Attribute to Y Refer to Correcting Non-Participants document Oregon Department of Education 49

Quality Checking - Errors Correcting Errors in Assessment Data ARUA Record Management Fix errors online (one record at a time) Review District Errors (from district-uploaded files) Review Vendor Errors (from vendor files) Download/Upload Errors (in a file, all subjects in one file) Most Common Types of Errors Student ID Match (incorrect or merged SSID) Special Ed Flag not Y (Oregon Extended or Challenge Down) Invalid Accommodation Code 1-6 (must match subject, and accommodation flag must be Y) Oregon Department of Education 50