Oregon Department of Education Training: Assessment Reporting and Accountability Details

Assessment Reporting (Editing and Checking Assessment Data, Third Period Cumulative ADM, and English Learners) May 11, 2023 Training - ODE District site (to participate, get handouts, and view recorded webinars) Welcome! Oregon Department of Education 1

Contacts Regional ESD Partners Accountability Reporting Team Jon Wiens, Manager 503-931-7682 Cindy Barrick, Research Analyst 971-208-0480 Bianca Llamas, Research Analyst 971-208-0437 Elyse Bean, Research Analyst 971-720-8659 Oregon Department of Education 2

Todays Agenda 1 Assessments and Accountability Assessment Data Reviewing and Editing Oregon Department of Education 3

Assessments and Accountability 1 ESSA, School Profiles, and Other Assessment Accountability Reports Timelines Links for Information Oregon Department of Education 4

ESSA and School Profiles Every Student Succeeds Act (ESSA) ESSA maintains annual testing in English language arts and mathematics (grades 3-8 and 11), and science (grades 5, 8, and 11). ESSA maintains report cards and requires the inclusion of English Learner progress and at least one measure of school quality/student success as indicators to display on report cards. ESSA requires that the denominator for the achievement calculations includes at least 95% of students enrolled at the school, or the number of students participating, whichever is higher. ESSA Amendment Oregon Department of Education 5

School and District At-A-Glance Profiles Displays: Information on students and staff Class Size and Regular Attenders Statewide assessments Ninth Grade On-Track and Graduation Local information on school programs, parent/community engagement TENTATIVE availability date: July 27, 2023 (subject to change) More details in the webinar in July Oregon Department of Education 6

Accountability Details Elementary & Middle Indicator Levels for all students and 12 student groups ELA Achievement, Average Gap Score Change, and Participation Math Achievement, Average Gap Score Change, and Participation Regular Attenders On Track to English Language Proficiency (ELP) TENTATIVE availability date: July 20, 2023 (subject to change) More details in the June webinar Oregon Department of Education 7

Accountability Details High School Indicator Levels for all students and 12 student groups ELA Achievement and Participation Math Achievement and Participation Regular Attenders On Track to English Language Proficiency (ELP) Ninth Grade On-Track Four-year Cohort Graduation Rate Five-year Completers Rate Oregon Department of Education 8

Other Accountability Reports Other Reports using Assessment Data (outside of ESSA) Public Assessment Group Reports for current and previous years Title III English Learners (formerly AMAO) Special Education At-A-Glance Profiles for Districts Federal EDFacts Reports Assessment Participation (C185, C188, C189) Academic Achievement (C175, C178, C179) Title III Former LEP (C126) ELPA Participation and Proficiency (C138, C139, C205) Indicator Status (C200, C201) Oregon Department of Education 9

Assessments and Accountability 2 ESSA, School Profiles, and Other Assessment Accountability Reports Timelines Links for Information Oregon Department of Education 10

Timelines for Reports Student and Classroom Reports (in Secure Assessment Reports 2.0 application) ELA, Math, Science available since February 2, 2023 ELPA Summative Batch 1 available March 14; Batch 2 available April 5; Batch 3 available May 12; Batch 4 (including remote administration) will be available June 2 Oregon Extended Assessments will be available June 16 Public Test Results (ELA, Math, Science) Validation Window (in ADI) August 10-18, 2023 Final 2022-23 Test Results Posted mid-September Oregon Department of Education 11

Timelines for Reports contd At-A-Glance School and District Profiles, Accountability Details Accountability Details Validation Window: July 20 October 2, 2023 At-A-Glance Profiles Validation Window: July 27 October 2, 2023 Final reports posted for district preview: October 5, 2023 Final reports posted on the public website: mid-October, 2023 Oregon Department of Education 12

Timelines for Data Sources Institutions for Accountability Reporting Validation Window in ADI: April 13 May 26, 2023 Boundary Calculators for 2022-23 due to ODE: May 1 @ noon (Institution changes for 2023-24 due to ODE: September 15, 2023) Third Period Cumulative ADM Collection Data Collection Window: April 27 June 2, 2023 Audit Review Window: June 8 - 16, 2023 Validation Window in ADI: June 1 August 18, 2023 Title III English Learners Collection Spring Data Collection Window: April 13 May 26, 2023 Review Window: June 8 - 23, 2023 Oregon Department of Education 13

Timeline for Assessments See slides 28-30 for Assessments Load Schedule Non-participation (virtual) records generated based on Spring Membership starting May 30 (editable in the ARUA) Demographic and Program fields filled from Third Period Cumulative ADM (Spring Membership) after May 26 Refreshed and available weekly through the summer (same schedule as for Accountability Details refresh) Oregon Department of Education 14

Timelines for Final Edits Last date for edits to enrollment and assessment records: August 18, 2023 The ARUA closes August 18 at 5 PM Third Period Cumulative ADM closes June 2 at 11:59 PM; reopened during audit window (June 8-16) and upon request to Regional ESD Partner or Amanda Leopard; final editing closes August 18 @ 11:59 PM Oregon Department of Education 15

Assessments and Accountability 3 ESSA, School Profiles, and Other Assessment Accountability Reports Timelines Links for Information Oregon Department of Education 16

Links for Webpages Quick Links to Accountability Information Assessment & Accountability Checklist Every Student Succeeds Act (ESSA) Training (on the ODE District secure website) Oregon Department of Education 17

Links for Assessment Help Documents, tools, guidance Quick Links to Accountability Information OSAS Online User Guides Contacts Regional ESD Partners Application assistance: ODE Helpdesk Accountability Reporting: Elyse Bean Oregon Department of Education 18

Questions? 1 Oregon Department of Education 19

Todays Agenda 2 Assessments and Accountability Assessment Data Reviewing and Editing Oregon Department of Education 20

Assessment Data Reviewing and Editing 1 Changes to Applications and Assessments for 2022-23 Things that Did Not Change for 2022-23 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 21

Changes for 2022-23 Assessment Reporting File Format (for AWE) NEW FIELD Remote Flag (RemoteFg) new field added to end of file layout (row 153, column EW) Yes/No flag Yes = all or part of test was remotely administered No = test was not remotely administered Oregon Department of Education 22

Assessment Data Reviewing and Editing 2 Changes to Applications and Assessments for 2022-23 Things that Did Not Change for 2022-23 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 23

Not Changed in 2022-23 Achievement Standards for ELA, Mathematics, and Science assessments remain the same as for 2021-22 Performance Standards for ELPA Summative remain the same as for 2021-22 Banked test records from this year s 11th graders who met the high school standard in a prior year Identify previous years records by test year (field R in Production Download) or by test date (field S in Production Download) Oregon Department of Education 24

Not Changed in 2022-23 contd Merged SSIDs If test record is already posted, then merged SSID automatically changes assessment record If test record is not yet posted, then loaded with error, and district can confirm SSID (and save record to post) Student demographics and District/School filled with information from SSID record Be sure SSID System has correct demographics at least two days prior to start of student testing; can get help from Regional ESD Partners How to Edit Demographic and Program Fields document EngLrnrCd filled from spring Title III English Learners Collection (calculated by ODE) See Inclusion Rules Manual, Table 8, for list of codes Oregon Department of Education 25

Not Changed in Applications Transactional (ARUA) Student & Institution info (Institution IDs, Student name, grade, etc.) Reporting (AWE) Student & Institution info (Institution IDs, Student name, grade, etc.) Test Identification info (Subject, PADMID, Test Type, Benchmark, Total Test Score & Perf Levels) Test Identification info (Subject, PADMID, Test Type, Benchmark, Total Test Score & Perf Levels) Administration & Accommodation Codes (editable fields) Administration & Accommodation Codes Claim Scores & Perf Levels Reporting Flags Oregon Department of Education 26

Assessment Data Reviewing and Editing 3 Changes to Applications and Assessments for 2022-23 Things that Did Not Change for 2022-23 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 27

Assessments Load Schedule for 2022-23 By May 26 (first closure of the ARUA for processing): ELA & Math tests completed through mid-May Scoring turnaround time for PT is about two weeks Scores are sent to ODE as soon as both parts are completed & scored All ELPA Summative IN-PERSON tests loaded in batches 1-3 High School Domain Proficiency rescores will be in by May 19 Science tests completed by May 26 Oregon Department of Education 28

Assessments Load Schedule for 2022-23 (contd) By June 6 (reopening of the ARUA after second closure for processing): More ELA & Math tests completed through mid-May All ELPA Summative REMOTE tests loaded by June 2 Science tests completed by June 1 (loaded on June 2) Non-Participation ( Virtual ) Records Initial load and available to districts on May 31 by 9 AM Refreshed and available weekly through summer (except week of July 4) Student Group Flags updated from Third Period Cumulative ADM & other student collections Same schedule as Virtual records Oregon Department of Education 29

Assessments Load Schedule (contd) Remaining Loads in June 2023: Oregon Extended Assessments in ELA, Mathematics, and Science Loaded in one batch: tests data-entered by June 2 will be loaded by June 16 Science Partials Loaded by June 16 ELA and Mathematics completed tests Final load by June 23 ELA and Mathematics Partials Loaded by June 30 Oregon Department of Education 30

Assessment Data Reviewing and Editing 4 Changes to Applications and Assessments for 2022-23 Things that Did Not Change for 2022-23 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 31

Accessing Data Different Applications for Different Functions OSAS Portal (vendor for ELA, Math, Science, and ELPA) Student, Class Roster, and Summary Reports BRT s website (Oregon Extended Assessments vendor) Tablet administration, Data Entry, Reports (raw scores) ODE District Secure Login Applications Accountability Warehouse Extract (AWE) Consolidated Collections ARUA for editing test records & correcting errors Secure Assessment Reports for Official Student, Class Roster, and Class Summary Reports Oregon Department of Education 32

Accessing Data - TIDE Cambium s Test Information Distribution Engine (TIDE) on the OSAS Portal Participation Reports Advantages: immediate status reports of tests in progress and completed Limitations: does not include Oregon Extended Assessments Accountability Limitations: students are not identified as accountable; school summary reports may have more students included than are accountable Oregon Department of Education 33

Accessing Data - CRS Cambium s Centralized Reporting System (CRS) on the OSAS Portal Student, Class Roster, and Summary Reports Advantages: immediate Science scores and performance levels; easy to access Limitations: does not include Oregon Extended Assessments Accountability Limitations: students are not identified as accountable; school summary reports may have more students included than are accountable Oregon Department of Education 34

Accessing Data CRS (contd) Cambium s Centralized Reporting System (CRS) CRS User Guide Training Videos (under OSAS Portal ) Oregon Department of Education 35

Accessing Data ODE Applications ODE District Secure Login Applications Oregon Department of Education 36

Accessing Data - AWE Accountability Warehouse Extract (AWE) Download Test Records in Bulk Files or by Selection Criteria Options: Student Search, Bulk Download, or Regional Warehouse Advantages: final scale scores and performance levels; complete test records with all fields; download as spreadsheet Limitations: one-day lag after records loaded or edited in the ARUA, to show up in AWE; cannot edit any data fields Accountability Limitations: none; all accountability report inclusion flags are available for download Oregon Department of Education 37

Accessing Data - SAR Secure Assessment Reports 2.0 (SAR) Individual Student, Class Roster, and Class Summary Reports Advantages: final scale scores and performance levels for total test and all reported score reporting categories Limitations: one-day lag after records loaded or edited in the ARUA, to show up in SAR Accountability Limitations: students are not identified as accountable; class summary reports may have more students included than are accountable Oregon Department of Education 38

Accessing Data - ARUA Assessment Record Updating Application (ARUA) (Consolidated Collections Assessment Collections) View or Edit Test Records, Fix Errors, Download/Upload Files Advantages: district can edit some data fields; displays final scores and performance levels Limitations: one-day lag after scores posted in CRS, to show up in ARUA; subset of fields from test record (no score reporting category / claim scores or performance levels) Accountability Limitations: accountability report inclusion flags not available for download Oregon Department of Education 39

Accessing Data ARUA Closures ARUA Temporary Closures: Closed May 26 (@ 5 PM) through May 30 Reporting & student group flags will be set Virtual records will be created Reopen May 31 by 9 AM Closed June 2 (@ 5 pm) through June 5 Reporting & student group flags will be set Virtual records will be created Reopen June 6 by 9 AM; will stay open until August 18 @ 5 PM 2022-23 Subjects will not be available from Secure Assessment Reports during these closures. Oregon Department of Education 40

Accessing Data ARUA Errors ARUA Error Management Review District Errors (from district-uploaded files) Review Vendor Errors (from vendor files) If none listed, then no errors to fix! ARUA User Guide Oregon Department of Education 41

Accessing Data ARUA Participation ARUA Participation Report All Records column all Best Scores Not in Denominator column not counted in accountability Participation denominator Participation Denominator column records counted in accountability Participation denominator Participants column records counted in Participation numerator Non-Participants column records counted in Participation denominator as non-participants Oregon Department of Education 42

Accessing Data ARUA Participation Report NOTE: All counts will be in the Not in Denominator column until Participation flags are filled (after May 26, 2023). Oregon Department of Education 43

Accessing Data ARUA Test Lookup Report ARUA Test Lookup Report Look up students at any school to see what subjects they ve tested in No scores are displayed on this report, but there are student names and SSIDs, so please keep secure. Oregon Department of Education 44

Assessment Data Reviewing and Editing 5 Changes to Applications and Assessments for 2022-23 Things that Did Not Change for 2022-23 Assessments Load Schedule Accessing Data Quality Checking Oregon Department of Education 45

Quality Checking Did all grades 3 8 and 11 students enrolled on May 1, 2023 participate in the required assessments? Last day to test students is June 2, 2023 This year s 11thgraders who met the high school standard in a prior year have banked test records. Did all the Oregon Extended Assessments get entered? Last day for ORExt data entry is June 2, 2023 Contact: Mason Rivers Oregon Department of Education 46

Quality Checking (contd) Are these my students? (Does Resident School = May 1 School?) Am I missing students? Review AWE records for accountability reporting flags Are my Non-Participants accurate? Log into AWE, Assessment Student Extract tab Choose the year and subject (i.e., 2022-2023 ELA) Choose Non-Participation under Test Type Open Record Resolution Attributes section, set desired Record Resolution Attribute to Y Refer to Correcting Non-Participants document Oregon Department of Education 47

Quality Checking - Errors Correcting Errors in Assessment Data ARUA Record Management Fix errors online (one record at a time) Review District Errors (from district-uploaded files) Review Vendor Errors (from vendor files) Download/Upload Errors (in a file, all subjects in one file) Most Common Types of Errors Student ID Match (incorrect or merged SSID) Special Ed Flag not Y (Oregon Extended or Challenge Down) Invalid Accommodation Code 1-6 (must match subject, and accommodation flag must be Y) Oregon Department of Education 48

Quality Checking Non-Participation Valid TstValid Values Test Attempt Code Applicable Subjects Definition Use in Participation Denominator Use in Participation Numerator Eligible for use in Performance Denominator N (Not Valid) P (Partial Attempt) Oregon Extended (CE, CM, NS) Not applicable (NA) NA NA NA N (Not Valid) N (Non-Attempt) OSAS ELA & Math (CE, CM) Any incomplete (1 or more but fewer than 5 CAT responses or no PT responses) test Yes, except CalcAdmnCd = 6, 9 No No N (Not Valid) N (Non-Attempt) OSAS Science (NS) Not applicable (NA) NA NA NA N (Not Valid) N (Non-Attempt) Oregon Extended (CE, CM, NS) Any incomplete (1 or more but fewer than 10 responses) test Yes, except CalcAdmnCd = 6, 9 No No V (Virtual Record) V (Virtual Record) ODE created (CE, CM, NS) Expected test matching Third Period Cumulative ADM Collection, but no test found or test was not attempted Yes, except CalcAdmnCd = 6, 9 No No [Exception in ELA where first year EL can take ELPA in lieu of ELA (EngLrnrCd = B)] Oregon Department of Education 49

Quality Checking Administration Codes Identify Test Administration Codes for the following: Home schooled, foreign exchange, or students resident or attending out of state (code = 6) Home schooled student should be registered with an ESD. Do not use this code to exclude regular students who did not take a class in the subject assessed. Students with a medical emergency unable to test during the entire testing window (code = 9) Students opted out of ELA or Math (code = X) Partial/completed test records will be used for participation and performance Virtual records count as non-participants Students taking ELPA Summative but have an IEP to exclude specific domains ELPA Summative tests are scored using responses to appropriate domains (see next slide) Oregon Department of Education 50