NYC Core Knowledge Early Literacy Pilot Year 3 Report

Evaluating the NYC Core Knowledge Early Literacy Pilot: Year 3 Report Research and Policy Support Group February 2012 1 1

Executive Summary Quantitative Analysis Core Knowledge Reading (CKR) students had significantly greater gains in Year 3 than comparison school students on nearly all measures. Overall Achievement Spring scores for 2nd grade CKR students were greater than that of comparison students on all tests.* High vs. Low Performers The CKR intervention had an impact for all students, regardless of their incoming fall reading scores, but the effects were strongest for students with lower incoming scores. CKR Program Exposure Students new to the CKR program this year (who also had lower incoming fall scores) displayed greater increases on the WJ Brief Reading than those who had been in the program longer. CKR students who had been in the program for multiple years displayed the highest spring scores. Post- Pilot CKR Participation CKR Kindergarten and 1st grade classes implementing the CKR program this year scored significantly higher than comparison schools in the spring on TerraNova Reading. *All differences were statistically significant except TerraNova Reading 2

Executive Summary Surveys & Site Visits Surveys The teacher and administrator surveys indicate high levels of satisfaction with the CKR Program and a preference for the program over other 2nd grade reading curricula. Most teachers found that students were very engaged with the content of the curriculum and that it was successful at sparking enthusiasm for reading. Site Visits Fidelity to the program was quite high. The main difficulty noted was in allotting an uninterrupted 60 minutes for program implementation. Most of the observed classrooms fully utilized the materials provided and students were largely attentive during lessons, participated in discussions, and were curious about the content of the materials. In surveys and interviews, teachers noted difficulty with differentiation of instruction using the provided materials for students with special learning needs, such as English Language Learners and Students with Disabilities. 3

Focus of the Evaluation Hypothesis:Students taught with the Core Knowledge Reading (CKR) Program will gain reading competencies and content knowledge (science and social studies skills) at a faster rate than their peers. Methodology: A multi-method, longitudinal research design PROGRAM YEAR 3: 2ND GRADE Assessments of 2nd Grade Students (at 10 CKR schools & 10 comparison schools)* Pre- and post-test of literacy skills Tests of science and social studies skills at the end of the year Additional test of content-specific literacy skills (the Core Knowledge Assessment for CKR students only) at the end of the year Teacher and Administrator Surveys (at 10 CKR schools): Assesses satisfaction with and impact of CKR Case studies (at 4 CKR schools): Classroom observations, administrator & teacher interviews ADDITIONAL GRADE LEVELS Assessments of Kindergarten and 1st Grade Students** (at 9 CKR schools & 9 comparison schools) Tests of literacy, science and social studies skills at end of the third year In Year 1 and 2, Reading First schools were used as a subgroup in the evaluation. However, these schools were no longer Reading First in Year 3. ** Kindergartners and 1st graders in CKR pilot schools no longer received full CKR program supports and two schools did not test additional students. 4

Demographics: While the comparison schools for the evaluation were initially picked for their similarities to the pilot schools, analyses control for demographics to account for any demographic differences that have emerged over the past two years. Percent of 2nd Grade Students CKR Students (N = 700) 88.0 82.0 Comparison Students (N = 357)* 78.2 70.1 * N = the number of students for whom both fall and spring data were available for any WJ test. A random sample of half of the students in the comparison schools were selected for testing. All students in CKR schools were tested. 39.8 39.1 12.6 9.5 6.8 5.7 ELL Free/ Reduced Lunch Black/ Hispanic Level 3/4 Students ELA School Size (in Thousands) * * ** Analyses in all slides exclude students with disabilities. Data from 08-09 was used to select comparison schools (data presented here are from the 2010-11 school year). * Difference between CKR and Control groups statistically significant. ** ELA percents based on total number of students who took the exams in the schools in this evaluation. 5

Evaluation of Achievement Gains 6

Quantitative Results Overview: CKR students displayed higher scores and gains on nearly all tests. Woodcock-Johnson III TerraNova W-J Brief Reading TerraNova Reading Test W-J Letter Word Identification W-J Passage Comprehension W-J Word Attack W-J Spelling of Sounds (Oral Reading Comprehension, Vocabulary, Basic Reading, Decoding) TerraNova Social Studies TerraNova Science (Basic Reading Skills) (Decoding) (Written Spelling) (Oral Reading Comprehension) X CKR Spring Scores at CKR Schools Compared to Comparison Schools CKR CKR Significantly Greater Spring Scores CKR CKR CKR No Significant Difference Significantly Greater Spring Scores Significantly Greater Spring Scores Significantly Greater Spring Scores Significantly Greater Spring Scores Significantly Greater Spring Scores (But CK students had higher scores) Change in Fall to Spring Scores at CKR Schools Compared to Comparison Schools * CKR CKR Greater Significantly Fall to Spring Change CKR Greater Significantly Fall to Spring Change CKR Greater Significantly Fall to Spring Change Not Not Not Significantly Greater Fall to Spring Change Administered in Fall Administered in Fall Administered in Fall Note: All analyses control for student demographic characteristics. 7 *Significant only at one-tailed level. Two-tailed p=.067

Quantitative Results Overview: CKR students literacy gains were more than double the gains of students at demographically similar comparison schools. Average Fall-Spring Gain in Scale Score Points Woodcock-Johnson (Brief Reading Test) CKR Students Displayed A Significantly Greater Change From Fall to Spring Than Comparison School Students (N = 683) (N = 352) 8

Impact of time in program: CKR had the largest impact among students who were new to the program (who had lower starting scores) However, the highest ultimate (spring) scores were found among those who had been in the program the longest. Average Fall and Spring Scale Score Points By Years in CKR Program in Year 3 Woodcock-Johnson (Brief Reading) 105.2 +0.8 pts 104.4 104.5 +2.6 pts 1 Year/2nd Grade Only (N=86) 2 Years/1st and 2nd Grade (N=162) 3 Years/All Grades (N=435) 101.9 100.7 + 3.2 pts 97.4 Fall Spring 9

Impact of time in program: The longer second grade students had been in the CKR program, the higher their TerraNova Reading scores were. Average Spring Scale Score Points By Years in CKR Program TerraNova (Reading Test) 610.0 50th percentile Nationwide (606) 600.0 594 590.0 582 580.0 577 570.0 560.0 10th percentile Nationwide (553) 550.0 540.0 3 Years/All Grades (N=447) 2 Years/1st and 2nd Grade (N=169) 1 Year/2nd Grade Only (N=129) Scores are not adjusted for demographics. 10

Effects of CKR Following Participation in the Pilot: Kindergarten students in schools implementing CKR showed higher TerraNova Reading spring scores than students in comparison schools in both the year they participated in the pilot and the current year. Spring Scores TerraNova Reading Test CKR Kindergartners in 2008-2009 Kindergarteners in Year 1 CKR Kindergartners in 2010-2011 Kindergarteners in Year 3 512.2 510.2 501.9 501.7 CKR Schools (n=693) Comparison Schools (n=307) CKR Schools (n=547) Comparison Schools (n=215) Scores are not adjusted for demographics. 11

Effects of CKR Following Participation in the Pilot: First grade students in schools implementing CKR showed higher TerraNova Reading spring scores than students in comparison schools in both the year they participated in the pilot and the current year. Spring Scores TerraNova Reading Test CKR First Graders in 2009-2010 First Graders in Year 2 CKR First Graders in 2010-2011 First Graders in Year 3 557.6 560.6 541.0 555.0 CKR Schools (n=698) Comparison Schools (n=391) CKR Schools (n=600) Comparison Schools (n=223) Scores are not adjusted for demographics. 12

Administrator and Teacher Surveys 13 13

Administrators reported satisfaction with the CKR program. Will your 2nd grade classrooms be using the CK Reading Curriculum next year? (n = 9) Would you recommend the CK Reading curriculum to other school administrators you know? (n = 9) Administrators overall satisfaction with CK Reading (n = 9) * No 2 1 No Not Sure Neutral 1 7 Very Satisfied 7 Somewhat Satisfied Yes Yes Each school is unique. I have a significant amount of ELL s and SWD's other schools may not. As I have said before, children gain in Phonics and Phonemic Awareness. They need more basic vocabulary development and comprehension skills and making a writing connection. We feel that it is not preparing them for the state standards for reading and Writing. The writing connection is missing. We don't feel there is enough aligned to the new CCSS and although the children are able to get the Phonics, and Phonemic Awareness down; the comprehension and independent thinking skills are lacking * Administrators had five options for answering this question: Very Satisfied; Somewhat Satisfied; Neutral; Somewhat Dissatisfied; Very Dissatisfied . 14 14

Most teachers were satisfied with CKR and reported that CKR was better than other reading programs. Teachers Views: The students enjoyed learning new concepts which sparked their reading and the need to know. Although their skills were limited they were always ready for a new concept or a new presentation of skills. Because this was a new program for me as a teacher it sparked my interests as well. Percent of Respondents 77.0% 72.2% 38.5% Over more than twenty years of teaching I have experienced a number of reading programs. Phonics, spelling, vocabulary and writing and comprehension skills were integrated throughout and the sequence made sense. Enrichment suggestions and remediation suggestions were presented throughout the materials. 38.9% 38.5% 33.3% Very Satisfied Much Better Somewhat Satisfied Somewhat Better Teachers overall opinion of CK Reading compared with other reading programs (n =13)b Teachers overall satisfaction with CK Reading (n = 18)a Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count. Other Answers: Question a Neutral=2 (11.1%); Somewhat dissatisfied=2 (11.1%); Very Dissatisfied=1 (5.6%); Question b About the Same=0; Somewhat Worse=2 (15.4%); Much Worse=1 (7.7%). In Year 2, Overall satisfaction was 82.7% very/somewhat satisfied, while overall opinion of CK relative to other programs was 50% much/somewhat better 15 15

Teachers indicated that both CKR strands had clear goals, were engaging, and that they had sufficient time to complete the lessons. In Year 2, teachers had rated the Skills Strands higher on these measures.* Percent of Respondents Strongly Agree Listening and Learning Strand Skills Strand Somewhat Agree 87.5% 81.3% 75.0% 75.0% 68.8% 68.8% 12.5% 25.0% 6.3% 18.8% 25.0% 25.0% 68.8% 62.5% 50.0% 50.0% 62.5% 50.0% Goals of lessons are clear (n = 16) I have enough time to complete daily lessons (n = 16) Students find activities engaging (n = 16) Goals of lessons are clear (n = 16) Students find activities engaging (n = 16) I have enough time to complete daily lessons (n = 16) Neutral: N=1 (6.3%) Disagree: N=0 (0.0%) Strongly Disagree: N=1 (6.3%) Neutral: N=1 (6.3%) Disagree: N=2 (12.5%) Strong y Disagree: N=1 (6.3%) Neutral: N=1 (6.3%) Disagree: N=2 (12.5%) Strongly Disagree: N=1 (6.3%) Neutral: N=2 (12.5%) Disagree: N=1 (6.3%) Strongly Disagree: N=2 (12.5%) Neutral: N=2 (12.5%) Disagree: N=1 (6.3%) Strong y Disagree: N=2 (12.5%) Neutral: N=1 (6.3%) Disagree: N=0 (0.0%) Strong y Disagree: N=2 (12.5%) * In Year 2, those that rated Strongly Agree or Somewhat Agree: Goals of lessons are clear =96.6%;Students find activities engaging=78.6%; I have enough time to complete daily lessons= 72.4%. Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count. 16 16

Compared to other programs they taught, most teachers found CKR much or somewhat better at teaching content/background knowledge. Somewhat Better Much Better Teachers Views: I like the content reading, the students really enjoyed learning about different areas and different people. I always got the question-what are we going to learn next? 84.6% 69.2% 69.2% 54.5% 50.0% 61.5% 38.5% 38.5% 9.1% 25.0% 45.5% 30.8% 30.8% 25.0% 23.1% Teaching Teaching decoding skills (N=13) Ability to engage students and spark enthusiasm for reading (N=13) Accommodations for different learning needs (N=11) Sequence of instruction (N=12) content/background knowledge (N=13) About the Same: N=1 (7.7%) Somewhat Worse: N=0 Much Worse: N=1 (7.7%) About the Same: N=1 (7.7%) Somewhat Worse: N=1 (7.7%) Much Worse: N=2 (15.4%) About the Same: N=1 (7.7%) Somewhat Worse: N=2 (15.4%) Much Worse: N=1 (7.7%) About the Same: N=2 (18.2%) Somewhat Worse: N=2 (18.2%) Much Worse: N=1 (9.1%) About the Same: N=3 (25%) Somewhat Worse: N=2 (16.7%) Much Worse: N=1 (8.3%) Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count. 17

Teachers believed the CKR program was better than other programs they have previously used at accommodating different learning needs and teaching vocabulary. Teachers Views: The writing component was weak and did not really teach students how to write a multi paragraph writing. Most of your writing pieces were quick and easy. They didn't really show the writing process or teach students how to write extendedresponses. Somewhat Better Much Better 76.9% 66.7% 63.6% 11.1% 9.1% 53.8% 38.5% 15.4% 55.6% 54.5% 23.1% 23.1% Teaching vocabulary (N=13) Accommodations for special education students (N=9) Accommodations for English language learners (N=11) Teaching writing skills (N=13) About the Same: N=0 Somewhat Worse: N=2 (15.4%) Much Worse: N=1 (7.7%) About the Same: N=0 Somewhat Worse: N=2 (22.2%) Much Worse: N=1 (11.1%) About the Same: N=2 (18.2%) Somewhat Worse: N=1 (9.1%) Much Worse: N=1 (9.1%) About the Same: N=3 (23.1%) Somewhat Worse: N=2 (15.4%) Much Worse: N=3 (23.1%) Note: Calculations were based only on responses that selected an answer other than N/A, which resulted in a lower overall N count. 18

Site Visits and Interviews 19 19

Measuring fidelity to the CKR curriculum Examining implementation fidelity allows us to better determine: 1) whether achievement gains can be attributed to the CKR program 2) which components of the CKR program teachers are struggling to implement and those they are implementing successfully We took several different approaches to measuring fidelity in the CKR Pilot schools: Site visits with classroom observations Site visits were conducted at 4 randomly selected Pilot schools 11 classrooms were observed Interviews with administrators and teachers Interviews were conducted with 16 teachers and 3 administrators Additional teacher and administrator survey questions 20 20

During the site visits, we used a classroom observation protocol developed from the Core Knowledge Reading Pilot Observation Form Examples from Checklist Check to see that CKR visuals are posted and that CKR materials are present and being used effectively These components were taken from the Core Knowledge Reading Pilot Observation Form Verify that the daily schedule allows for both the Skills and the Listening and Learning Strands to last a full 60 minutes each Whether or not students are familiar with language introduced in the domain Teacher engaging all students in exercises, practice, class discussions, and games Whether or not students are attentive to the teacher during instruction Whether or not students are eager to participate in the daily activities of the program These components were added as additional measures of student engagement Whether or not students are generally on task and complete assignments in a timely fashion Evidence of differentiated instruction and assessment techniques 21 21

Nearly all classrooms demonstrated high fidelity to the CKR program. In particular, we observed high levels of student engagement in lessons. Number of Classrooms (N=11) 1 2 3 11 Observed During Visit 10 10 9 8 Not Observed During Visit 1 Students are attentive to their teacher during instruction Full 60 minutes is allotted for both the Listening and Learning and Skills Strands* Teachers have Skills Strand materials posted and used Word Walls effectively Small group and/or individual practice time Students respond appropriately to their teacher s directions, questions, and assignments * 60 minutes for the Strands typically not allotted uninterrupted. 22

Overall, teachers and administrators reported high fidelity to the program. One area where they reported struggling to implement the program was in finding time for daily small group instruction. Fidelity Survey Questions Spring 2011 Teacher Responses Please indicate your overall level of satisfaction with the CK Reading Curriculum so far. 72.2% Very Satisfied or Somewhat Satisfied Typically, I find I have enough time to complete the daily Skills Strand Lesson 81.3% Strongly or Somewhat Agreed Typically, I find I have enough time to complete the daily Listening and Learning lesson 75% Strongly or Somewhat Agreed Core Knowledge Reading engages students and sparks enthusiasm for reading 69.2% Strongly or Somewhat Agreed The Skills Strand materials are developmentally appropriate 75% Strongly or Somewhat Agreed The Listening and Learning materials are developmentally appropriate 62.5% Strongly or Somewhat Agreed The CKR curriculum accommodates different learning needs 54.5% Strongly or Somewhat Agreed 23

Next Steps: Year 4 NYCDOE/RPSG s Year 4 Evaluation Track student achievement through 3rd grade on the NYS English Language Arts (ELA) test Comparison of 3rd grade 2012 NY State ELA test scores among: Students who participated in the CKR Pilot as second graders in the 2010-2011 vs. students who were assessed at comparison schools in that year. Students who had previously participated in the CKR Pilot as second graders (one year in the pilot) versus those who had previously participated in the CKR Pilot for multiple years. Exploration of associations between 3rd grade scores on the 2012 NYS ELA test and previous performance on tests of reading comprehension (WJIII, TerraNova) while participating in the pilot as second graders. 24