ELM and Heart Baseline Report October 2015: Research Findings and Sample Analysis

ELM AND HEART BASELINE REPORT OCTOBER 2015 Cindy Sobieski, Specialist, Research Learning Lauren Pisani, Senior Specialist, Research Learning

Overview I. Research questions II. Sample III. Tools used IV. Baseline skill scores Motor Development Emergent Numeracy Emergent Literacy Socio-emotional Development SEARS-PRE V. Conclusion

I. Research questions Key question for the baseline: Are Malawian children in both Heart & ELM, ELM only, HEART only and Control similar in terms of their measurable early learning and development skills? Differences found between groups at baseline may be controlled for in follow-up study planned for April 2016 Few differences gives more confidence in our comparison of children s learning and development growth during school year 2015-2016 To what extent are teachers able to identify the social & emotional assets and resilience in the preschool sample? Are there differences between the four groups? What can be learned about children s psychosocial development from their drawings and responses to open ended-narrative responses?

II. Sample 553 five-year-old children in Zomba, Malawi All children in this study are enrolled in early childhood development centers (CBCCs) and are divided into four groups: Save the Children s Early Literacy and Mathematics program (ELM): 10 centers and 139 children Save the Children s HEART program: 10 centers and 135 children Both ELM and Heart program: 10 centers and 140 children Previously established early childhood program (Control): 10 centers and 139 children 14 children were randomly chosen at each CBCC 14 children were not available at some smaller centers All children were given the opportunity to decline participation without penalty

II. Sample Table 1: CBCC Sample, by Group HEART Only Heart & ELM Comparison ELM Only Year Established (Average) 2011 2013 2010 2011 Total Enrollment Boys Girls 49 48% 52% 47 49% 51% 60 45% 55% 56 49% 51% Sampled as evenly as possible from both genders: Boys (47%) and Girls (53%). Sample reflects total enrollment at all CBCCs slightly more girls than boys. ELM Only group established, on average, more recently than the other groups

III. Tools used to measure outcomes IDELA (International Development and Early Learning Assessment), with added socio-emotional items The core IDELA assessment has 22 child-reported items covering 4 areas: Motor development, early numeracy, early literacy, and socio-emotional development. 20 of these were used. 3 additional items (strengths, preferences, and free drawing) were added to learn more about children s socio-emotional development 1 cognition item related to executive functioning was also added (memory) SEARS-PRE (Socio-Emotional Assets and Resiliency Scale for Preschool) 25 teacher-reported items adapted by Dr. Girija Kaimal from Dr. Nancy Ravitch s original work Teachers rated children s behavior in class on a 3 point scale from Rarely to Often For example: Child smiles when playing with other children and adults

IV. Motor skills scores, by group Table 2. Baseline Motor Development Points Possible Comparison Heart & ELM ELM Only HEART Only Drawing a Human 8 37% 34% 28% 41% av Folding 4 40% 39% 41% 44% Copying a Triangle 4 32% 38% 35% 46% Total Motor Index 3 36% 37% 35% 44% N 139 139 135 140

IV. Motor skills scores, by group Figure 1: Baseline Motor Development Index, by Group On average, children completed an average of 1.1 out of 3 motor items correctly (38% correct) 50% 44% 45% 37% 40% 36% 35% 35% 30% No significant differences between the groups 25% 20% 15% 10% 5% 0% Comparison ELM Only HEART Only Heart & ELM Note: Asterisks relate to group with the highest score; * p <.05, ** p < .01, *** p < .001

IV. Early numeracy skills scores, by group Table 3. Baseline Early Numeracy Points Possible Comparison 4 2 5 20 3 3 4 7 ELM Only 76% 48% 29% 3% 23% 45% 15% 34% 139 HEART Only 84% 46% 26% 5% 24% 42% 20% 35% 135 Heart & ELM 83% 46% 28% 3% 27% 54% 21% 37% 140 Measuring Size/Length Sorting Shape ID Number ID Counting Simple Operations Puzzle Completion Total Numeracy Index N 78% 50% 26% 2% 27% 47% 16% 35% 139 av

IV. Early numeracy skills scores, by group Figure 2: Baseline Early Numeracy, by Group On average, children answered an average of 2.5 out of 7 mathematics questions correctly (36% correct) 50% 45% 37% 40% 35% 35% 34% 35% No significant differences between the groups 30% 25% 20% 15% 10% 5% 0% Comparison ELM Only HEART Only Heart & ELM Note: Asterisks relate to group with the highest score; * p <.05, ** p < .01, *** p < .001

IV. Early literacy skills scores, by group Table 4. Baseline Early Literacy Skills Points Possible 3 20 20 5 3 4 6 Comparison 48% 1% 28% 45% 7% 28% 26% 139 ELM Only 51% 1% 27% 45% 6% 29% 27% 139 HEART Only Heart & ELM 47% 2% 25% 41% 7% 27% 25% 135 Print Awareness Letter ID Expressive Vocab Oral Comprehension Phonetic Awareness Writing Total Literacy Index N 57% 2% 32% 48% 9% 33% 30% 140 av

IV. Early literacy skills scores, by group Figure 3: Baseline Early Literacy Index, by Group On average children had an emergent literacy score of 1.6 out of 6 possible points (27% correct) 50% 45% 40% 35% No significant differences between the treatment groups and the comparison group 30% 27% 30% 26% 25%* 25% 20% 15% The Heart & ELM Group scored significantly higher the HEART Only group 10% 5% 0% Comparison ELM Only HEART Only Heart & ELM Note: Asterisks relate to group with the highest score; * p <.05, ** p < .01, *** p < .001

IV. Socio-emotional skills scores, by group Table 5. Baseline Socio-emotional Development Skills Points Possible Comparison ELM Only HEART Only Heart & ELM Personal Information 4 49% 48% 50% 54% av Friends 10 47% 47% 44% 50% Empathy 5 34% 41% 44% 49% Solving Conflict 3 32% 31% 33% 33% Total Socio-Emotional Index 4 41% 42% 43% 47% N 139 139 135 140

IV. Socio-emotional skills scores, by group On average children had a score of 1.7 out of 4 possible points (43% correct) Figure 4: Baseline Socio-Emotional Skills, by Group 47% 50% 43% 42% 45% 41% There were no significant differences between groups 40% 35% 30% 25% 20% 15% 10% 5% 0% Comparison ELM Only HEART OnlyHeart & ELM Note: Asterisks relate to group with the highest score; * p <.05, ** p < .01, *** p < .001

IV. Overall IDELA score, by group On average children had an IDELA score of 6.9 out of 20 possible points (36% correct) Figure 5: Total IDELA 50% 45% 39% 40% 35% 35% 34% 35% 30% No significant differences between groups 25% 20% 15% 10% 5% 0% Comparison ELM Only HEART Only Heart & ELM Note: Asterisks relate to group with the highest score; * p <.05, ** p < .01, *** p < .001

IV. SEARS-PRE scores, by group On average teachers rated children with a SEARS-PRE score of 57 out of 74 possible points (77%) Figure 6: Total Sears-Pre Score 70.0 58.2 57.6 56.9 60.0 55.6 50.0 40.0 There were no significant differences between any of the groups 30.0 20.0 10.0 0.0 Comparison ELM Only HEART Only Heart & ELM Note: Asterisks relate to group with the highest score; * p <.05, ** p < .01, *** p < .001

IV. SEARS-PRE IDELA Relationships Figure 7: Relationship between Sears-Pre & IDELA Scores Overall there is a significant, but small, relationship between teachers ratings of children s socio-emotional development and IDELA scores. This relationship is found for the total IDELA score, and the Motor and Numeracy sub-Indices This result is driven by children in the ELM Only group. No significant relationships exist between the teachers rating and the IDELA scores for the other groups. 0.45 0.4 IDELA Score 0.35 0.3 0.25 0.2 20 30 40 50 60 70 80 Sears-Pre Score IDELA Motor Development Early Numeracy

V. Conclusion and Next Steps Conclusions Only one significant difference between the groups (HEART & ELM scored significantly higher than HEART Only on emergent literacy sub-index), but none on total IDELA score Baseline scores will be controlled for in follow-up analyses Few significant differences validates the selection of the groups, especially the control, and sets up for a clear endline & gains analysis Next Steps Follow-up study scheduled for June 2016 to measure gains and assess impact Close monitoring of teacher trainings, inputs, and other project activities of all sample schools (including comparison) in the meantime av

www.savethechildren.org Lauren Pisani (lpisani@savechildren.org) Cindy Sobieski (csobieski@savechildren.org)

FINDINGS FROM SEARS-PRE AND IDELA OPEN-ENDED RESPONSES

Purpose of open-ended responses Open ended responses: Are helpful in new/ pilot studies to identify context-specific considerations Can help identify unique, unidentified & relevant information about the population being studied Can corroborate, explain, and enrich the quantitative data Identify new questions and/ or categories of relevance to the study

SearsPre: Teacher Open Response on Student Table 6. SearsPre Teacher Open Response on Student HEART Only 76% Heart & ELM 82% Comparison ELM Only 65% At Least One Comment from Teacher Student Challenges: Family Adversity Physical Disability Learning Disability Emotional Difficulties Often Absent Student Strengths/Support: Family is Committed to School Academic Strengths Emotional Strengths N 82% 29% 3% 9% 21% 1% 22% 4% 7% 15% 1% 24% 2% 4% 8% 2% 32% 7% 8% 19% 2% av 37% 51% 44% 139 41% 55% 54% 138 39% 60% 59% 131 48% 60% 53% 139 No statistically significant differences

Sears-Pre Open-Ended Responses: Student Challenges Figure 8: Challenges Students Face 35% 30% Family Adversity most common, followed by Emotional Difficulties No significant differences between the groups 25% 20% 15% 10% 5% 0% Comparison ELM Only HEART Only Heart & ELM Family Adversity Learning Disability Often Absent Physical Disability Emotional Difficulties

Sears-Pre Open-Ended Responses: Student Strengths & Family Support Figure 9: Student Strengths/Support 70% Across all groups, teachers had many positive things to say No significant differences between the groups Similarity indicates likely comparability of the groups, good for the endline analysis 60% 50% 40% 30% 20% 10% 0% Comparison ELM Only HEART Only Heart & ELM Family is Commited to School Academic Strengths Emotional Strengths

Analytic strategy for Free Drawing Set of responses was reviewed to identify overall themes Each non-response was coded as a 0 Each response was coded by a number assigned to recurring themes in the responses Frequencies were then tallied for each question by group

Completion of the Free Drawing Figure 10: % of Children Completing Free Drawing 100% Children that completed both the drawing and were able to describe it Heart & ELM group drew significantly more than HEART Only group Several students in all groups declined to do the free drawing (16% in Comparison, 7% in ELM Only, 15% in HEART Only, 5% in Heart & ELM) 95% 95% 93% 90% 85%* 84% 85% 80% 75% Comparison ELM Only HEART Only Heart & ELM Note: Asterisks relate to group with the highest score; * p <.05, ** p < .01, *** p < .001

Themes in the free drawings Table 7: Free Drawing Themes, by Group HEART Only 7% 37% 14% 7% 14% 1% 4% 0% 15% 135 Heart & ELM 4% 29% 22% 12% 14% 4% 11% 0% 5% 140 Comparison ELM Only 5% 20% 22% 7% 16% 4% 9% 1% 16% 139 Buildings (House, School) People (Boy, Girl, Mother, Friend) Nature (Trees, Animals) Letters/Numbers/Shapes Vehicles/Roads (Car, Bus, Airplane) Food Things (Ball, Pencil, Doll) Other Did Not Draw N 6% 32% 19% 12% 9% 6% 7% 1% 7% 139 People and Nature the most common items drawn

Listing Preferences Table 8: Number of Preferences Listed, by Group HEART Only Heart & ELM Comparison ELM Only # of Items Named that Child Likes to Eat # of Items/Games Child Likes to Play With/Play N 3.8 4.0 3.5* 4.1 1.6* 139 1.8 139 1.4** 135 2.1 140 Note: Asterisks relate to group with the highest score; * p <.05, ** p < .01, *** p < .001 Generally, children named more items they liked to eat than games they liked to play/toys they liked to play with Children in the HEART & ELM group named significantly more food preferences than children in the HEART Only group Children in the HEART & ELM group named significantly more game/toy preferences than children in the HEART Only group and children in the Comparison group

Themes in Strengths Table 9: Strengths Named, by Group HEART Only Heart & ELM Comparison ELM Only Strengths Named: Reading/Writing Dancing Sports/Athletics Reflective/Gets Along Well W/Friends Helping at Home (Cooking, Farming) # of Strengths Named N 10% 12% 37% 21% 26% 1.4 139 12% 11% 37% 18% 26% 1.4 139 16% 8% 31% 13% 24% 1.5 135 18% 11% 36% 20% 35% 1.5 140 Only top 5 most common strengths listed; No significant differences between the groups Overall, 9% of children could not name any strengths. There were no significant differences between groups.

Next Steps Endline analysis in June 2016 Will allow for comparison of teacher responses between groups, and how closely they are connected to actual student performance Can compare gains in response rates, and in the number of responses, across groups Can look for meaningful shifts in qualitative data between groups.