Using Child Outcomes Data for Program Improvement

Explore the significance of using child outcomes data to enhance early intervention and special education programs. Learn about national data on EI and ECSE recipients, how data is utilized for policy and program enhancement, and the ultimate goal of enabling children to thrive across various settings.

• Child outcomes
• Program improvement
• Early intervention
• Special education
• Data utilization

alee_109 Follow

Uploaded on Sep 24, 2024 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

Presentation Transcript

1. Using Child Outcomes Data to Drive Program Improvement: New Developments Christina Kasprzak Frank Porter Graham Child Development Institute Kathleen Hebbeler SRI International Presented at The Division for Early Childhood Annual Conference, Atlanta, GA October 2015

2. What we will cover: What we know nationally about how EI and ECSE recipients are doing How we know this How these data are being used for policy and program improvement 2

3. General Background Emerging importance of accountability in public programs over last 20 years Not enough to have data on how many clients were served or how many hours of service were provided In special education in particular, the emphasis shifted from getting children access to services to focusing on the outcomes being achieved. ***Accountability is about demonstrating results.***

4. Outcomes and Federal Funding Every year the Office of Special Education Programs (OSEP) in the US Department of Education distributes federal IDEA funds to state Part C and Part B Preschool agencies. For 2014, Part C: Part B Preschool: All federal agencies are required to report on the outcomes achieved by their programs. In 2005, OSEP required states to report data on child outcomes so the agency could: Justify the funding for Part C and Part B Preschool. Fulfill its requirement to report data on the outcomes of Part C and Part B Preschool.

5. Ultimate Goal for EI and ECSE To enable young children to be active and successful participants during the early childhood years and in the future in a variety of settings in their homes with their families, in child care, preschool or school programs, and in the community. Based on the ECO stakeholder process when identifying 3 functional outcomes Entire document available at http://projects.fpg.unc.edu/~eco/assets/pdfs/ECO_Outcomes_4-13-05.pdf

6. Three Child Outcomes Children have positive social- emotional skills (including social relationships) Children acquire and use knowledge and skills (including early language/ communication [and early literacy]) Children use appropriate behaviors to meet their needs 6

7. Child Outcomes Step by Step Available at: http://projects.fpg.unc.edu/~eco/pages/vide os.cfm

8. OSEP Reporting Categories Percentage of children who between entry and exit: a. Did not improve functioning b. Improved functioning, but not sufficient to move nearer to functioning comparable to same-aged peers c. Improved functioning to a level nearer to same- aged peers but did not reach it d. Improved functioning to reach a level comparable to same-aged peers e. Maintained functioning at a level comparable to same-aged peers 3 outcomes x 5 measures = 15 numbers 8

9. Illustration of 5 Possible Paths 70 60 50 Score 40 30 20 10 0 1 6 11 16 21 Age in Months 26 31 36 41 46 51 56 Maintained functioning comparable to age peers Achieved functioning comparable to age peers Moved nearer functioning comparable to age peers Made progress; no change in trajectory Did not make progress 9

10. The Summary Statements 1. Of those children who entered or exited the program below age expectations in each outcome, the percent who substantially increased their rate of growth by the time they turned 3 [6] years of age or exited the program. 2. The percent of children who were functioning within age expectations in each outcome by the time they turned 3 [6] years of age or exited the program.

11. Building a National Measurement System for Child Outcomes

12. State Approaches to Measuring Child Outcomes FFY 2013-14 Approach Part C (N=56) Preschool (N=59) COS 7 pt. scale 42/56 (75%) 37/59 (63%) One tool statewide 8/56 (14%) 9/59 (15%) Publishers online analysis Other 1/56 (2%) 6/59 (10%) 5/56 (9%) 7/59 (12%) 12

13. State Approaches: Part C

14. State Approaches: Part B Preschool

15. Method for Calculating National Estimates & Criteria Weighted average of states that met minimum quality criteria Minimum quality criteria for inclusion in national analysis: Reporting data on enough children Part C 28% or more of exiters Part B Preschool 12% or more of child count Within expected patterns in the data category a not greater than 10% category e not greater than 65% 15

16. Number of States that Met Criteria for Inclusion in the National Analysis (N=51) 50 40 30 20 10 0 08-09 19 15 09-10 29 33 10-11 39 36 11-12 33 39 12-13 41 41 13-14 47 43 Part C Part B Preschool

17. The Data

18. Part C: Greater than Expected Growth 100 90 77 80 76 76 74 73 73 73 73 72 72 71 71 71 70 68 70 66 66 66 60 50 40 30 20 10 0 Social Relationships Knowledge and Skills Action to Meet Needs 2008-9 (19 states) 2011 - 12 (33 states) 2009-10 (29 states) 2012-13 (41 states) 2010-11 (39 states) 2013 - 14 (47 states)

19. Part C: Exited within Age Expectations 100 90 80 70 62 61 61 61 61 61 60 60 59 59 59 59 60 55 54 54 52 52 52 50 40 30 20 10 0 Social Relationships 2008-9 (19 states) Knowledge and Skills 2009-10 (29 states) Action to Meet Needs 2010-11 (39 states) 2011 - 12 (33 states) 2012-13 (41 states) 2013 - 14 (47 states)

20. Part B Preschool: Greater than Expected Growth 100 90 83 83 83 82 82 82 81 81 81 81 81 80 80 80 80 79 77 77 80 70 60 50 40 30 20 10 0 Social Relationships 2008-9 (15 states) 2011 - 12 (39 states) Knowledge and Skills 2009-10 (33 states) 2012-13 (41 states) Action to Meet Needs 2010-11 (36 states) 2013 - 14 (43 states)

21. Part B Preschool: Exited Within Age Expectations 100 90 80 67 67 66 66 70 65 63 60 59 59 59 59 57 60 53 53 53 52 52 51 50 40 30 20 10 0 Social Relationships 2008-9 (15 states) 2011 - 12 (39 states) Knowledge and Skills 2009-10 (33 states) 2012-13 (41 states) Action to Meet Needs 2010-11 (36 states) 2013 - 14 (42 states)

22. Pause to reflect. What do you see? How do you interpret these data? 22

23. What We See Large percentages of children showing relatively good outcomes at exit from EI and ECSE Continuing to see consistency in the data over time Increasing number of states who meet minimum quality criteria for national analysis Increasing number of children in the child outcomes data 23

24. The Quality of the Data

25. Part C: Percent of States by Completeness of Child Outcomes Data* 100 90 80 65 70 65 63 59 55 60 50 40 33 30 24 22 22 20 20 16 20 14 14 12 10 0 2009-10 (N=49) 2010-11 (N=51) < 34% 2011-12 (N=51) 2012-13 (N=51) 2013-14 (N=51) 34% - <70% 70% or more * Completeness = (total with outcomes data/total exiters) 14

26. Part B Preschool: Percent of States by Completeness of Child Outcomes Data 100 90 80 70 60 53 49 49 49 50 45 43 41 39 39 40 27 30 20 20 14 12 10 8 10 0 09-10 (N=49) 10-11 (N=51) 11-12 (N=49) 12-13 (N=49) <12% * Completeness = (total with outcomes data/child count) 13-14(N=50) 12 - <33% >33% 15

27. State Level Variation and Patterns 27

28. Part C: State Variation: Exited within Age Expectations Knowledge and Skills, 2013- 2014, All States 100% 90% National: 52% 80% 72% 70% 60% 50% 9% 40% 30% 20% 10% 0%

29. Part B Preschool: State Variation: Exited within Age Expectations Knowledge and Skills, 2013- 2014, All States 100% 90% National: 52% 80% 74% 70% 60% 50% 10% 40% 30% 20% 10% 0%

30. Part C: Exited within Age Expectations by State Percent of Exiters Not Eligible for Part B* 2013-14 - All States 100% 90% Average Percent of Children 80% 70% 61% 60% 60% 60% 60% 54% 53% 51% 50% 50% 45% 40% 30% 20% 10% 0% Social Relationships Knowledge and Skills Actions to Meet Needs <20% (N=14) 20% to <30% (N=27) 30% or greater (N=10) * Not eligible for Part B defined as the sum of the percent of children exiting as No longer eligible for Part C prior to reaching age three, Not eligible for Part B, exit with referrals to other programs, and Not eligible for Part B, exit with no referrals.

31. Part C: Average Percentage Who Exited within Age Expectations by State Percent Served*, 2013- 14 All States 100% 90% 80% Average Percent of Children 70% 64% 64% 61% 61% 60% 55% 54% 53% 52% 50% 45% 40% 30% 20% 10% 0% Social Relationships Knowledge and Skills Action to Meet Needs <2.5% (N=24) 2.5% to <3.9% (N=16) 3.9%+(N=11) *http://www2.ed.gov/programs/osepidea/618-data/state-level-data-files/ 31

32. Part C: Average Percentage Who Exited within Age Expectations by ITCA Eligibility Category, 2012-13, All States 100% 90% 80% Average Percent of Children 70% 63% 62% 59% 57% 60% 56% 54% 54% 50% 47% 50% 40% 30% 20% 10% 0% Social Relationships Knowledge and Skills Action to Meet Needs C - Restrictive (N=19) B - Medium (N=17) A - Broad (N=15) 23

33. Part B Preschool: Average Percentage Who Exited within Age Expectations by State Percent Served, 2013-14, All States (N=50)* 100% 90% 80% Average Percent of Children 66% 70% 64% 59% 58% 58% 58% 60% 54% 52% 50% 50% 40% 30% 20% 10% 0% Social Relationships Knowledge and Skills Action to Meet Needs <5.7% (N=21) 5.7% to <7.5% (N=16) 7.5%+(N=13) *child count data was not available for 1 state to compute percent served 33

34. Variation Over Time 34

35. Interpreting Year-to-Year Change Types of Change Small variations from year to year are expected Large consistent increases are good news particularly when linked to programmatic changes Large consistent decreases require explanation (e.g. changing population) Large up and down changes are an indicator of questionable data quality and require explanation 2008-09 2013-14 35

36. Part C: Trends Over Time: Greater Than Expected Growth Social Emotional, 2013-2014, All States 100 90 80 Percent of Children 70 60 50 40 30 20 10 0 2010-11 2011-12 2012-13 2013-14 Year

37. Part B Preschool: Trends Over Time: Greater Than Expected Growth Social Emotional, 2013-2014, All States 100 90 80 Percent of Children 70 60 50 40 30 20 10 0 2010-11 2011-12 2012-13 2013-14 Year

38. Part C: Statistically Significant* Change between 2012-13 and 2013-14: All States (N=51) Statistically Significant Change OC1- SS1 OC2- SS1 OC3- SS1 OC1- SS2 OC2- SS2 OC3- SS2 2 9 4 10 34 7 7 8 Negative 44 5 36 6 39 8 41 3 39 4 None Positive *p<.05 38

39. Part B Preschool: Statistically Significant* Change between 2012-13 and 2013-14: All States (N=51) Statistically Significant Change OC1- SS1 OC2- SS1 OC3- SS1 OC1- SS2 OC2- SS2 OC3- SS2 9 7 7 14 30 6 11 32 7 13 33 4 Negative 31 10 34 9 35 8 None Positive *p<.05 39

40. TA Requested by States Data Quality Increasing the number of children/families in the data Pattern checking to identify data quality issues Training, guidance, supervision, etc. 40

41. Lets Talk What do you know about your state s data? Have you seen it? Are you contributing data? Have you participated in discussions about the data? Do you have any concerns about your state s data and why? What is your state doing to improve the quality of its child outcomes data?

42. Who Uses the Data? What For?

43. Uses of the Child Outcomes Data Policy-making Justifying the amount of federal funding for the Part C and Part B Preschool Accountability Federal monitoring of states (Determinations) States monitoring of locals Program improvement State Systemic Improvement Plan (SSIP) General program improvement

44. Federal Funding Part C Preschool Special Education (Section 619) $438,498,000 $353,238,000

45. Federal and State Monitoring IDEA requires the federal government monitor how states are implementing the program the State Education Agency/Lead Agency for Part C monitor the locals For many years, focus was on compliance (access) Process: timelines, signatures, procedures followed, paperwork in place Recent shift to emphasis on the results the programs are achieving ****THIS IS A HUGE CHANGE***

46. Determinations U.S. Department of Education, Office of Special Education Programs (OSEP) issues Determinations based on its review of each state s data. Each state is found to : Meets the requirements and purposes of IDEA Needs assistance in implementing the requirements of IDEA Needs intervention in implementing the requirements of IDEA, or Needs substantial intervention in implementing the requirements of IDEA. In 2015, for the first time, the child outcomes data were examined as part of the determinations for Part C. Preschool is included under Part B (ages 3 to 21) and the preschool child outcomes data is not included in Part B determinations.

47. Part C Determinations: 2015 No. of States Determination Meets requirements 23 Needs assistance (one year) Needs assistance (2 or more consecutive years) Needs intervention (5 consecutive years) 19 13 1 Needs substantial intervention 0

48. What is the SSIP? Multi-year, achievable plan that: Increases capacity of EIS programs/LEAs to implement, scale up, and sustain evidence-based practices Improves outcomes for children with disabilities (and their families) 48

49. Who has to develop an SSIP? State Part C program State Part B program The Part B program will only need to develop one plan. Role of Part B 619 program? 619 incorporated into the Part B 619 focus for the Part B Part C and Section 619 Coordination across 0-21

50. Year 1 - FFY 2013 Delivered by April 2015 Year 2 - FFY 2014 Delivered by April 2016 Years 3-6 Delivered by Feb . 2017 Updated 2018, 2019, 2020 Phase I Analysis Phase II Plan Phase III Implementation and Evaluation Results of Ongoing Evaluation Extent of Progress Revisions to the SPP Infrastructure Development Support for EIS Program/LEA in Implementation of Evidence- Based Practices Evaluation Plan Data Analysis Analysis of State Infrastructure to Support Improvement and Build Capacity State-Identified Measurable Result(s) for Infants and Toddlers with Disabilities and their families* Selection of Coherent Improvement Strategies Theory of Action * Families are included for Part C only. Part B includes results for children with disabilities

Load More ...