Leadership Candidate Skill Development and Assessment Insights

Performance Assessment for Performance Assessment for Leaders: Leaders: Lessons learned from California Lessons learned from California and Massachusetts assessment design and and Massachusetts assessment design and implementation implementation New York State Education Department Latham, New York January 31, 2018

What are the considerations to determine leadership candidate skill development and readiness? What is ascertained? Leadership skills and indicators Skill level of initial leaders How candidates demonstrate skills? Authentic work Scoreable work products How skills are evaluated? Rubrics and scoring systems Trained, reliable skilled scorers

Assessment options Options Advantages Limitations Constructed response, short answer test Easy to administer and standardize Can ascertain reliability Limited to knowledge assessment Lacks authenticity as measure of leadership Simulations Easy to standardize and score performance Can ascertain reliability Lacks authenticity as a measure of leadership Limits performance Small scale assessments, such as course based or internship based, program based Potential to be more authentic and cover many standards Harder to standardize Lacks validity and reliability Cannot compare performance among programs Performance assessment tasks Can be standardized and score performance Can ascertain validity and reliability Somewhat more costly

Theory of performance assessments for professional readiness Attributes: Authentic Constructivist, educative Scalable to reflect views of professional readiness Can be used to sample important performance standards Can assess multiple skills in combination Use in teacher preparation and medical preparation Efficacy depends upon the validity and reliability of the assessment system: Components Candidate supports scoring

Ensuring valid and reliable Assessments: Development Process Design Implementation and standards setting Development and readiness Fall and spring pilot Field trial

Alignment State and national standards for educational leaders Leadership preparation program accreditation Preparation standards Preparation assessment requirements Job requirements and state licensure tiers State school improvement and educational priorities

Components of the Performance Assessment for School Leaders in Massachusetts Four assessment tasks Represent the primary educational improvement levers of school leaders Based on MA Leadership Standards and Indicators 3-4 artifacts and commentary produced per task 3-4 rubrics with 6-8 indicators for scoring each task

PAL Assessment Task Components Four parts to the work for each task Three types of products Artifacts Other documents Commentary Four-point rating scales Beginning Developing Meeting Exceeding Investigate Assess Prepare Act

Task 1: Leadership through a vision of high student achievement Steps Work products Artifact #1 Priority area and its context Artifact #2 Plan for action strategies Artifact #3 Findings, feedback and recommendations Commentary leadership skills developed and learning experienced Collect and analyze 3-5 years of quantitative student performance indicators, qualitative indicators of school culture and student learning. Identify a priority student learning area of interest with attention to federally-designated priority student groups Collect additional qualitative and quantitative information for the priority student learning area, Document existing school programs, services, and practices and ascertain the gaps and opportunities for improvement. Develop action strategies to improve student learning in the priority student learning area. Seek leadership feedback and support

Task 2: Leadership for a Professional Learning Culture Steps Work products Document how teacher teams and groups are used in the school. Form a professional learning group of teachers Select a priority academic area and focus for improvement Engage group in meetings and support teachers collectively and individually on improving practice Collect evidence on new or improved practices and analyze Solicit team feedback on their learning and improvement Artifact #1 identify priority area, professional group and professional learning plan Artifact #2 describe the learning process and work accomplished by the group, emphasizing candidate s role Artifact #3 present group member feedback on the process, learning and benefits Commentary leadership skills developed and learning experienced

Task 3: Leadership in observing, assessing and supporting individual teacher effectiveness Steps Conduct a pre-observation conference. Document a teacher observation using a district guide on effective teaching practices Describe teacher performance using the rubric indicator performance levels. Conduct a post-observation conference that facilitates teacher rapport and learning Provide constructive feedback and strategies for improvement in a post-observation conference Collect and analyze teacher feedback on the effectiveness of the observation, feedback and support Work products Artifact #1 Preobservation template Artifact #2 Teacher observation video recording Artifact #3 Post-observation meeting video recording Artifact #4 Analysis of observed teaching Artifact#5 Teacher feedback on observation and post-conference Commentary leadership skills developed and learning experienced

Task 4: Leadership for family engagement and community involvement Steps Identify a priority area for improving family and community engagement that would directly or indirectly enhance student learning in a priority area Engage staff, leaders and family and community leaders as a planning group to enhance family and community engagement Create a multi-strategy plan on how to improve family and community engagement. Implement one planned strategy Gather and analyze feedback and other evidence on the plan and strategy s effectiveness for improving family and community engagement Work products Artifact #1 Analysis of data, priority area and plan Artifact #2 Implementation of one strategy Artifact #3 Feedback on plan and strategy implementation Commentary leadership skills developed and learning experienced

PAL assessment validity and reliability The four tasks are valid and reliable assessments of initial leadership readiness PAL has had positive intended outcomes: Educative (yielded expected learning benefits for candidates) Evaluative (identified candidates who are ready for initial leadership) Beneficial for school settings (lead to improvements based on authentic work completed) Informative (lead to positive changes in preparation programs content, organization and internships)

California Administrator Performance Assessment (APA) Similar to Massachusetts Three cycles parallel Tasks 1-3 of Massachusetts PAL Followed similar assessment development process Design committee Pilot one cycle per candidate in 2017 Field trial for all three cycles per candidate statewide, 2017-18. Differences Greater emphasis on addressing diversity and cultural responsiveness Cycle 3 focuses on coaching an individual teacher, rather than observing and providing feedback using a teaching rubric Narrower focus on initial school leader readiness (Tier 1 in California)

Commonly raised assessment use questions Can these assess candidates readiness in all types of school and district settings? In Massachusetts, candidates submitted work products from all school levels (elementary through high school), from all settings (rural to urban) and all types of nontraditional schools (juvenile justice, special education, and vocational education) Will districts support candidates in completing authentic work in their settings? What about confidentiality considerations? Communication with school and district leaders about the assessment expectations helped to create field readiness Leadership preparation program faculty were provided resources to use in helping to structure field experiences and trouble shoot possible challenges All candidates had to adhere to professional standards on the ethnical conduct of research, observations, and program evaluation, as part of their professional preparation

Considerations (continue) Will candidates be able to video record a teacher observation without union and parent objection? Among the support materials provided were consent forms for candidates to use in gaining permission for video-recording teacher observations. Schools and districts were already prepared for video-recording teacher observations as part of edTPA or other teacher candidate assessments and teacher professional development practices. Why can t preparation programs score their own candidates? Preparation programs are strongest in conducting formative assessments of their candidates in their growth and development and in focusing on developing specific skills Preparation programs address the full range of leadership standards in their programs content and internships but may vary widely in their depth, breadth and effectiveness A common performance assessment, that is summative and focuses on a core set of leadership skills, can provide evidence of readiness and an independent check on preparation quality and effectiveness

Considerations and related effects PAL Assessment development Strengthen instructions, scorer training and supervision Testing out use of cut-scores and fee payment on submission quality Early work shows greater quality, narrowing the performance range Fewer candidates are seeking licensure Irregularity problem Candidate and program benefits Learning and readiness benefits reported by most candidates Program alignment and improvement reported by many programs Gradual reshaping school and district expectations for leadership preparation Other states are interested in replication

Implementation Year Outcomes Fewer candidates completed all four tasks during the Program Year than the field trial Tasks continue to be rated by candidates and programs as relevant, feasible, educative and beneficial. Total average scores were slightly higher (12% scored below 2.5 and very few below 2.1 threshold) Task 4 remained most challenging Program candidates performed somewhat better than nonprogram candidates, especially for Task 1 and 4.

Further information For information on the research, contact Margaret Terry Orr at Fordham University (morr4@Fordham.edu) Orr, M. T. & L. Hollingworth (forthcoming). How Performance Assessment for Leaders (PAL) Influences Preparation Program Quality and Effectiveness. Journal of School Leadership and Management. Orr, M. T., Hollingworth, L., & J. Cook (forthcoming). Large-Scale Performance Assessment for Leaders in California: A Pilot Study, Journal of School Leadership. Orr, M. T., Pecheone, R., Hollingworth, L., Beaudin, B., Snyder, J., Murphy, J. (2017). The Performance Assessment for Leaders: Validity and Reliability Evidence. Journal of Research on Educational Leadership. http://journals.sagepub.com/doi/full/10.1177/1942775117742646 Orr, M. T., Pecheone, R., Snyder, J, Murphy, J., Palanki, A., Beaudin, B., Hollingworth, L, & J. Buttram (April 2017). Performance assessment for principal licensure: Evidence from content and face validation and bias review. Journal of Research on Educational Leadership