Advancing STEM Teacher Practices and Research Instruments Review
Explore a review of research instruments for STEM teacher practices, PCK, and content knowledge presented by CADRE. Learn about project characteristics, funded types, research conducted, and more. Data from 248 projects, project materials reviewed, RFP guides on STEM education innovation, and disciplines addressed in projects are highlighted.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
COMMUNITY FOR ADVANCING DISCOVERY RESEARCH IN EDUCATION (CADRE) A Review of Research Instruments for STEM Teacher Practices, PCK, and Content Knowledge PRESENTED BY DAPHNE MINNER ALINA MARTINEZ Abt Associates JUNE 15, 2012 C A D R E DR-K12 PRINCIPAL INVESTIGATOR ANNUAL MEETING ARLINGTON, VA
DR K-12 Portfolio Overview Purpose Describe important characteristics Provide evidence of progress toward goals Identify areas for targeted studies Inform CADRE s support activities Questions What are the characteristics of projects? What types of projects are being funded? What research is being conducted? Are projects evaluating their work? How are projects going to disseminate their work? C A D R E June 15, 2012
Data from 248 Projects Includes projects that submitted at least a proposal Cohort 4 (71 Projects) Cohort 1 (75 Projects) Sources: Provided by PIs voluntarily in response to requests Cohort 3 (51 Projects) Cohort 2 (51 Projects) Found in the public domain C A D R E June 15, 2012
Project Materials were Reviewed and Coded Descriptions of DR K-12 portfolio characteristics were compiled for the annual report Project materials were reviewed and coded Coded information was recorded in a database C A D R E June 15, 2012
RFP Guides Innovation in STEM Education K-12 math and science assessments Elementary grade science Cutting-edge STEM content in K-12 Grand Challenges (2006) Contextual challenges Frontier challenges Challenge Strands (2008) Assessments Opportunity to learn for all students Ability of teachers Promising innovations Cyber-enables learning (2011 only) Challenge Areas (2009, 2010, 2011) Assessment Learning Teaching Scale-up and sustainability Strands (2012) C A D R E June 15, 2012
Disciplines Addressed in Projects Sciences 41% Multi-disciplinary 19% 3% Other Mathematics Engineering 2% 34% Computer 1% C A D R E June 15, 2012
Target Segment of the Education Population 100% Teachers Students 80% 80% 75% 60% 40% Other Higher Education Faculty School Administrators 20% 13% 5% 4% C A D R E 0% June 15, 2012
Outcomes Studied 100% 90% 80% Students Teachers 66% 70% 65% 60% 50% Implementation Fidelity 40% Other & Unknown Research Quality, Progress 30% 20% 14% Administrators 10% C A D R E 6% 10% 5% 0% June 15, 2012
Types of Student and Teacher Outcomes Studied Students Teachers 100% 100% 90% 90% Classroom Practices 80% 80% Achievement Pedagogical Content Knowledge 70% 70% 62% 60% 60% 49% Attitudes 50% Attitudes 50% Content Knowledge 40% 35% 40% 34% 32% Behavior 30% 30% 23% 18% 20% 20% C A D R E 10% 10% 0% 0% June 15, 2012
Teacher Instrument Study Overview Rationale Methods C A D R E June 15, 2012
75 projects assessed teacher outcomes with existing instruments Table A Table B Outcome Domain Number of Projects 32 Number of Teacher Instruments Used 1 Number of Projects Practice 35 PCK 14 2 21 PCK & Practice 8 3 9 Content 7 Practice & Content 6 4 5 6 4 PCK & Content 4 C A D R E All three 4 June 15, 2012
82 unique instruments identified 42 used to assess Practices Reformed Teaching Observation Protocol (modified) 15 studies Inside the Classroom Observation and Analytic Protocol 8 Surveys of Enacted Curriculum (modified) 5 24 used to assess Content Views of Nature of Science (Form C) 3 studies National Assessment of Educational Progress (modified) 3 Praxis content tests/ Earth & physical science (modified) 2 TIMSS content tests (modified) 2 C A D R E June 15, 2012
82 instruments identified 22 used to assess Pedagogical Content Knowledge Mathematical Knowledge for Teaching 14 studies Knowledge of Algebra for Teaching (modified) 2 Science Teaching Efficacy Belief Instrument 2 Reformed Teaching Observation Protocol (modified) 2 C A D R E June 15, 2012
More Refined Look at Constructs Instruction practices Content knowledge Instructional beliefs; teaching self-efficacy Class management Assessment practices/ skills Classroom social context Classroom physical context Administrative or policy context Student / teacher demographics Instructional planning C A D R E June 15, 2012
Instructional Practices AFM Assessment of the Facilitation of Mathematizing obs PK-6 M EMCO Early Mathematics Classroom Observation obs PK M ETAP EdTech Assessment Profile Instructional Strategies Classroom Observation Protocol LFCPO Lesson Flow Classroom Observation Protocol survey - Tech ISCOP obs mid SC obs mid SC LoFTI Looking for Technology Integration obs - Tech O-TOP OCEPT-Classroom Observation Protocol obs post SC, M STIR Science Teacher Inquiry Rubric Transforming Instruction by Design in Earth Science-- Teacher assignment quality rubrics obs elem SC TIDES rubric - SC Artifact rubric rubric Scoop Notebook mid SC C A D R E The Quality of Instruction Measure - M June 15, 2012
Instructional Practices plus The Collaboratives for Excellence in Teacher Preparation core evaluation classroom observation protocol Physical context, demo CETP-COP obs, inter K-16 SC,M EQUIP Electronic Quality of Inquiry Protocol demo obs K-12 SC,M survey, assessment KAT Knowledge of Algebra for Teaching content Mid, high M ICOT ISTE Classroom Observation Tool tech use obs K-12 Tech MQI Mathematical Quality of Instruction content obs Elem, mid M Class mgmt, assessment PRAXIS Praxis Teaching Foundations: Science survey Mid, high SC Preschool Rating Instrument for Science and Mathematics PRISM Physical context obs PK SC,M SESAME Self-Evaluation of Science and Math Education planning - PK SC,M SIOP Sheltered Instruction Observation Protocol planning obs, rubric elem Gen Third International Mathematics and Science Video Study (TIMSS) Ohio Middle Level Mathematics and Science Education Bridging Study - Teacher Questionnaire TIMSS social, demo obs, survey mid SC,M C A D R E content, demo survey K-12 SC June 15, 2012
Teachers Content Knowledge Diagnostic Teacher Assessments in Mathtematics and Science Diagnostic Mathematics Assessments for Elementary Teachers--algebra DTAMS PCK test mid SC,M DTAMS PCK test elem M instruction, assessment M-SCAN The Mathematics Scan obs elem M American College Testing Anvanced Placement Content Specialty Test Earth Science for New York Teacher Certification FACETS Diagnoser Tools ACT AP St. test St. test post high GEN GEN CST test K-12 SC St. test Mid, high SC,M High, post test post Elem, mid Elem, mid Elem, mid FCI Force Concept Inventory assessment St. test SC GRE Graduate Record Exam GEN ITBS Iowa Test of Basic Skills St. test GEN MAP Missouri Assessment Program St. test SC,M C A D R E MKT Mathematical Knowledge for Teaching test M June 15, 2012
Teachers Content Knowledge Misconceptions-Oriented Standards-Based Assessment Resources for Teachers MOSART test K-12 SC NAEP National Assessment of Educational Progress St. test K-12 GEN PISA Program for International Student Assessment St. test high GEN PRAXIS content tests/ Earth & physical science (modified) test post SC Regents New York State Regents exam St. test K-12 GEN SAT Stanford Achievement Test St. test high GEN TAGLIT Taking a Good Look at Instructional Technology test post TECH TIMSS content tests St. test Elem, mid SC,M West-E Washington Educator Skills Test-Endorsements test K-12 GEN WESTEST Science WESTEST St. test Gr. 4-9 SC American Chemical Society Division of Chemical Education Examinations Institute Classroom Test of Scientific Reasoning test post SC C A D R E St. test - SC IL Certification Testing System Study Guide-Science: Biology test post SC June 15, 2012
Instructional Beliefs IMBS Indiana Mathematics Beliefs Scale survey - M MTEBI Mathematics Teaching Efficacy Belief Instrument survey preservice M PSI-T Principles of Scientific Inquiry-Teacher survey high SC Self-Efficacy Teaching and Knowledge Instrument for Science Teachers SETAKIST survey elem SC STEBI Science Teaching Efficacy Belief Instrument survey elem SC TBI Teacher Belief Interview interview high SC TSES Teachers' Sense of Efficacy Scale survey PK-6 GEN C A D R E TSI Teaching Science as Inquiry survey elem SC VNOS-C Views of Nature of Science Form C survey - SC June 15, 2012
Multidimensional Admin context Demographics Management Assessment Practices SYSTEM-WIDE REFORM FOCUSED planning Content Physical Beliefs Social CIP Inside the Classroom Teacher Interview Protocol Inside the Classroom Observation and Analytic Protocol COP CTRI Coaching /Teacher Reflection Impact Surveys FFT Danielson's Framework for Teaching Domains LSC Core Evaluation Classroom Observation Protocol LSC SEC Surveys of Enacted Curriculum SII Study of Instructional Improvement TIMSS-R TIMSS-R Science Teacher Questionnaire C A D R E TIMSS-R TIMSS-R Mathematics Teacher Questionnaire Inside the Classroom Teacher Questionnaire: Math or Science version June 15, 2012
System-wide Reform Focused CIP Inside the Classroom Teacher Interview Protocol interview K-12 SC,M Inside the Classroom Observation and Analytic Protocol COP obs K-12 SC,M CTRI CoachingTeacher Reflection Impact Surveys survey K-8 M FFT Danielson's Framework for Teaching Domains rubric K-12 GEN LSC LSC Core Evaluation Classroom Observation Protocol obs K-12 SC,M SEC Surveys of Enacted Curriculum survey K-12 SC,M,LIT SII Study of Instructional Improvement survey elem M, LIT elem, mid elem, mid TIMSS-R TIMSS-R Science Teacher Questionnaire survey SC,M TIMSS-R TIMSS-R Mathematics Teacher Questionnaire survey SC,M C A D R E Inside the Classroom Teacher Questionnaire: Math or Science version survey K-12 SC,M June 15, 2012
Admin context Demographics Management Assessment Practices planning DIOLOGUE FOCUSED Content Physical Beliefs Social CLASS The Classroom Assessment Scoring System CLO Classroom Lesson Observation Instrument COEMETClassroom Observation of Early Mathematics Environment and Teaching DAISI The Dialogic Activity in Science Instruction EAS The Emergent Academic Snapshot The Early Language and Literacty Classroom Observation ELLCO IQA Instructional Quality Assessment Inquiring into Science Instruction Observation Protocol ISIOP RTOP Reformed Teaching Observation Protocol SPC Standards performance continuum C A D R E Classroom Snapshot Mathematics Classroom Observation Protocol Science Classroom Observation Guide (NCOSP) June 15, 2012
Dialogue Focused CLASS The Classroom Assessment Scoring System obs K-12 GEN CLO Classroom Lesson Observation Instrument obs post GEN COEMETClassroom Observation of Early Mathematics Environment and Teaching DAISI The Dialogic Activity in Science Instruction obs PK-6 M obs elem M,LIT EAS The Emergent Academic Snapshot obs PK-6 GEN ELLCO The Early Language and Literacty Classroom Observation obs, interview PK-6 LIT IQA Instructional Quality Assessment obs, artifact elem M,LIT ISIOP Inquiring into Science Instruction Observation Protocol obs, survey mid, high SC RTOP Reformed Teaching Observation Protocol obs K-16 SC,M SPC Standards Performance Continuum obs elem, mid GEN Classroom Snapshot obs K-12 GEN C A D R E Mathematics Classroom Observation Protocol obs, rubric elem, mid M,LIT Science Classroom Observation Guide (NCOSP) obs K-12 SC June 15, 2012
Reliability Information Reliability type missing information (19, 38%) internal stability alpha (11, 22%) (6 acceptable, 5 good) interrater, % agreement (7, 14%) (2 low, 2 accept., 3 good) interrater, Kappa (2, 4%) (acceptable) Interrater, Spearman rank-order correlation (1, good) Other (2, good and N/A) More than one type (8, 16%) (all acceptable to good) Level missing information <.59 or equivalent: low .6-.79 or equivalent: acceptable > .8 or equivalent: good C A D R E June 15, 2012
Validity Evidence Validity Type Missing information (28, 54%) Content/ face (11, 21%) Criterion/ predictive (9, 17%) Construct (8, 15%) Concurrent/ convergent (5, 10%) Discriminant /divergent (2, 4%) C A D R E June 15, 2012
Trends Instructional practices (n=22) 11 instruments predominantly observation 11 instruments with 1-2 other constructs, observation supplemented with survey methods Teachers content knowledge (n=25) primarily assessed domain-specific conceptual or procedural knowledge by multiple choice test majority of these instruments being used for teachers were originally designed for students 8 science, 3 math, 1 technology-specific tests C A D R E June 15, 2012
Trends cont. Instructional beliefs (n=9) Predominated by survey methods 6/9 were science specific with only 2 for math Multidimensional (n=23) System-wide reform (predominantly surveys) (10) Dialogue focused observation instruments to capture classroom climate and social aspects (13) Making reliability and validity evidence more readily available with information about instruments is essential C A D R E June 15, 2012
Discussion Surprises? Gaps? Strategies? Resources? C A D R E June 15, 2012
Contact Information Abt Associates Daphne Minner Alina Martinez Alina_Martinez@abtassoc.com Alina_Martinez@abtassoc.com Daphne_Minner@abtassoc.com Daphne_Minner@abtassoc.com (617) 520-2454 (617) 349-2312 C A D R E June 15, 2012
