Assessment and Evaluation in General Education: Integrative Studies Overview

General Education Assessment and Evaluation: Model and Tools for Integrative Studies Course Faculty Barbara Masi Learning Outcomes Assessment Office Co-chair, Joint Committee on General Education Assessment May 15, 2017 COURSE ASSESSMENT DESIGN : loa@psu.edu, 814-863-8721 Office of Planning and Assessment 1

Topics Assessment and evaluation of general education: Oversight structure Overall Gen Ed goals, metrics and measures relevant to Integrative Studies courses Integrative Studies course assessment and evaluation: Course goals, metrics, measures Course assessment and evaluation criteria Course faculty tools: Course memo: Alignment of course objectives, activities, assessments with Gen Ed objectives Course faculty reflection Student and faculty surveys Interdisciplinary writing rubric Course design and assessment handbook Resources and support Office of Planning and Assessment 2

Gen Ed Assessment and Evaluation Plan: Gen Ed Office, Learning Outcomes Assessment Office and Joint Committee on General Education Assessment With oversight by Faculty Senate Curricular Affairs Responsible for implementation of General Education Assessment and Evaluation Plan Established goals, metrics and measures relevant all Gen Ed, including Integrative Studies courses Office of Planning and Assessment 3

Gen Ed Goals: Provide solid foundation of core competencies and breadth of knowledge Strengthen links between foundational core competencies and upper level studies along a developmental trajectory Provide opportunities for transformative exploration Provide opportunities for integrative learning University curriculum oversight mechanisms will guide and support achievement of general education learning objectives Office of Planning and Assessment 4

Gen Ed Assessment Model: Importance of Course-Level Assessment and Evaluation Course quality How does general education course design and assessment impact achievement of general education objectives at program level? Program quality Tools engagefaculty in assessment and evaluation of how integrative studies course design impacts learning Focus on authentic assessments at course level as instances of integrative thinking evidence Focus on faculty reflection on pilot course development and implementation: What did faculty learn about Integ.Studies course design, how to engage students, and path as teachers? Office of Planning and Assessment 5

GOAL for curriculum: Provide GOAL for curriculum : Provide solid foundation of core competencies and breadth of solid foundation of core competencies and breadth of knowledge knowledge ACTIVITIES METRICS MEASURE General Education Courses ALL GEN ED COGNITIVE LEARNING OBJECTIVES: Students will embark on developmental path toward achievement of gen ed objectives DIRECT MEASURE, IN COURSE: Course assessments that align with SELECTED gen ed objectives (course memos) DIRECT MEASURE, CROSS-COURSE: Assessment designed by University or external to measure one or more gen ed objectives (e.g. CAT Critical and Analytical Thinking Test) INDIRECT MEASURE: University-level student survey of development, connection to major development INDIRECT MEASURE: University alumni survey of development, connection to post- grad career paths EDUCATION METRIC: Each student will have sufficient opportunity to achieve all Gen Ed Objectives MEASURE: Gen ed course enrollment data set organized to examine course enrollment by domain, major, campus and student academic term Office of Planning and Assessment 6

GOAL for curriculum: GOAL for curriculum : Provide opportunities for transformative Provide opportunities for transformative exploration exploration ACTIVITIES METRICS MEASURE General education courses ALL GEN ED AFFECTIVE LEARNING OBJECTIVES: Students will be both motivated to engage, and find value in gen ed learning experiences that support their personal and professional growth INDIRECT MEASURE: University-level student survey of attitudes, quality of learning experiences INDIRECT MEASURE: Alumni survey of attitudes, connection to post- graduation career paths MEASURE: Course enrollment data will show students choosing wider range of courses to fit personal learning and topic interests EDUCATION METRIC: Students will choose to engage in gen ed courses that link to engaged scholarship experiences MEASURE: Dataset that links gen ed course enrollment with linkages to engaged scholarship experiences Office of Planning and Assessment 7

GOAL for curriculum: Provide opportunities for transformative exploration GOAL for curriculum : Provide opportunities for transformative exploration Activity Metric Measure Faculty development workshops / consultations with Gen Ed Office, Fellows, TLT, Libraries, SITE, LOA Faculty will be effectively and efficiently supported in development of gen ed courses that utilize knowledge of pedagogy and assessment to support achievement of Gen Ed objectives MEASURE: Faculty survey will evaluate faculty satisfaction with professional development opportunities Office of Planning and Assessment 8

GOAL for GOAL for Integ Provide Provide opportunities for integrative opportunities for integrative learning Integ. Studies: . Studies: learning- - Cognitive Cognitive Sub-Goal METRICS MEASURE Students will develop integrative thinking abilities as a result of integrative studies course participation Evaluation of in-course assignments by course faculty will show student demonstration of integrative thinking outcomes Evaluate course assignments using course-specific grading scheme (course materials + course memo) Evaluation of assignments across course assignments will show student demonstration of integrative thinking outcomes Evaluate course assignment selection using integrative thinking rubric (Gen Ed review committee) Students will demonstrate a high degree of confidence to complete tasks that require integrative thinking Integrative Studies student course survey (LOA office) Office of Planning and Assessment 9

GOAL for GOAL for Integ Provide Provide opportunities for integrative learning opportunities for integrative learning- - Affective Integ. Studies: . Studies: Affective Sub-Goal METRICS MEASURE Students will develop integrative thinking abilities as a result of integrative studies course participation Students will demonstrate positive attitudes about the importance of utilizing integrative thinking in tackling complex issues, events and ideas or creation of new ideas Integrative Studies student course survey (LOA Office) Students will demonstrate positive attitudes about the role of integrative thinking in Gen Ed and overall undergraduate education. Students will demonstrate ability to consciously reflect on personal and disciplinary bias and the role such bias may play in consideration of issues, events, and ideas as well as the development of ideas or solutions. Integ. Studies course faculty reflections on course (course memo) Students will demonstrate high motivation to engage in integrative learning activities associated with Gen Ed courses. Office of Planning and Assessment 10

GOAL for GOAL for Integ Provide Provide opportunities for opportunities for integrative learning Integ. Studies: . Studies: integrative learning- - Evaluate Evaluate Sub-goal Students will develop integrative thinking abilities as a result of Integrative Studies course participation METRICS Students will rate quality of course teaching, activities, and assessments as supporting their learning MEASURE Integrative Studies student course survey (LOA Office) Faculty will use course activities that support development of integrative thinking outcomes Integrative Studies student course survey (LOA Office) Students will have sufficient opportunities to develop integrative thinking outcomes Integ. Studies course assessment evidence, other course artifacts and faculty reflections on course (course materials + course memo) Faculty will choose appropriate course activities and assessments that fit student prior knowledge and abilities Faculty will have sufficient support and resources to develop high quality Integrative Studies courses Faculty survey will evaluate faculty satisfaction with professional development opportunities and course resources Office of Planning and Assessment 11

Gen Ed Objective: Integrative Thinking The ability to synthesize knowledge across multiple domains, modes of inquiry, historical periods, and perspectives, as well as the ability to identify linkages between existing knowledge and new information. Individuals who engage in integrative thinking are able to transfer knowledge within and beyond their current contexts. Office of Planning and Assessment 12

Integrative Thinking: Assessment Criteria and Learning Objectives (1) (course memo) Disciplinary grounding Degree to which student work is grounded in carefully selected and adequately employed disciplinary insights IT1. Define/ recognize/ apply disciplinary theories, findings, examples, methods, validation criteria, genres, communication forms Advancement through integration Degree to which disciplinary insights are clearly integrated IT2. Use integrative structures such as conceptual frameworks, graphic representations, models, metaphors, explanations, solutions that result in more complex, effective, empirically grounded or comprehensive accounts or products than would have been possible under single disciplinary framework. (1) Boix Mansilla & Duraising, 2007 Office of Planning and Assessment 13

Integrative Thinking: Assessment Criteria and Learning Objectives (1) (course memo) Critical awareness Degree to which work exhibits clear sense of purpose, framing issue in ways that invite interdisciplinary approach IT3. Frame problems or solutions in ways that show reflection on choices, opportunities, compromises by taking interdisciplinary approach IT4. Exhibit awareness of of disciplinary contributions, how disciplines are integrated, limitations of integration IT5. Recognize personal and disciplinary bias and the role such bias may play in framing of issues, events, ideas or works as well as the development of ideas or solutions (optional). (1) Boix Mansilla & Duraising, 2007 Office of Planning and Assessment 14

Integrative thinking: How-to inspirations embedded in Integ. Studies course memo Boix Mansilla et al. 2009: Course structure framing for integrative thinking, rubric to assess integrative writing (and other works) Golding 2009: The best how-to design integrative studies courses ever Asso. For Integrative Studies and Institute in Integrative Studies: Guide to Syllabus Preparation Penn State Gen Ed Assessment Committee: Suggested rubrics and assessments for Key Learning Objectives Pull it all together: Course memo Course design rationale: The hook Alignment is crucial: Gen Ed and course objectives, course objectives with activities, assessments Assessment tools: What rubrics will you use? Boix Mansilla writing rubric? Gen Ed Key Learning Objective suggested rubrics? Formative assessment is crucial to integrative disciplinary perspectives, tools to create integrative thinking works Reflection on what happened! Reflection as teacher of integrative thinking! Office of Planning and Assessment 15

Integrative Thinking: Evaluation Criteria (2) (course materials and course memo) Course curriculum clearly presents: Main purpose: What is significant topic, issue, or product to be approached in interdisciplinary fashion? Reason for interdisciplinary approach: Why is an interdisciplinary approach valuable or necessary for this issue(s) or topic? Is there a hook or grabber at the beginning that draws students into the issue, motivating them to learn about it? Integrative objects for student work: What is aim of taking integrative approach? What are students expected to produce? Interdisciplinary tasks: What sort of interdisciplinary tasks will students need to make to produce the integrative objects? Is integration ongoing throughout the course after presentation of disciplinary perspectives? Disciplines to be integrated: Why is it important for interdisciplinary work on this issue? What substantial contribution does each discipline make? How does each present a clearly distinct perspective, mode of knowing and inquiry? What would be missing if this discipline were not represented? Course structure: Does the syllabus serve as a map of, or orientation, to the course? Do tools, readings, message for each week reinforce each other and take students on a developmental path toward integrative thinking? Office of Planning and Assessment From: Golding 2009 and Asso. Integrative Studies and Instit. in Integ. Studies syllabus guide 16

Curriculum content: Evaluation Criteria (course materials and course memo) Presents multiple perspectives and ways of knowing Has clear interdisciplinary expectations and aims Gives students clear understanding of what interdisciplinary inquiry is and its importance Makes explicit each different way of knowing presented in subject Explicitly describes integrative works students will need to produce Explicitly describes tasks needed to synthesize perspectives and produce integrative works Has assessment tasks that reflect course learning objectives Office of Planning and Assessment 17

Its all connected: Course memo asks faculty to connect the pieces What course activities (instructional strategies, pedagogies) will help me to achieve my course objectives with all of my students? What do I want students to know and do with course content? What attitudes and motivations do I want them to develop that will help them learn in my course (and beyond)? What student assessments will answer my questions? Wiggins, McTighe Understanding by Design Office of Planning and Assessment 18

Integrative thinking: General course structure Integrative thinking: General course structure o Learning synthesis: connect frameworks & tools of analysis and creation o Students may have learned to disconnect frames and tools Domain 1 frame, theory, methods Domain 2 frame, theory, methods Integrative frame to examine or create phenomena, ideas, events, works o Support students in command of each domain and integrating domains. Office of Planning and Assessment 19 19

Course memo basis: Course learning outcomes assessment process Course memo basis: Course learning outcomes assessment process TEACHING AS RESEARCH Office of Planning and Assessment 20

Course reflective memo See memo handout. Goals: Gather course artifacts (due Fall 2017 for Integ. Studies courses) Align course learning objectives, activities, and assessments with Gen Ed objectives and Integrative Thinking Objectives (due Fall 2017) Assessment evidence and findings (due after course implemented) Reflect on pilot course: What happened, what will I change next time? (due after course implemented) Office of Planning and Assessment 21

Course reflective memo: Artifacts and alignment (due Fall 2017) Course artifacts: Syllabus, readings, lecture notes, assignments, exams Alignment of course and Gen Ed learning objectives: Knowledge Domain Objectives: 3 out of 5 per Domain (6 total) Key Objectives: 2 out of 7; minimum of 1 more since must choose integrative thinking Integrative Thinking learning objectives (5 total, 1 optional) Alignment of course learning objectives with course activities and assessments Office of Planning and Assessment 22

Alignment of course and Gen Ed Objectives (course memo) COURSE LEARNING OBJECTIVE Gen Ed Knowledge Domain Objectives Addressed GH2, GH3 Gen Ed Key Learning Objective Addressed Gen Ed Integrative Thinking Objectives Addressed Example: 1. Students will be able to use methods of literary analysis to examine science fiction novel and produce an essay. K1, K2, K3 IT1 Example: 2. Students will be able to discuss use of science concepts in science fiction novel with peers. GN1 K1, K2 IT1 Example: 3. Students will be able to analyze use of science concepts as a literary device in early 20th century science fiction novels in presentation to class and leading class discussion. GH1, GN1 K1, K2, K3, K4 IT2 Office of Planning and Assessment 23

Alignment of course learning objectives, activities, assessments (course memo) Helps you to design course, great visual of linkages Course learning objectives Course activity Course assessments and scoring scheme Assessment 1: 500 word essay uses literary analysis method. 1. Students will be able to use methods of literary analysis to review science fiction novel and produce an essay. Class discussion of Moby Dick in order to learn about process of literary analysis from instructor and peers Essay scored using rubric. 3. Students will be able to analyze use of science concepts as a literary device in early 20th century science fiction novels and present analysis in oral presentation to class and lead class discussion. Collaborative in-class team project to develop ideas for novel analysis for presentation to class. Team must also develop questions for class discussion. Assessment 2: Team analyzes science fiction novel and create presentation and discussion questions. Assessed using rubric. Office of Planning and Assessment 24

Integrative thinking Integrative thinking- - assessment methods assessment methods How to Assess Student Work Learning Objective Examples of Types of Assessment Require students to evaluate, judge or critique readings, performances, or products against established criteria or standards. Examples: Journals, Critiques, Case Studies, Debates Critically analyze or evaluate idea using theories/ methods of integrative knowledge domains Analytic or Holistic Rubrics scored by instructor, juries Create using theories/ methods of integrative knowledge domains Creative works, research projects, essays, service learning project as individuals or groups. Rubrics, scored by instructor, etc. Office of Planning and Assessment 25

Rubric to assess interdisciplinary writing (can modify for other works) See LOA handout, Boix Mansilla rubric 2009 Element Does framing invites integrative approach? Does work use writing genre effectively to communicate with intended audience? Use disp. knowledge accurately and effectively? Use disp. methods accurately and effectively? Include selected disp. perspectives or insights from 2/more disp. Traditions that a relevant to work s purpose? Is there an integrative device or strategy? Is there balance in how students bring disp. perspectives or insights together? Is there awareness of limitations/ benefits of contributing disciplines and how they intertwine? Office of Planning and Assessment Does student exhibit self-reflection? 26

Key Learning Key Learning O Objectives: Suggestions for Assessment (coming soon!) bjectives: Suggestions for Assessment (coming soon!) 1. Effective Communication The ability to exchange information and ideas in oral, written, and visual form in ways that allow for informed and persuasive discourse that builds trust and respect among those engaged in that exchange, and helps create environments where creative ideas and problem-solving flourish. Assessment Ideas Suggested Scoring Criteria Debate Oral communication Organization Language choice Delivery Supporting material Central message Examples Analyze passage of argumentation in text. Highlight main argumentative moves. Evaluate argument logic and discuss strengths and weaknesses. Presentation provide other students opportunities to interact with speaker Blog that allows comments from other students Student group works through statistics case study related to their discipline with faculty who is also from their discipline. Case study analysis shared with classmates (VALUE rubrics; Oral communication, Written communication, https://www.aacu.org/value/rubrics) Video production shared with class to facilitate interaction Office of Planning and Assessment 27

Assessment method suggestions: Lower order objectives Learning Objective Remember recall recognize Examples of Types of Assessment How to Measure Accuracy correct vs number of errors Multiple Choice items Reciting (orally, in writing) Scoring or performance rubrics that identify critical components of the work and discriminates between differing levels of proficiency in addressing the components Comprehend interpret compare explain Compare/ contrast 2 theories, events, processes, etc. Identify examples of a concept, principle Activities that require students to use procedures to solve or complete familiar or unfamiliar Apply execute implement Accuracy scores, check lists Office of Planning and Assessment tasks. 28

Assessment method suggestions: Higher order objectives Learning Objective Examples of Types of Assessment How to Measure Require students to test, monitor, judge or critique readings, performances, or products against established criteria or standards. Examples: Journals, Critiques, Case Studies. Evaluate check critique Analytic or Holistic Rubrics scored by instructor, juries, external client, employer etc. Create generate plan produce Research projects, essays, business plans, website designs Rubrics, scored by instructor, etc. Office of Planning and Assessment 29

Reflection: What happened and why? (course memo, after course) What prior student knowledge or abilities did you assume for the course? Were issues, events, ideas, or creative works chosen for course appropriate for development of integrative thinking outcomes? Did classroom activities support student achievement of course objectives? Did course assessments balance scaffolding or formative assessments so that students could achieve integrative thinking outcomes? What will you change next time? What did you learn as a teacher about teaching Integrative Studies courses? Office of Planning and Assessment 30

Integrative Studies Course: Student Survey Delivered by LOA Office via Qualtrics to students Questions: Course quality: teaching, activities, assessments as contributing to Integrative Thinking ability development Self-assessment: Gen Ed objectives (Domain, Key, Integrative Studies) Suggestions for improvement Attitudes about integrative thinking, gen ed curriculum and contribution to undergraduate education Survey logic: Add questions that are course specific Incentives to complete survey Office of Planning and Assessment 31

Integrative Studies Course: Faculty Survey Post-course: Satisfaction with professional development and resources for course development and delivery Office of Planning and Assessment 32

Add an indirect assessment Mid-course survey? How am I doing? What can I improve? Two questions, on paper in class Office of Planning and Assessment 33

Another note: Formative vs. Summative Assessment Integrative thinking o scaffolding = formative assessment oProvide opportunities to practice and develop proficiency in each disciplinary domain and integrative thinking at course level Formative assessment methods Integrative thinking will be new for students. Provide multiple opportunities to observe how well students are developing Requires faculty observation of students practice (discussions, in class collaborative work, presentations) Faculty provide constructive feedback so that a student can improve performance before end of course May or may not be graded Summative assessment methods Formal assessment or judgment of student work, e.g. final course works Office of Planning and Assessment 34

Essential resources and LOA support General Education Assessment and Evaluation of Integrative Studies Courses: Course memo Golding 2009: Integrative studies course design handbook Boix Mansilla et al. 2009 Targeted assessment rubric: An empirically grounded rubric for interdisciplinary writing The J. of Higher Education 80(3) 334-353. Other super books or papers: W. Newell 1990, Interdisciplinary Curriculum Development. Issues in Integrative Studies No. 8, 69- 86. Wiggins and McTighe 1998, Understanding by Design. Angelo and Cross, Classroom Assessment Techniques (easy to use formative assessment techniques) H. Stewart-Gambino and J. Stroud Rossmann. Often Asserted, Rarely Measured: The Value of Integrating Humanities, STEM and Arts in Undergraduate Learning. National Academies of Sciences, Engineering and Medicine, 2015. LOA SUPPORT loa@psu.edu , 814-863-8721 Office of Planning and Assessment 35

Questions and support Department of Learning Outcomes Assessment 814-863-8721 loa@psu.edu Office of Planning and Assessment 36