Effective Assessment Methods and Quality Assurance in Education

Modes of assessment: Rationale, Strengths, and Weaknesses Prof Gavin T L Brown (gt.brown@auckland.ac.nz) Presentation to School of Biological Sciences, November 2016

Curriculum Map: Assessment Because assessment is an interaction of purposes, formats, and quality requirements, we need to consider our choices around format. Validity Quality Reliability Test-like, Formal, teacher-centric Assessment Format Interactive, informal, student-centric Improvement (Formative) Purpose Evaluative (Summative)

What is assessment? Generic definition all methods used to appraise performance of an individual or a group. a broad appraisal including many sources of evidence and many aspects of a pupil s knowledge, understanding, skills and attitudes; or to a particular occasion or instrument. An assessment instrument may be any method or procedure, formal or informal, for producing information about pupils: e.g., a written test paper, an interview schedule, a measurement task using equipment, a class quiz Gipps, Brown, McCallum and McAlister, 1995, p. 10-11

Assessment Processes: Technological Approach Content Gather Information Decide Design M ea su re Evaluate Quality Te ac h Interpretation Action Decision Plan

Gathering Information: How do we collect data? Methods Observe Behaviour Observe Performance on Set Task(s) Oral Question & Answer Paper & Pencil Questions Test, Examination One-off Collections; On-demand Prepared Many methods: all strong & weak

Threats to Quality of Assessment Relationship of Items to Content Do items fit the content of interest? Design of Assessment Does the assessment cover important things; in desired balance? Are the distribution and type of items suited to the content of interest? Item Construction Quality Are the items well crafted? Range of cognitive processes required? Range of response formats?

Alignment to Curricular Goals If you want assessment to be aligned to curricular goals, the assessment must be within the curriculum Which is better? Assessment event, task, item Assessment event, task, item Domain of interest Domain of interest

Some response type options Objectively scored: SELECTION Strengths/Weaknesses? 1. Multiple Choice Easy to mark/answer Large sample of construct efficiently Appropriate for certain kinds of knowledge domains Hard to write well Helped by recognition 2. Binary e.g., T/F, R/W 3. Category sorting e.g., find all odd numbers 4. Matching (list, terms, shapes ) 5. Order or sequence 6. Highlight or underline

Some response type options Objectively scored: SUPPLY Complete/solve equation Strengths/Weaknesses? Relatively efficient time use 7. Complete/change diagram/visual Easy to answer/mark 8. Forces construction Measure and record 9. Suitable for certain knowledge domains Proof or edit text 10. Complete an expression Hard to write well 11. Modified binary e.g., T/F with correction 12. Cloze or insert text 13. Short Answer 14.

Supply response type options Subjectively Scored 15. Write mathematical expression Strengths/Weaknesses? Easy to write Takes time to answer/mark Forces construction Suitable for certain knowledge domains Requires rubrics, marking guides Requires moderation of marking to control for subjectivity 16. Write short answer (number, text) 17. Write multiple short answers (numbers, text) 18. Multiple binary category

Supply response type options Subjectively Scored 19. Essay Strengths/Weaknesses? Easy to write Takes time to answer/mark Forces construction Suitable for certain knowledge domains Requires rubrics, marking guides Requires moderation of marking to control for subjectivity 20. Write math equation 21. Draw (shape, pattern, graph ) 22. Explain reasoning or rule 23. Formulate hypothesis 24. Generate solutions

Implications Validity is a JUDGEMENT of Degree of Suitability Format alignment with curriculum goals matters There are no perfect universal methods Format decisions are constrained by feasibility concerns Mixing methods and multiple opportunities good practice These choices represent a chain of evidence Weakest link limits validity of assessment

Instructor Role in Assessment Collection of Evidence & Exercise of Interpretive Judgment Must be considered by society as competent to collect, evaluate, & respond to evidence Competence comes from expertise in the performance of whatever is being taught Content Knowledge in collecting, interpreting accurately, and responding appropriately to valid evidence for the purpose of improvement Assessment Literacy in acting appropriately on evidence Teaching (pedagogical content knowledge)

So what? Designing assessment for your course Focus on what skills and knowledge/understanding you want them to have by the end Scaffold assessment to ensure development of those skills/knowledge Early, low-weight tasks that generate feedback and early warning Assessments that gradually get more demanding and carry more weight Assessments that support development of desired skills/knowledge Assessments that force application of desired skills/knowledge Let me give you some examples

Educ 224/225-Educational Assessment #1; 15%: week 3 Administer a standardized Test of spelling to a person over the age of 16 years. Describe the assessment and include a copy of the instrument. Describe the content of the test, the level for which it is designed, and how it was administered. Report the scores you obtained from the person s results and indicate what those mean relative to any norms provided by the test manual. A template will be made available for the report. #2, 25%: week 6 You will be given test reports concerning the performance of students who have been administered an asTTle test. Using the scoring information and rubric supplied, interpret the assessment, and write a brief report giving feedback on the student s strengths and weaknesses based on their test performance. A template will be made available for the report. #3, Item construction & review 10%: Week 10 Construct five multiple-choice questions according to best practices taught in this course for material in ONE of the chapters 8-13 of the required text book. Upload these to PEERWISE for review by class mates. Read, answer, and comment on and evaluate 10 questions uploaded by at least two other classmates. Marks are for participation only, with 5% given for uploading 5 questions and 5% for reviewing 10 peer questions. #4, Investigation 50%: Week 14 (in exam break) Create a test of ONE chapter of the course required text using only chapters 8-15. Administer it, score the performance of two classmates, and provide feedback to them. Evaluate your test against the principles of assessment taught in this course. Detailed steps are given in the course handbook and in tutorials.

Educ 380-Research Methods Assignment 1 (2 reviews each worth 5%) Write a brief review of the methods used in a published research paper, identifying strengths and weaknesses of the design, data collection, and analysis of the paper. Each review will be worth 5%. Qualitative Article Report Week 6 Quantitative Article Report Week 8 Assignment 2 (worth 40%) Develop a research proposal to address the question you had approved. The design must use two methods (i.e., one quantitative and one qualitative). As you attempt to design an approach, you may have to re-frame your approved question to best suit the design please check with your lecturer before proceeding. Write a 10-12 page proposal that outlines the problem, briefly reviews the literature, describes and justifies the 2 methods, and provides a management plan based on 12 months with $1000. Research Proposal Week 12 Examination (worth 50%) Analyse the research questions, design, data collection, data analyses, and interpretations of a published research paper. Design a follow-up study that makes use of a contrasting research paradigm to answer or extend the research questions posed in the paper. You will be given access to the research paper 2 weeks prior to the end of the class for study. The exam will have a mixture of short answer and essay questions.