Managing Examiners in Onscreen Marking Environment

Managing examiners in the Onscreen marking environment A paper presented at the 12th SAAEA Annual Conference. Pretoria: 13-16 May 2018 by Ebba Masiri (Mrs) Zimbabwe School Examinations Council emasiri@zimsec.co.zw

Introduction The study explored the utility of the onscreen marking (OSM) generated reports that provide information that could enhance examiner management in the Zimbabwean context. Scripts where candidates have constructed responses to examination questions are marked by people Called by different names: examiners; scorers; raters; markers; assessors (Ofqual 2014a; 2014b: 2014c; 2014d). Examination scripts marking is gradually migrating from paper based marking (PBM) to computer screens (onscreen marking OSM) OSM- candidates scripts are scanned and their images are distributed to examiners who mark them on computer screens as illustrated on the next slide

Introduction cont.. Exam authorities around the world follow established procedures to recruit, train and manage examiners who mark specific subject components (Hill 2013; Odendal 2011; Tze Ho & Chong 2013; Ofqual 2014c; 2014d). The OSM technology generates reports that provide information about examiners (Ramakrishna et al 2012; Roan 2009) The Zimbabwe School Examinations Council adopted the OSM technology in June 2012, with Principal marking Supervisors (PMS who supervise examiners during marking) complaining that the technology does not allow them to do what they used to do on paper. This study explored how the reports can be used to manage examiners in the OSM environment in Zimbabwe.

Statement of the problem Examination authorities around the world recruit and train examiners who mark scripts for specific subject components with constructed responses to examination questions, resulting in established procedures for managing examiners. The examination script marking is, however, gradually migrating from PBM to OSM, a technology that automatically generates data about examiner performance. The ZIMSEC introduced some Ordinary Level subject components onto the OSM platform in June 2012. However, the PMSs for some components that are currently marked on screen alleged that they could no longer do what they used to do in PBM. It therefore became necessary to explore the utility of OSM generated reports in providing information that could enhance examiner management in the Zimbabwean context.

Objectives of the study To identify the OSM generated reports that relate to the examiners. To establish how each report could be utilised to provide useful information for managing examiners in the OSM environment.

Significance of the Study help ZIMSEC to extract and utilise the electronically generated data to guide examiner management so as to improve the quality of marking contribute literature to examiner management in the OSM environment could guide further research on the possible uses of data that are automatically generated by the OSM technology

Conceptual Framework guided by Key Literature Examination authorities maintain teams of trained examiners, from where they appoint senior markers who supervise examiner teams. The utility of the OSM generated report was studied using the framework of established practices for the recruitment and training of examiners to build teams; appointment of senior examiners; monitoring of examiner marking and; retraining of examiners (Hill, 2013; Odendahl, 2011; TzeHo& Chong Sze, 2013; Ofqual, 2014c; 2014d).

Subject Manager: Monitors examiner numbers, arranges examiner training, builds examiner teams for particular marking sessions guided by senior marker reports, appoints senior markers, maintain teams Pool of trained examiners Retraining: During marking/after marking Senior examiners: Set marking standards, mark scripts, supervise examiners, write reports about examine performance; report to Subject Manager Script Marking On paper/onscreen Examiners: Mark scripts

Key questions raised about examiners in OSM When the University of Cambridge Local Examinations Syndicate (UCLES) considered adopting the OSM technology, Raikes et al (2004, p. 11) raised research issues related to examiners, which include: How do we retain existing examiners and recruit new ones? How do we support examiners so that they lead in shaping the technology? How should examiner payments be changed to be fair in the new environment? What training do examiners need? How do we identify and respond to the needs of the examiners and senior examiners? Some researches have been conducted on attitudes of examiners towards the technology and comparison of marking behaviours on paper and on screen (Coniam 2009; 2011;2013; Johnson et al 2012) This research sought to answer some of these questions in the

Methodology The study adopted the qualitative case study methodology premised on the constructivist worldview which assumes that reality is subjective and constructed through communication, interaction and practice, where researchers seek to gain insights from multiple participants and from themselves The OSM of O Level Biology was the case and the components, Papers 2 and 4, were the embedded units. reports generated by the OSM technology for the November 2015 examination series were used, binding the case by activity and time

Sampling Biology components Papers 2 and 4 were selected as typical cases where OSM generated examiner reports could be studied, mainly because the researcher is familiar with the Biology examinations and would make more meaningful interpretations of data collected for Biology than any other subject. Criterion sampling was further used to select the participants (Palys, 2008, p. 697) who were likely to have experience with access and use of the reports. Four subject managers and six senior markers (three from each component) agreed to participate in the research Access and consent were sought from ZIMSEC management and participants respectively

Data collection and analysis Semi-structured interviews conducted at various times convenient to the participants to avoid disrupting work document analysis examiner- related reports were studied in relation to interview responses Data were analysed thematically, an inductive approach that mines data to derive the structure of analysis by identifying emerging themes and categories The interview and document analysis data were read and annotated to identify themes. A coding scheme was developed and codes applied to the dataset. The emerging themes were refined by collecting additional data from the participants. Themes that emerged from the interviews were corroborated with document analysis Member-checking was used to increase the credibility of data interpretation. Three participants reaffirmed the categories that emerged from the data, one subject manager and one senior examiner from each of the Biology components.

Results Seven reports were identified for each of the components, Biology paper 2 and 4, displaying similar information Senior markers said they were seeing them for the first time Subject Managers said there was no formal training on access and use of the reports They learnt by trial and error and passed on the skills to colleagues reports generated can be downloaded from the administrator module accessible to Subject Managers only Reports are generated real-time but completed at the end of marking Senior markers assess examiners under their charge and assign grades Some participants suggested that Subject Managers could assume that role in the OSM environment Others however said Senior markers should continue assessing examiners since grades cannot be assigned on the basis of mere figures A good marker can be truant, foul-mouthed, arrogant, disruptive and disobedient behaviours not recorded on the reports. Some Subject Managers use the reports for payment purposes, identify absentees for that session, monitoring quality of seeds and daily examiner activities Following section details the seven report.

Quality of marking by marker Information displayed List all examiners and the questions they marked, the seeds they marked and the seeds failed Possible uses Assess examiner competences Evaluate the quality of seeds set Evaluate the quality of the marking guide Assess Senior Markers competences in setting seeds

Quality of marking by related part Information displayed Number of scripts each examiner marked per question, number of seeds marked and failed number of times stopped, number of scripts marked for training Possible uses Evaluate the quality of seeds set Evaluate the quality of the marking guide Assess Senior Markers competences in setting seeds Could be a basis for identifying examiner type, i.e. clerical or expert Could guide examiner payment

Quality of marking by related part Information displayed List all examiners questions allocated to each examiner Total scripts marked by each examiner Seeds marked and seeds failed Suspect/deleted seeds failed Average % seeds failed Possible uses Assess examiner competences Evaluate the quality of seeds set Evaluate the quality of the marking guide Could guide appointment of senior markers Can be used to guide examiner payment

Marker activity Information displayed Times when each examiner logged in and out of the system, the activity they were undertaking, time when a seed was marked, whether the examiner failed or passed the seed. Indicates absent examiners Possible uses To monitor examiner activities Assess how examiners used time during the marking session Select examiners for promotion Identify examiners who need close supervision

Permanently stopped markers Information displayed List markers who were permanently stopped from marking particular questions Only one marker was permanently stopped for Biology paper 2 (stopped by qualification seed-so never marked live scripts) Probably stopped by error or never made a follow up with the senior marker Possible uses Assess examiner competences Evaluate the effectiveness of the coordination exercise Assess the competence of Senior Markers in using the stopped marker management tool Evaluate the marking guide

Markers not in quota groups Information displayed Lists all examiners and their marking status; senior marker or normal marker Possible uses A record of the examiners who were invited to mark in a particular session Eventually becomes a record of who actually attended the marking session Used to allocate script portion to examiners

Conclusion The results of this study indicate that reports that are automatically generated by the OSM technology can be used to efficiently manage examiners in the OSM environment. The reports can be used to monitor examiners during marking, pay them, identify competent examiners who can be promoted to senior positions and incompetent examiners who need to be dropped from marking teams. Examiners who need retraining can also be identified. Some of the questions raised by Raikes et al (2002, p. 11) related examiner recruitment and retention, examiner payment and examiner support in the OSM environment can be answered by the results of this research. There is, however, need for ZIMSEC to create a formal training platform for senior examiners and subject managers for the effective use of the reports in managing examiners, as well as a framework for access and use of the reports that can be used for examiner assessment by senior markers.