Training STANAG Raters: Background, Overview, and Norming
Explore the essential aspects of STANAG rater training, including scale introduction, testing protocols, scoring conventions, initial norming, and guided practice with language samples. Get insights into the process through guided discussions and mock interviews.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
A Case for STANAG 6001 Rater Training & Norming David Oglesby Defense Language Institute English Language Center
The Way I See It Two individuals can witness the same external event, but interpret and perceive it totally differently, depending on how they think about or cognitively process it. Stephen Diamond http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg Magura
Testing the Productive Skills http://t2.gstatic.com/images?q=tbn:q2mSPux49ZohtM:http://www.imageenvision.com/150/37204-clip-art-graphic-of-an-olive-green-guy-character-speaking-with-a-microphone-by-jester-arts.jpg http://t2.gstatic.com/images?q=tbn:kE02k4CkLn4JBM:http://www.acclaimimages.com/_gallery/_images_n300/0515-0909-2722-3526_business_icon_button_report_writing.jpg We want to set tasks that form a representative sample of the population of oral tasks that we expect candidates to be able to perform The tasks should elicit behavior which truly represents the candidate s ability The samples of behavior can and will be scored validly and reliably http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg From Hughes Testing for Language Teachers
Reliability in Testing A reliable test should be a consistent measure of performance. Immune to Boredom Grin & Bear It Bored Can t Read My Stoic Face OMG! I m Bored Miserable Fog of Ennui Bored to Tears http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg Levels of Boredom
Fair & Balanced Testing? I object to your subjectivity I won t be subjected to your objectivity Rating Productive Skills http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg
Objective vs Subjective Scoring http://t0.gstatic.com/images?q=tbn:FGcUyKuEfaPXiM:http://4.bp.blogspot.com/_FQAe9NHopIo/TCyj0L4TMBI/AAAAAAAAAcA/LFs6pkvuQl0/s1600/photo_verybig_362946.jpg Given this undirected graph, what would be the result of a depth first iterative traversal starting at node E? Write a fictional account of this young woman s life up to the moment reflected in the picture. a. EABCFDG b. EDBFCG c. EDBGFCA d. EADBCFG e. EGDCFBA http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg
Training STANAG Raters Background & Overview: Introduce scale, testing protocol & scoring conventions Initial Norming: Review & discuss benchmark language samples Guided practice: Rate samples using rating scale & rubric Mock Interviews: For speaking assessment, complete practice interviews Certification: Select raters who provide reliable and efficient ratings http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg
Background & Overview Discuss the STANAG testing mission and purpose Introduce the modality-specific specifications and testing protocol Demonstrate/preview a model interview Issue training materials (handbook) and discuss contents http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg
Business of Testing Protocols 1. General: Unwritten rules or guidelines that are peculiar to every culture or organization, and are supposed to be observed by all parties in the conduct of business, entertaining, negotiating, politics, etc. 2. Product development: Statement of attributes (features and benefits) that a new product must be designed to have. A product protocol is prepared by consulting all parties to the project. 3. Technology: Set of agreed upon, and openly published and distributed, standards that manufacture compatible devices to the same specifications. All devices made under the same protocol work with one another without any adjustment or modification. enables different firms to http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg
Initial Norming Present calibrated (benchmark) performances at each base/plus level As a group, analyze the performance considering content, task & accuracy Participants identify parts of performance that constitute a ratable sample Discuss the rating with respect to rubric http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg http://t2.gstatic.com/images?q=tbn:ccOiDo3AR0QYWM:http://us.123rf.com/400wm/400/400/marykal/marykal0908/marykal090800003/5420197.jpg
Benchmarks Reduce the Likelihood of Rater Drift http://t0.gstatic.com/images?q=tbn:MLzUqGCI3ClZtM:http://www.bradfitzpatrick.com/stock_illustration/images/cartoon_charter_boat_001.gif http://t1.gstatic.com/images?q=tbn:dib1kqbh7avJ_M:http://www.clker.com/cliparts/1/2/b/5/11949896831046083558boat_jarno_vasamaa_.svg.hi.png http://t1.gstatic.com/images?q=tbn:CXJpKyBNapgrjM:http://www.clipartguide.com/_named_clipart_images/0060-0808-1915-1227_Overheated_Man_in_a_Rowboat_Pouring_Sweat_clipart_image.jpg http://t3.gstatic.com/images?q=tbn:FNwnXa0B2IdHBM:http://etc.usf.edu/clipart/27300/27320/anchor_27320_lg.gif Level 1 http://t3.gstatic.com/images?q=tbn:FNwnXa0B2IdHBM:http://etc.usf.edu/clipart/27300/27320/anchor_27320_lg.gif Level 2 http://t3.gstatic.com/images?q=tbn:FNwnXa0B2IdHBM:http://etc.usf.edu/clipart/27300/27320/anchor_27320_lg.gif http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg Level 3
Guided Practice Present speaking/writing performances for individual practice rating Participants report scores and criteria applied Group discussion leads to consensus (or not) Compare group rating with expert rating http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg
Evaluating at the Microlevel Purpose of evaluating = Select level = Elicit sample Collecting information = Apply criteria Interpreting information Decision making = Determine rating http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg http://t1.gstatic.com/images?q=tbn:Rrp1MAaFgk-2QM:http://www.appliedlanguage.com/flags_of_the_world/large_flag_of_bulgaria.gif
Mock Interviews for Speaking Assessment Pairs of trainees conduct mock interviews Other trainees observe & take notes After the interview, all trainees assign rating Group discusses conduct of interview, effectiveness of elicitation, ratings Trainees paired anew and repeat process Certify reliable & efficient raters http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg
The Norming Conquest Construct-irrelevant variance can cause rater drift over time Moral dimension (severe lenient) Halo effect Central tendency Analyze inter-rater/intra-rater reliability Renorm raters using benchmarks http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg
People are Dying for Change A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it. Max Plank Evolution Road http://t0.gstatic.com/images?q=tbn:_zFQBef3XZsUWM:http://pictures.toprentacar.bg/bulgaria-logo.jpg
