Understanding Consequential Validity in Testing

Examining Consequential Validity Latvian experience Inga F rte-Z l te Head, STANAG 6001 Testing Team NAF Language School, Latvia Brno, Sept 2016

Background Consequential validity and what it implies Guidelines for determining whether a given test should be used for a particular purpose in a particular way Another way of looking at validity Consequences of current test use in Latvia Summary

Background STANAG tests in Latvia since 2003 Job/ rank requirements 2-2-2-1 since 2008 Job/rank requirements 2-2-2-2 since 2013 Plus levels since 2014

STANAG test in Latvian NAF Qualification/ promotion a) for captains, senior & higher officers (SLP at least 2-2-2-2; valid 5 years); b) for positions at NATO HQs (2 years); c) for studies at BALTDEFCOL (2 years) At the end of an intensive course check the effectiveness of the training programme

Training and STANAG test Intensive English Language Course 16 weeks; 3 locations; divided in 7 different levels depending on entrance ALCPT score + short writing test for placement One cannot retake the STANAG test while having a valid certificate, unless s/he has taken a language course

Consequential Validity The idea of validity is connected to the idea of inference-making . Those inferences depend on how a test is used. Tests are not used in a vacuum; they are always intended to serve the needs of an educational system or of society at large. (L.Bachman)

Consequential Validity We as testers must recognize that considerations about the rights and interests of test takers, institutions responsible for testing and making decisions based on those tests, and the public interests are essentially political; that they change over time; that they have implications a) for the practice of our professions; b) for the kinds of tests we develop and c) the ways in which we justify their use.

Consequential Validity We as testers must go beyond the scientific demonstration of empirical and logical evidence to the consideration of the potential consequences of testing; From applied linguistics and testing public policy.

Guidelines for determining whether a given test should be used for a particular purpose in a particular way are not well defined.

Areas to be considered in the ethical use and interpretation of test results the construct validity/ evidence that supports the particular interpretation we wish to make; the value systems of test takers, developers and users and also those of the educational program and society at large that inform the particular test use; the practical usefulness of the test; the consequences of the education system or society of using test results for a particular purpose

Consequences: continued a) List the intended uses of the test; b) List the potential consequences, both positive and negative, of using a particular test; c) Rank them in terms of (un)desirability of their occurring + assign values; d) Gather information that will help you determine the likelihood of each consequence (prior experience, empirical studies of relationships between similar tests and their consequences, historical documents)

OR Consider alternatives to testing as a means of achieving the same purposes

What undermines valid use of test results different stakeholders have competing agendas in education; highly technical aspects of testing often are poorly understood within the general public; test security can promote a sense of secrecy; policy-makers and practitioners often can use testing results in ways that were not intended by test developers. All of it feeds back into public mistrust.

Validity = Overall Quality allows one to express one s opinion that some tests are better than others in a given situation, without having the connotation of precision and objectivity that the jargon of degrees of validity suggests.

Validity = a subjective summary of evidence There is no coefficient like a reliability coefficient, saying this has a validity of 0.80 and that has a validity of 0.40. There is good evidence of the validity of x. There is good evidence of the lack of validity of y.

Consequences for NAFL and testing Senior officers who have problems achieving SLP 2-2-2-2 choose to leave NAFL Suggestions to change language testing policy lower requirements, set up national standards/ tests Organize STANAG test trips to Lithuania/ SHAPE/ elsewhere Allow retaking the STANAG test until the required SLP is achieved Establish appeals procedure involving independent experts (from other NATO countries)

Consequences for teaching Job (self)evaluation for teachers Assigning students to groups based on their desired end-of course STANAG test results, not entrance test results. Evaluating a course based on their desired end-of course STANAG test results.

Summary The questions involved in the appropriate use of the tests are not easy, but everyone has faced them and they will persist. Appropriate answers will vary across cultures, specific contexts, and time. Tests should be labeled just like dangerous drugs: Use with care! (Spolsky)

books and articles Bachman, L. F. Fundamental considerations in language testing. OUP, 1990. pp 279-285 Hitchcock, J.H., A.J. Onwuegbuzie and H.B. Khoshaim. Examining the consequential validity of standardized examinations via public perceptions: a review of mixed methods survey design considerations. International Journal of Multiple Research Approches, Taylor & Francis, 2015. Norman, G. The negative consequences of consequential validity. Springer Science+Business Media Dordrecht, 2015. Welner, K. G. Consequential validity and the transformation of tests from measurement tools to policy tools. Teacher College Record Volume 115, Sept 2013.