Uneven Language Proficiency: Exploring Spiky Profiles in EAP Candidates

Uneven language proficiency: How spiky can a spiky profile be? Karen Smith Lecturer EAP/Study Skills University of Central Lancashire, Preston

Overview Research questions: What is the shape of uneven proficiency in EAP candidates across the 4 skills? Are there correlations between the skills? What are spiky profiles and how spiky can they get?

spikiness Should we expect big score differences across the skills? How much variation is possible or natural? Should we expect some skills to be always stronger than others? Does variation matter?

Are the 4 skills connected? In L1 listening and speaking develop together Then literate skills grow from speaking & listening Writing skills grow from speaking & listening (Berninger & Abbot,2010) The same brain mechanism is used for syntactic encoding in speaking and writing (Cleland & Pickering, 2006) Speaking does not always precede writing in L2 Individual differences have an impact: modality strength, preference, cognitive style, affective factors (Weissberg, 2006; Kormos & Trebits 2012; Baba,Takemoto & Yokochi 2013)

UCLans TELL Listening (30 mins) 3 different audio recordings, diff types of interaction: informal, semi-formal and formal 20 marks Reading (55 mins) range of texts, diff. skills tested 30 marks Writing (45 mins) essay writing task of 180-200 words (B2) 250-280 (C1) 20 marks (10 bands) Listening & reading marked optically Min 20% of all scripts 2nd marked Speaking 11 mins (B2) 13 mins (C1) 3 parts: intro, interactive discussion, responding to questions Grammar, vocab, pronunciation, discourse management, interactive ability 30 marks (5 bands)

Exam Data from the TELL B2 & C1 levels under investigation December 2016 sittings in 4 centres: Preston, Sunderland, Hull & Coventry 542 B2 candidates l/r/w/s scores compared 56 C1 candidates l/r/w/s scores compared Mixed nationalities, all adults, 21-25 years old

One-upmanship among the skills? B2 data n=542 C1 data n=56 Writing Reading Listening Speaking 96.9% (0.7% same) 100% 79.5% (2% same) 84% (7.1% same) 70.8% (2.4% same) 71.4% (1.8% same) Speaking better than 84.5% (7.2% same) 96.4% (1.8% same) 63.5% (3.3% same) 78.6% 26.8% (2.4% same) 26.8% (1.8% same) Listening better than 76.9% (2.4% same) 94.6% (1.8% same) 33.2% (3.3% same) 21.4% 18.5% (2% same) 8.9% (7.1% same) Reading better than 20.7% (2.4% same) 3.6% (1.8% same) 8.3% (7.2% same) 1.8% (1.8% same) 2.4% (0.7% same) 0% Writing better than

Spearmans rho correlation of each skill with overall score Skills Type B2 (n=542) C1 (n=56) rs Reading & Overall Receptive Literate .890 (79%) L .849 (72%) L Listening & Overall Receptive Innate .860 (74%)L .642 (41%) L Speaking & Overall Productive Innate .797 (64%) L .715 (51%) L Writing & Overall Productive Literate . 785 (62%) L .841 (71%) L

December 2016 Skills correlations at B2 & C1 Skills Type B2 (n=542) R Coefficient of determination in brackets B2 (n=542) rs C1 (n=56) rs Listening & Reading Receptive .698 (49%)L .706 (50%)L .316 (10%) M Reading & Writing Literate . 654 (43%) .662 (44%) . 794 (63%) L Speaking & Listening Innate . 574 (33%) .594 (35%) . 557 (31%) L Speaking & Reading . 556 (31%) .571 (33%) . 459 (21%) M Listening & Writing . 524 (27%) .546 (30%) . 282 (8%) S Speaking & Writing Productive .518 (27%) .523 (27%) . 374 (14%) M

Quartiles of the overall score at B2 Overall score % of sample mean score Speaking mean score Listening mean score Reading mean score Writing S-W% gap L-W% gap R-W% gap S-R% gap S-L% gap L-R% gap 0-25 0 26-50 20.3 59.32 45.41 37.55 35.77 23.55 9.64 1.78 21.77 13.91 7.86 51-75 55.5 75.12 68.55 61.24 47.76 27.36 20.79 13.48 13.88 6.57 7.31 76-100 24.2 92.92 85.76 84.44 65.99 26.93 19.77 18.45 8.48 7.16 1.32

A closer look at Writing vs. Speaking at B2 Speaking scores of sample n=542 Mean % gap with writing score SD Writing scores % of sample n=542 Mean % gap with speaking score SD 0-25 0.2 35.00 0 0-25 3.4 44.11 14.50 26-50 3.3 10.22 6.99 26-50 61 29.20 13.66 51-75 31 22.91 12.17 51-75 45.2 21.16 10.62 76-100 4.6 9.80 6.48 76-100 51.3 32.97 14.29

Writing vs. speaking scores B2 December 2016 120 100 80 Percentage weighted score 60 40 20 0 CEG08035196 CEG08032062 CEG08036211 CEG08035452 CEG08034409 CEG08032941 CEG08032834 CEG08035363 CEG08034774 CEG08034560 CEG08036069 CEG08034984 CEG08032066 CEG08033739 CEG08029912 CEG08035050 CEG08035542 CEG08023854 CEG08032405 CEG08034289 CEG08035159 CEG08036084 CEG08034522 CEG08035118 CEG08036015 CEG08034894 CEG08031988 CEG08034294 CEG08034772 CEG08032998 CEG08034873 CEG08035317 CEG08036036 CEG08033735 CEG08034976 CEG08035560 CEG08033736 CEG08034929 CEG08035129 CEG08036050 CEG08035345 CEG08029256 CEG08033815 CEG08034712 CEG08035655 CEG08032632 CEG08034769 CEG08035327 CEG08036182 CEG08034503 CEG08035607 CEG08033587 CEG08031596 CEG08034182 CEG08035701 CEG08034525 CEG08034529 CEG08034254 CEG08032850 CEG08033826 CEG08035669 CEG08032227 CEG08034950 CEG08036126 CEG08034271 CEG08035657 CEG08033642 CEG08031737 CEG08022749 CEG08036127 CEG08036153 CEG08035088 CEG08036296 CEG08030011 CEG08033156 CEG08035825 CEG08035852 CEG08035269 Candidates n=542 Series2 Series1

A closer look at Reading vs. Listening Reading scores of sample n=542 Mean % gap with listening score SD Listening scores of sample n=542 Mean % gap with reading score SD 0-25 3% 26.13 11.92 0-25 1% 9.50 9.00 26-50 27% 14.85 11.08 26-50 20% 10.97 8.69 51-75 41% 12.46 9.25 51-75 45% 12.89 9.47 76-100 29% 8.46 6.18 76-100 34% 12.58 10.52

The picture so far What are the parameters of uneven proficiency across the 4 skills? Speaking appears to be the strongest, even in the weakest candidates Writing is the weakest link for all (no surprise there!) Writing is very rarely better than speaking (2.4% of candidates) Reading also lets down the weakest Top quartile writing scores show smallest gap with speaking Are there correlations between the skills? receptive skills (R&L) are the most correlated at B2, not C1 literate skills (R&W) also strongly correlated, more at C1 Despite both being productive, Speaking & Writing are least correlated

Further study in spikiness Compare with other proficiency test data Closer inspection of the speaking and writing scores: what is going on within the scores? Compare performance under exam conditions against relaxed conditions (affective dimension) Explore modality preference & self-efficacy (affective/personality-related)

References Baba, K., Takemoto, Y. and M. Yokochi. 2013. Relationship between second language speaking and writing skills and modality preference of university EFL students . Japanese Institutional Repositories Online. (Retrieved 14 March 2015 from http://jairo.nii.ac.jp/0263/00000548.) Bereiter, C. & M. Scardamalia. 1987. The Psychology of Written Composition. Hillsdale, NJ: Lawrence Erlbaum Associates. Berninger, V. W., & Abbott, R. D. 2010. Listening comprehension, oral expression, reading comprehension, and written expression: Related yet unique language systems in grades 1, 3, 5, and 7 . Journal of Educational Psychology, 102/3: 635-651. Cleland, A. and M. Pickering. 2006. Do writing and speaking employ the same syntactic representations? . Journal of Memory and Language 54: 185 198. Council of Europe. 2011. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Language Policy Unit, Strasbourg. (Retrieved 14 March 2015 from http://www.coe.int/t/dg4/linguistic/Source/Framework_EN.pdf.) D rnyei, Z. 2005. The Psychology of the Language Learner: Individual Differences in Second Language Acquisition. Mahwah, NJ: Lawrence Erlbaum. Kormos, J. and A. Trebits. 2012. The role of task complexity, modality and aptitude in narrative task performance . Language Learning 62/2: 439 472. Weissberg, R. 2008. Critiquing the Vygotskian approach to L2 literacy in D. Belcher & A. Hirvela, A. (eds.). The Oral- literate Connection: Perspectives on L2 Speaking, Writing, and Other Media Interactions. Ann Arbor: University of Michigan Press. Williams, J. 2008. The speaking-writing connection in second language and academic literacy development in D. Belcher & A. Hirvela, A. (eds.). The Oral-literate Connection: Perspectives on L2 Speaking, Writing, and Other Media Interactions. Ann Arbor: University of Michigan Press.