Exploring Language Barriers in Online Learning for Non-Native Speakers

Assessing language barriers in online learning: The suitability of English language Open Educational Resources for non-native speakers Irina Rets (irina.rets@open.ac.uk / @IrinaRets) Supervised by Dr. Ursula Stickler Dr. Tim Coughlan Dr. Lluisa Astruc

POTENTIAL QUESTIONS TO DISCUSS - feedback on the research design and analysis of my 1st study - sampling in qualitative research to what extent the sample needs to be homogeneous or is it better to look broader (in line with Robinson, 2014)? - is it conventional to show numbers to support the results of qual research? E.g. 2nd RQ of my 2nd study: <Is there a difference in approaches to text simplification between English teachers with different L1?> Name of code Building a mental model of the text chunking ideas into smaller pieces frequency Giving more context to understand a concept Impersonal to personal language Spanish 4% 30% 37% 12% 17% Chinese 7% 38% 30% 12% 13% Russian 7% 33% 34% 12% 14% 2

MOTIVATION OF THE STUDY Obstacles facing OER for their wider use in developing and/or non- English speaking regions (Cobo, 2013; Hatakka, 2009; Kanwar, et al., 2010): - predominant use of English as the language of instruction Linguistic accessibility is among the issues that determines the quality of learning resources (Quality Guidelines for OER in Higher education, UNESCO, 2015). Very few studies have explored accessibility of OER to EFL learners. Available studies of linguistic accessibility are in the area of healthcare (e.g. Kher et al., 2017; Xie et al., 2018). 3

STRUCTURE OF PHD Pilot Identification of a problem 200 OER courses Readability analysis 9 participants eye-tracking experiments Descriptive statistics Content analysis stimulated recall interviews One-Way ANOVA Cluster analysis Lens: teachers 24 participants, 3 language groups Online simplification task, stimulated recall interviews Lens: learners 30 participants Eye-tracking experiments Learning gain test Content and thematic analysis Statistical analysis 4

1ST STUDY METHODS

Table 1. Readability metrics automatically calculated by Textinspector 1. Average sentence length Having fewer words per sentence makes the text easier. 2. Type/token ratio (TTR) The lower the measures are, the easier the text is. 3. VOCD e.g. manager, boss, chief, head, leader 4. MTLD 5. Words with more than 2 syllables Percentage The fewer syllables the words have on average in the text, the easier the text is. 6. Average syllables per sentence e.g. extra vs additional 7. Average syllables per word 8. Syllables per 100 words 9. Flesch Reading Ease Higher scores of Flesh Reading Ease indicate material that is easier to read. 10. Flesch-Kincaid Grade The results of the other two formulas correspond to years of formal education a person needs to understand the text on the first reading. 11. Gunning Fog index 12. Noun elements per sentence Less nominalisation contributes to the text being easier. e.g. We had a discussion vs we discussed 13. A1;14. A2; 15. B1; 16. B2; The more A1 lexis and the less C1 and C2 lexis the text has, the easier the text is. 17. C1; 18. C2 19. 0-6K (combined) The higher the percentage of the numbers before 6, the more frequently used vocabulary the text includes and, thus, the easier the text is. 20. 6K-100K (combined) e.g. purchase vs buy 22. Scorecard The Scorecard of above B2 level indicates a difficult text accessible to language learners of the highest level of proficiency. 6

METHODS RQs Methods of data analysis 1) To what extent are OER courses offered on major platforms accessible to EFL learners? 2) Is there a difference in the readability level of OER according to the stated educational level of the courses? 3) Is there a difference in the readability level of OER according to the subject matter? Using scorecards, counted the number of courses that require more than intermediate level of language proficiency. Independent sample T-tests, Mann-Whitney U tests and ANOVA tests; identified the variables that show significant differences consistently between all educational levels. One-Way ANOVA; Hierarchical and K-means cluster analysis. 7

Materials 150 most popular courses from OpenLearn (2018) at 3 educational levels 50 courses from Saylor Academy (Saylor) (2018) at 5 educational levels 8

RESULTS

RESULTS RQ1 1) To what extent are OER courses offered on major platforms accessible to EFL learners? Proficiency Level Proportion of OpenLearn courses Proportion of Saylor courses B1+ 2% 0% B2 / B2+ 7% 14% C1/C1+ 35% 24% C2/C2+ 56% 62% Answer: - more than 86% of courses on both OER platforms are only considered suitable for the learners at the highest or advanced level of English proficiency. Thus, OER might not be accessible to EFL learners who do not read in English fluently. 10

RQ2 2) Is there a difference in the readability level of OER according to the stated educational level of the courses? OpenLearn Statistical analysis showed there is significant statistical difference in some metrics between all three levels. OpenLearn: difficulty between the levels 25 20 15 The biggest difference was observed between Level 1 vs. Level 3 (14 variables that showed statistically significant difference; large effect sizes average effect size is 1.07). 10 5 0 average sentence length word length lower frequency words advanced lexis Level 1 Level 2 Level 3 The smallest difference was observed between Level 2 vs. Level 3 (8 variables that showed statistically significant difference; no effect sizes larger than 0.75) Level courses are the most difficult to read among the three levels and Level 1 (introductory) courses are the easiest. 3 (advanced) 11

RQ2 2) Is there a difference in the readability level of OER according to the stated educational level of the courses? Saylor ANOVA analysis showed much less pronounced difference between the five levels. Saylor: difficulty between the levels 30 25 20 15 The only statistically significant difference was observed between courses at remedial level 0 vs. courses at levels 1, 3 and 4 (with large effect sizes, all effect sizes were bigger than 1.14). 10 5 0 average sentence length word length basic lexis lower frequency words Level 0 Level 1 Level 2 Level 3 Level 4 Level 0 courses are the easiest to read among the five levels 12

Answer to RQ2: - Analysis of OpenLearn materials showed that there are statistically significant differences between OER at all three given educational levels (in measures of word and sentence length; amount of advanced lexis); - Analysis of Saylor courses showed that such differences occur only between remedial level 0 courses and more senior courses. Evidence from both platforms: Courses at the lowest educational level ( remedial level 0 and introductory level 1 courses) are the easiest to read. 13

RQ3 3) Is there a difference in the readability level of OER according to the subject matter? ANOVA analysis showed no statistically significant difference between any of the subject labels on both platforms. We conducted cluster analysis as subject classifications are rather broad on both OER platforms. Figure 1. Excerpt from the dendrogram of OpenLearn courses, arising from the hierarchical clustering algorithm Figure 2. Dendogram of Saylor courses, arising from the hierarchical clustering algorithm 14

RQ3 3) Is there a difference in the readability level of OER according to the subject matter? To decide on the number of clusters and check which cluster is the most difficult in readability we followed the following steps: - identified cluster intervals in OpenLearn and Saylor data; - conducted K-means analysis for each cluster solution; - identified the intra-cluster similarity and inter-cluster dissimilarities amongst clusters for each solution and arrived at 5-cluster solution for both platforms. Examples of subjects assigned to cluster 5 in OpenLearn Examples of subjects assigned to cluster 5 in Saylor Digital forensics the autistic spectrum: from theory to practise Attachment in the early years understanding dyslexia organisations and management accounting Factors that influence health: An introduction Exploring the English language What is politics? The Enlightenment understanding operations and management management: perspective and practice Criminology beyond crime BIO307 Microbiology BUS101 Introduction to Business COMM411 Public Relations HIST362 Modern Revolutions HIST363 Global Perspectives on Industrialization POLSC101 Introduction to Political Science POLSC401 Ethics and Public Policy POLSC402 Global Justice CS301 Computer Architecture ECON307 International Trade PSYCH101 Introduction to Psychology BUS305 Small Business Management 15

Difficulty of clusters OpenLearn clusters 70 5th cluster - most difficult texts in all three measures, 3rd and 4th clusters - the easiest texts. 60 50 40 30 20 10 0 Flesch Reading Ease A1 C1 cluster 1 cluster 2 cluster 3 cluster 4 cluster 5 Saylor clusters 70 5th cluster - most difficult in two measures (Flesh Reading Ease and the amount of A1 lexis), 1st cluster is the most difficult in the third measure, amount of C1 lexis. 60 50 40 30 20 10 0 Flesch Reading Ease A1 C1 cluster 1 cluster 2 cluster 3 cluster 4 cluster 5 16

Answer to RQ3: - no statistically significant difference between any of the subject categories on either of the platform. - very diverse courses subject-wise are positioned close together and assigned to one cluster. Thus, both methods of analysis (ANOVA and cluster analysis) applied to the two OER platforms suggested no effect of subject matter on readability of the courses. 17

IMPLICATIONS

- majority of English OER texts at different educational levels and subject categories are only suitable for native speakers or EFL learners with advanced language proficiency. Recommendations for OER material writers: - check the text difficulty level of their materials with advanced online readability tools (https://textinspector.com/); - avoid using long words, split long sentences; - avoid using advanced lexis unless necessary (http://www.englishprofile.org/wordlists) 19

Practical example Text from a course Applying social work skills in practice (Level 2): 1.1 Complex issues: social work, risk and the media original text and a text simplified by an English teacher (participant in Study 2) It can be argued that, on many occasions, social workers engage with some of society s most complex issues, often being called on to intervene alongside other professionals or when a series of interventions have not been effective enough to keep a service user supported or safe. For example, many young people coming to the attention of social workers may already be known to teachers, school counsellors, health professionals or others, before their situation arrives at the point of needing social work action. At this point, social workers do not make decisions about how to respond to perceived risks alone. Their judgements are made in conjunction with service users, their kin and networks, and in discussion with other significant professional people for example, doctors, teachers, other health workers, lawyers and the police. 20

Cobo (2013) concluded that as long as educational materials continue to be based on a few (Western European) languages, the use and repurposing of OER remains limited to people who have had the privilege to learn one of those elite (foreign) languages . 21

References Beare, K. (2018). How Many People Learn English? [Web log post]. Retrieved 25 January 2018 from https://www.thoughtco.com/how-many-people-learn-english-globally-1210367 Banzato, M. (2012). Barriers to teacher educators seeking, creating and sharing open educational resources: An empirical study of the use of OER in education in Italy. 2012 15th International Conference on Interactive Collaborative Learning (ICL), (pp. 1 6). Berendes, K., Vajjala, S., Meurers, D., Bryant, D., Wagner, W., Chinkina, M., . . . Graham, S. (2018). Reading Demands in Secondary School: Does the Linguistic Complexity of Textbooks Increase With Grade Level and the Academic Orientation of the School Track? Journal of Educational Psychology,110(4), 518-543. Berndt, A., & Wayland, J. (2014). Evaluating the readability of marketing research textbooks: An international comparison. Journal of International Education in Business, 7(1), 47-59. Betschart, P., Zumstein, V., Bentivoglio, M., Engeler, D., Schmid, H., & Abt, D. (2017). Readability assessment of online patient education materials provided by the European Association of Urology. International Urology and Nephrology, 49(12), 2111-2117. Cobo, C. (2013). Exploration of Open Educational Resources in Non-English Speaking Communities. International Review of Research in Open and Distance Learning, 14(2), 106-128. Crossley, S.., Greenfield, J., & McNamara, D. S. (2008). Accessing Text Readability Using Cognitively Based Indices. TESOL Quarterly, 42(3), 475-493. Harrison, C. J. O. (1980). Readability in the classroom. Cambridge University Press Hatakka, M. (2009). Build it and they will come? Inhibiting factors for reuse of open content in developing countries. The Electronic Journal of Information Systems in Developing from http://www.ejisdc.org/ojs2.../index.php/ejisdc/article/view/545. Countries, 37(0). Retrieved 10 June 2018 Huang, W.-H. D., Lin, M.-F. G., & Shen, W. (2012). Understanding Chinese-speaking open courseware users: a case study on user engagement in an open courseware portal in Taiwan. Open Learning: The Journal of Open, Distance and e-Learning, 27(2), 169 182. Jatowt, A., & Tanaka, K. (2012). Is wikipedia too difficult?: Comparative analysis of readability of wikipedia, simple wikipedia and britannica. Proceedings of the 21st ACM International Conference on Information and Knowledge Management, (pp. 2607-2610). 22

Kanwar, A., Kodhandaraman, B., & Umar, A. (2010). Toward sustainable open education resources: A perspective from the global south. The American Journal of Distance Education, 24(2), 65-80. Kher, A., Johnson, S., & Griffith, R. (2017). Readability assessment of online patient education material on congestive heart failure. Advances in preventive medicine, 2017. Knyazeva, S. (2010). Open educational resources in CIS: Preliminary study of the state-of-the-art and prospects for development. BSU. Retrieved 13 October 2018 from http://elib.bsu.by/handle/123456789/13100 OpenLearn. (2018). Retrieved July 10, 2018, from https://www.open.edu/openlearn/ Robinson, O. C. (2014). Sampling in interview-based qualitative research: A theoretical and practical guide. Qualitative research in psychology, 11(1), 25-41. Saylor Academy. (2018). Retrieved August 7, 2018, from https://www.saylor.org/ Textinspector (2018). Retrieved 1 December 2018, from https://textinspector.com/ UNESCO. from http://www.unesco.org/new/fileadmin/MULTIMEDIA/HQ/CI/CI/pdf/Events/Paris%20OER%20Declaration_01 (2012). Paris Declaration on Open Educational Resources. Retrieved 1 June 2018 UNESCO. (2015). Guidelines for open educational resources (OER) in higher education. Commonwealth of Learning. Paris. Retrieved 8 September 2018 from https://unesdoc.unesco.org/ark:/48223/pf0000213605 Wiley, D., & Gurrell, S. (2009). A decade of development . Open Learning: The Journal of Open and Distance Learning, 24 (1), 11 21. William and Flora Hewlett Foundation (2018), Open Educational Resources. Retrieved 3 April 2018 from https://www.hewlett.org/strategy/open-educational-resources/ Xia, M., Kochmar, E., & Briscoe, T. (2016). Text readability assessment for second language learners. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications (pp. 12-22). Xie, D. X., Wang, R. Y., & Chinnadurai, S. (2018). Readability of online patient education materials for velopharyngeal insufficiency. International journal of pediatric otorhinolaryngology, 104, 113-119. 23

THANK YOU irina.rets@open.ac.uk / @IrinaRets