Uncovering the Impact of English Language Skills on Academic Performance

Warum qualitative Methoden nicht zeigen, was hinter dem quantitativen Effekt steckt Judith Schoonenboom, Universit t Wien

Lee and Greene 2007: design This study investigated the relationships between graduate students placement test scores for English as a second language (ESL) and three measures of academic performance (grade point average [GPA], faculty evaluations, and student self-assessments). Qualitative and quantitative data collection techniques (archival, questionnaires, and interviews) were used to collect data from 100 students and 55 faculty members. (Lee & Greene, 2007, p. 366)

Lee and Greene: conclusions Although nonsignificant correlations were found between test scores and GPA, qualitative findings indicated that English skills are an important factor affecting students course performance. Additional mixed methods analyses found that variations in students views of academic success and their relevant background knowledge can help explain the overall insignificant relationship between ESL placement test scores and GPA. This mix of methods thus illuminated particular strands of the complex relationships between English proficiencies and graduate-level academic performance. (Lee & Greene, 2007, p. 366)

ID C GPA Quotation 0607 4 4.00 I do not have any language problems. 2020 3 4.00 I understand 80% of the lectures. Careful reading complements 20% of lack of understanding. 1315 3 4.00 My biggest problem is related with speaking in English. I will get a good grade because my mathematical background is strong. 0609 3 4.00 The professor is old so that his pronunciation is not clear. He usually handed out important contents and I could understand it by reading the textbook. 0620 2 4.00 I understand only 60-70% of the lectures. It has made my scores less than my expectation. 2036 3 3.57 The major problem is speaking. I spoke once or twice during the whole semester. 0610 3 3.53 Listening is a problem. Lack of cultural knowledge interferes with understanding the concept. 2037 2 3.22 Lack of knowledge about idiomatic expressions prevents me from understanding questions on the homework assignment. 0608 4 2.89 I understand almost 100% of the lectures. I understand the professor completely. [ ] Grades are not important. That is the least of my worries.

Argumentation in Lee and Greene 2007 English language skills affect course performance (qualitative) English language skills do not affect course performance (quantitative) because of variations in students views of academic success and their relevant background knowledge (qualitative)

ID C GPA Quotation 0607 4 4.00 I do not have any language problems. 2020 3 4.00 I understand 80% of the lectures. Careful reading complements 20% of lack of understanding. 1315 3 4.00 My biggest problem is related with speaking in English. I will get a good grade because my mathematical background is strong. 0609 3 4.00 The professor is old so that his pronunciation is not clear. He usually handed out important contents and I could understand it by reading the textbook. 0620 2 4.00 I understand only 60-70% of the lectures. It has made my scores less than my expectation. 2036 3 3.57 The major problem is speaking. I spoke once or twice during the whole semester. 0610 3 3.53 Listening is a problem. Lack of cultural knowledge interferes with understanding the concept. 2037 2 3.22 Lack of knowledge about idiomatic expressions prevents me from understanding questions on the homework assignment. 0608 4 2.89 I understand almost 100% of the lectures. I understand the professor completely. [ ] Grades are not important. That is the least of my worries.

Argumentation in Schoonenboom & Johnson 2021 English language skills do not affect course performance (quantitative) English language skills affect course performance (qualitative) unless students deliberately use compensation strategies (mixed)

Lee and Greene (2007) Schoonenboom and Johnson (2021) English language skills affect course performance (qualitative) English language skills do not affect course performance (quantitative) English language skills do not affect course performance (quantitative) English language skills affect course performance (qualitative) because of variations in students views of academic success and their relevant background knowledge (qualitative) unless students deliberately use compensation strategies (mixed)

Warum qualitative Methoden nicht zeigen, was hinter dem quantitativen Effekt steckt Der quantitative Effekt entwickelt sich in der Folge der qualitativen Untersuchung weiter und h rt auf, in seiner alten Form zu existieren. Neue Effekte entstehen, meistens unterschiedliche Effekte f r unterschiedliche Grundgesamtheiten. Diese Weiterentwicklung ist der Mehrwert der Mixed-Methods-Forschung. What works for whom in what circumstances (Pawson and Tilley 1997, Realistic evaluation, p. 85)

Literatur Lee, Y.-J., & Greene, J. (2007). The predictive validity of an ESL placement test: A mixed methods approach. Journal of Mixed Methods Research, 1, 366-389. https://doi.org/10.1177/1558689807306148 Pawson, R., & Tilley, N. (1997). Realistic evaluation. Sage. Schoonenboom, J. (2019). Develop your case! How controversial cases, subcases, and moderated cases can guide you through mixed methods data analysis. Frontiers in psychology, 10(1369). https://doi.org/10.3389/fpsyg.2019.01369 Schoonenboom, J., & Johnson, R. B. (2021). The case comparison table: A joint display for constructing and sorting simple tables as mixed analysis. In A. J. Onwuegbuzie & R. B. Johnson (Eds.), The Routledge reviewer's guide to mixed methods analysis (pp. 277- 288). Routledge. https://doi.org/10.4324/9780203729434-24

Vielen Dank fr Ihre Aufmerksamkeit! judith.schoonenboom@univie.ac.at