Cognitive Transfer in Simulation-Based Intro Statistics Curriculum

COGNITIVE TRANSFER OUTCOMES FOR A SIMULATION-BASED INTRODUCTORY STATISTICS CURRICULUM Matthew Beckman Pennsylvania State University Robert delMas Joan Garfield University of Minnesota

OVERVIEW Learning Statistics: Intrinsic & Extraneous Cognitive Load Cognitive Transfer: Near and Far Simulation-Based Methods: How they address Cognitive Load Comparative Study: Simulation-Based course and Non-Simulation-Based courses Discussion of Results

LEARNING STATISTICS Discipline of statistics has high intrinsic cognitive load (Paas, 1992; Sweller, 1994) concepts are difficult and highly interrelated Extraneous cognitive load: Cognitive load added by teaching methods (e.g., covering too much content too quickly for students to adequately process) Need teaching methods (e.g., scaffolding) that help students learn concepts and connections between concepts But, teaching statistical concepts and procedures may not be enough to develop transfer (Garfield, 2000; Garfield et al., 2012; Lovett & Greenhouse, 2000).

COGNITIVE TRANSFER Ability to apply skills and knowledge students learn in a course to novel problems (Cox, 1997; Singley & Anderson, 1989). Distinction between Near and Far transfer Near: tasks & contexts highly similar to curriculum tasks Far: little or no explicit training in the context of the new task (in this study) Transfer requires a rich schema for the content area built slowly by experiencing numerous problems (e.g., Bransford, Brown, & Cocking, 2000) Transfer requires intentional training (Perkins & Salomon, 1988) and understanding constructed by the students (Broers, Mur, & Bude, 2004)

SIMULATION-BASED METHODS Argued to be (Cobb, 2007; Efron, 2000; Ernst, 2004; Garfield et al., 2012; Rossman, 2008 ): Conceptually simpler than Normal theory procedures Align well with student s intuitions Principles generalize to many situations May reduce extraneous cognitive load, making it easier to understand and connect concepts such as: Sample variability Sampling distributions p-values & Confidence intervals Connecting concepts promotes rich schema

CATALST: A SIMULATION-BASED INFERENCE CURRICULUM Develops statistical thinking through bootstrapping and randomization testing methods (Garfield et al., 2012) TinkerPlots used to build models and simulations Three Modules Chance Models: Develop understanding of chance models and simulation methods; segue to use of p- values to quantify evidence against a model Comparing Groups: Develop understanding of how to build simulations to conduct randomization tests Estimate Effects: Develop understanding of how to build simulations for bootstrap confidence intervals

THE STUDY Fall 2011 and Spring 2012 729 students from 8 institutions Compared two implementations of CATALST course to non-CATALST introductory statistics courses CAT-MINN: CATALST course at the University of Minnesota (4 classes; N = 138) CAT-REPL: CATALST course at 5 other institutions (5 classes; N = 151) NON-CAT: Non-CATALST courses at 7 other institutions (10 classes; N = 440; 5 institutions same as CAT-REPL institutions) Neither students nor classes were randomly assigned to course

ASSESSMENT INSTRUMENT 21 multiple-choice items 16 items adapted from CAOS test (delMas et al., 2007) Five new items on randomization-based methods and understanding statistical significance Topics: random sampling & assignment; generalization of results; variability; sampling variability; informally compare groups; understanding of null hypothesis; interpret p-values; interpret confidence intervals; interpret statistical significance Testlets: Items that shared a common context were treated as a testlet (proportion correct); 8 single items and 5 testlets.

NEAR & FAR TRANSFER Each assessment item mapped to the frequency and type of contact in CATALST curriculum Type of Contact: Direct Instruction: Concept specifically identified and instruction designed to develop understanding of the concept Indirect Instruction: Concept not identified in instruction, but learning experience can lead to understanding of the concept Practice: Number of times concept is required in in- class activities or homework assignments

NEAR & FAR TRANSFER Measured Learning Goal Purpose of random assignment Factors that allow a sample of data to be generalized to the population Correlation does not imply causation Match a scatterplot to a verbal description of a bivariate relationship Expected patterns in sampling variability Near Transfer: Direct instruction (9 scores) Direct 3 Indirect 3 Practice 3 Total 9 Transfer Near 3 - 4 7 Near - 5 2 7 Far - - - 0 Far 2 3 - 5 Near Far Transfer: No direct instruction (6 scores)

CONCEPTS by TRANSFER NEAR FAR RandomAssignment Generalization to Population Representative Sample Correlation Causation Trend in a Scatterplot Variability (Repeated Measurements) Understanding of Statistical Significance Sampling Variability Compare Groups Understand Null Hypothesis Interpret p-Values Interpret Confidence Intervals Interpret p-Values Interpret Confidence Intervals

LME MODEL Linear mixed effects (LME) model with random intercepts Fixed effects for curriculum group (CAT-UMINN, CAT-REPL, NON-CAT) and item transfer type (NEAR vs FAR) Random effect for random intercepts within class (19 levels) 2 contrasts for curriculum group CAT-UMINN CAT-REPL CAT-REPL NON-CAT Contrast for item transfer type: NEAR FAR

MODEL ASSUMPTIONS Intra-Class Correlation (ICC): 0.225 No evidence of extreme violations of assumptions Levene s Test: (F(5, 1452) = 1.624, p = 0.150), Box s M Test: M(6) = 10.633, p = 0.100

RESULTS Good precision in random effects estimates Random Effect Class Student level standard deviation 0.172 [0.166, 0.178] Estimate [95% CI] 0.069 [0.044, 0.093] All fixed effects statistically significant Fixed Effect Curriculum Transfer Type Interaction df F 7.461 20.112 5.713 p 2, 15.1 1, 1435.1 2, 1435.1 0.0056 < 0.0001 0.0034

CONTRASTS Fixed Effect Curriculum group: UMINN REPL Curriculum group: REPL NON-CAT Transfer item type: Near Far UMINN REPL by Transfer REPL NON-CAT by Transfer df t p 15 15 3.030 3.541 4.485 1.373 3.380 0.008 0.003 < 0.001 0.170 < 0.001 1435 1435 1435

DISCUSSION CATALST more effective than Non-CATALST courses CATALST students outperformed Non-CATALST students on both Near and Far transfer items Limitation: Non-CATALST curriculum varied All items covered important topics in introductory statistics On Far transfer items, CATALST effect most pronounced for UMINN group Instructors may require more training and experience to achieve Far transfer effects

REFERENCES Bransford, J., Brown, A. L., & Cocking, R. R. (2000). How People Learn: Brain, Mind, Experience, and School: Expanded Edition, Washington: National Academy Press. Broers, N. J., Mur, M. C., & Bude, L. (2004). Directed self explanation in the study of statistics. In G. Burrill, & M. Camden (Eds.), Curricular development in statistics education (pp. 21-35). Voorburg, The Netherlands: International Statistical Institute. Cobb, G. W. (2007). The introductory statistics course: A Ptolemaic curriculum. Technology Innovations in Statistics Education, 1(1), 1-15. Cox, B. D. (1997). The rediscovery of the active learner in adaptive contexts: A developmental-historical analysis. Educational Psychologist, 32(1), 41-55. doi:10.1207/s15326985ep3201_4 delMas, R., Garfield, J., Ooms, A., & Chance, B. (2007). Assessing students conceptual understanding after a first course in statistics. Statistics Education Research Journal, 6(2), 28-58. Efron, B. (2000). The bootstrap and modern statistics. Journal of the American Statistical Association, 95(452), 1293-1296. Ernst, M. D. (2004). Permutation methods: A basis for exact inference. Statistical Science, 19(4), 676-685. Garfield, J. (2002). The Challenge of Developing Statistical Reasoning. Journal of Statistics Education, 10(3). Garfield, J., delMas, R., & Zieffler, A. (2012). Developing Statistical modelers and thinkers in an introductory, tertiary-level statistics course. ZDM: The International Journal on Mathematics Education, 44(7), 883-898. Lovett, M. C., & Greenhouse, J. B. (2000). Applying Cognitive Theory to Statistics Instruction. The American Statistician, 54(3), 1-11. Paas, F. G. W. C. (1992). Training Strategies for Attaining Transfer of Problem Solving Skill in Statistics: A Cognitive-Load Approach. Journal of Educational Psychology, 84(4), 429-434. Perkins, D. N., & Salomon, G. (1988). Teaching for Transfer. Educational Leadership, 22-32. Rossman, A. J. (2008). Reasoning about informal statistical inference: One statistician s view. Statistics Education Research Journal, 7(2), 5-19. Retrieved from http://iase-web.org/documents/SERJ/SERJ7(2)_Rossman.pdf Singley, M. K., & Anderson, J. R. (1989). The transfer of cognitive skill. Cambridge, MA: Harvard University Press. Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and instruction, 4(4), 295-312.

Robert delMas University of Minnesota 250 Education Sciences Building Minneapolis, MN 55455 delma001@umn.edu 612-625-2076