Learning Progression and Probabilistic Framework in SCM Model

Saltus and the SCM Model Revisited Presenter: Yunting Liu Advisor: Mark Wilson

Problem background: Learning Progression Learning Progressions (LPs) describe the way the learner successively develops more sophisticated ways of thinking about an important domain of knowledge and practice A common method for developing an LP is to anchor the top end with the level of understanding expected at the end of a course. LPs are often tested by creating assessment items that correspond to different levels of understanding (Briggs et al., 2006; Breslyn et al, 2016, Jin et al., 2019) . BAS four building blocks for constructing quality assessments (Wilson, 2005), including four parts: Construct Maps, Item Design, Outcome Space, Measurement Model.

Probabilistic framework ?(???= 1|???= 0) ? ???= 1 ???= 1 ? ???= 1 = ???(?(???= 1)) A mathematical representation would be viewing it as a series of ordered latent classes, where the regression relationship relates to the probability of a person being in the two specific latent classes i.e., being in one level of the prerequisite construct (prerequisite construct denote as ?, thus person p achieve level k for construct ? is elaborated as ???= 1) will make it more likely for a person to be in a specific level of the target construct (target construct denote as ?, thus person p achieve level j for construct ? is elaborated as ???= 1), as explained above. The function ???may take a variety of forms, the simplest being a linear relationship.

Overview of the presentation Study background/ Purpose of the study Multidimensional IRT model Structured Construct Model Model formulation Saltus Model ADMSR data Simulation study Compare & Contrast Discussion

Model 1: Model 1: Multidimensional IRT model A unidimensional Rasch model ???(?? ??) 1+???(??? ??) ? ???= 1 ?? = i and p index item and person, respectively; and b represent latent trait and item difficulty, respectively. Usually, the constraint of the model is to either set the mean of person proficiency to 0 or the mean of item difficulties to 0. A multidimensional Rasch model(between item) ???(??? ??) 1+???(??? ??) ? ???= 1 ??? = a subscript, d, is added to differentiate latent traits between different dimensions. Typically, the constraint for the model is ????= 0, fixing the mean of item difficulty on each dimension to be 0, but this may need to be modified in a Learning Progression.

Between item Between item Multidimensional IRT model A direct comparison between different different dimensions In multidimensional models, a direct comparison of the item and student distributions cannot be made without a transformation to the parameters for the multidimensional model, and delta dimensional alignment method (DDA) is commonly used (Schwartz, Ayers & Wilson, 2017; Feuerstahler & Wilson, 2021). After a common scale is established, levels of threshold can be compared between different dimensions. Reliability and Validity of the Measurement tool the reliability of the measurement will increase relative to the consecutive approach in each dimension, close to the unidimensional approach. The underestimated/attenuated correlation between different dimensions can also be corrected (Wang, 1999).

Model 2 Model 2: Structured Construct Model In a time-series analysis

Structured Construct Model -- Based on change point analysis(Shin, Wilson & Choi, 2017) Reparameterized Rasch model (for a group of item in construct R), no discontinuity; Constraint: person mean set to zero. the estimation of the change point ? the average magnitude of abrupt change at the change point ?

Model 3: Model 3: The saltus model (Wilson, 1989) A psychological sequence: The theory of Piaget Example: Subtraction problem with borrowing numbers An empirical sequence, the observed order of learning First order discontinuity: a sudden spurt in development on a single developmental scale Second order discontinuity: an abrupt change in the relation between performance for two or more domains or conditions Can also be framed as a Structured Mixture Item Response Model(Lee, Suh & Jeon. 2024)

Model formulation(also known as the StrMIRT model ) subject group membership, summarized by the vector ??, Item indicator variable ???= 1 if item j belongs to group k, exp(?? ??+ ?? ???? ?) 1+exp(?? ??+ ?? ???? ?) ? ???= 1 = A 2-group formulation Saltus parameter ? ,?11= ?12= ?21= 0 for identification exp(?? ??+??2??2?22) 1+exp(?? ??+??2??2?22) ? ???= 1 =

Comparison between three proposed models Model MIRT SCM with change point The Saltus 1. Unidimensional model versus multidimensional model Multidimensional Rasch, SCM: Multidimensional; The saltus model: Unidimensional Disattenuated correlation 2. Diagnostic information The estimation of group (dichotomous variable) 3. Learning progression versus attribute hierarchy 4. Effect size/ Class separation Multidimensionality Yes Yes No Diagnostic information/ Latent class information No Yes Yes Learning progression / Attribute hierarchy No Learning progression attribute hierarchy Require class separation No No Yes

Hierarchical structures versus Learning Progression Possible Hierarchical structure between six related constructs (adopted from Leighton, Gierl & Hunka, 2004)

Data description Data Modeling and Statistical Reasoning Learning Progression(ADMSR) DaD: Data Display, trace a progression of learning to construct and read graphical representations of the data from an initial emphasis on cases toward reasoning based on properties of the aggregate. CoS: Conceptions of Statistics, how student come to recognize the statistics measure qualities of the distribution, such as center/spread, then develop understandings of stat as generalizable and as subject to sample-to-sample variation Cha: Chance, how chance and elementary probability operate to produce distributions of outcomes. MRC: Meta-Representational Competence, making claims about data and to consider trade- offs among representations in light of these claims. MoV: Models of Variability, employing chance to model a distribution of measurement. InI: Informal Inference, students inferences based on single or multiple samples.

Hypothesized Learning Progression CoS 4 CoS 0.76 0.75 Cha 6 DaD Cha Cha 5 0.75 Cha 4 DaD 5 DaD 4 CoS 3 Cha 3 There are N=1002 respondents who participated the study, with 59 items in total (measuring the above mentioned three dimensions, 21 items measuring DaD, 18 items measuring CoS, 20 items measuring Cha), yet due to the missing by block design, nearly 60% of the data is missing. Each participant answered 21 items on average. DaD 3 CoS 2 Cha 2 DaD 2 CoS 1 Cha 1 DaD 1

Primary result from Shin, Wilson & Choi(2017) The original polytomous responses on the same items were dichotomized (i.e., CoS 1,2 nonproficient (score 0); CoS 3,4 proficient (score 1); Cha 1,2 nonproficient (score 0); Cha 3,4,5,6 proficient (score 1)) so that being proficient in the CoS construct was considered to be the requirement to reach the proficient level in the Cha construct, as shown in Figure 1. In the analysis, 489 middle school students who answered at least two items in each construct were included. the cut-score estimate from the change-point SCM is not statistically different from zero, incorporating the discontinuity parameter appears to be more crucial for these empirical data than incorporating the cut-score parameter. Magnitude & Interpretation: In the change-point SCM, the discontinuity parameter, , was estimated to be 1.63 with a standard error of 0.18: the examinees at the proficient level of CoS had an advantage in reaching the proficient level in Chance as much as 1.63 logits on average.

Interpretation: 1. From DaD to Cha, The focal group, which has an average of 0.45, has a higher overall performance compared to the reference group, whose mean is fixed at 0. The Saltus parameter suggests that the focal group demonstrates better performance on Cha item (item from the target dimension). The estimate from SCM is not significant(as the lambda parameter is not significantly different from 0). 2, From DaD to CoS. The focal group, which has an average of 0.41, has a higher overall performance compared to the reference group. The Saltus parameter suggests that the focal group demonstrates significantly better performance on CoS item (item from the target dimension). ). The estimate from SCM is not significant(as the gamma parameter is not significantly different from 0).

Simulation study - why these parameters value why they might be important 1. Correlation between two dimensions (? = 0.7,0.8,0.9) SCM is inherently a multidimensional model yet Saltus is a unidimensional model 2. Discontinuity/Saltus parameter (small: 1, medium 1.5) How large the discontinuity needs to be to allow identification 3. Group location(small: 0.5; medium: 1; large: 1.5) How qualitatively different two groups are at. 4. Mixing proportion(balance; imbalance) Balanced/imbalanced design

I = 500, J = 60 Discontinuity parameter = 1, Correlation between dimensions = 0.7 Generating model Parameter name SCM Saltus True value N(0,1) 1 (1,1) SCM estimates Cor = 0.99 1.10(0.15) (1.01,0.93) Saltus estimates Cor = 0.99 1.35(0.07) True value N(0,1) 1 SCM estimates Cor=0.98 1.46(0.11) (1.14,0.95) Saltus estimates Cor = 0.99 0.96(0.06) Beta1~beta60 Lambda/saltus person_sd corr Rho 0.7 0 0.67(0.05) -0.07(0.07) 0.95(0.01) -0.03(0.06) Recovered rho Person_mean(focal group) Person_sd(ref group) Person_sd(focal group) Mixing proportion % of group membership overlap 0.1 0.23(0.13) -0.2 0.99(0.06) 1 1.00 1.02 0.5 0.5 0.49 0.52 50% 51% 92% 47% 67% 0.5 52% 99% 52% 99%

I = 500, J = 60 Discontinuity parameter = 1, Correlation between dimensions = 0.9 Generating model Parameter name SCM Saltus True value SCM estimates Cor = 0.99 1.05(0.12) (1.09,0.98) Saltus estimates Cor = 0.99 1.13(0.07) True value SCM estimates Cor = 0.99 0.64(0.13) (1.47,1.63) Saltus estimates Beta1~beta60 Lambda/saltus person_sd N(0,1) 1 (1,1) N(0,1) 1 1.06(0.08) corr Rho 0.9 0 0.85(0.03) -0.12(0.06) 0.96(0.01) -0.06(0.13) Recovered rho Person_mean(focal group) Person_sd(ref group) Person_sd(focal group) 0.2 0.89(0.15) 1 0.93(0.12) 0.92 0.95 1 1 1.14 1.05 Mixing proportion % of group membership overlap 50% 52% 95.8% 53.2% 85.2% 0.5 52.6% 90% 49% 93%

Comparison between three proposed models Model MIRT SCM with change point The Saltus 1. Unidimensional model versus multidimensional model Multidimensional Rasch, SCM: Multidimensional; The saltus model: Unidimensional Disattenuated correlation 2. Diagnostic information The estimation of group (dichotomous variable) 3. Learning progression versus attribute hierarchy 4. Effect size/ Class separation Multidimensionality Yes Yes No Diagnostic information/ Latent class information No Yes Yes Learning progression / Attribute hierarchy No Learning progression attribute hierarchy Require class separation No No Yes

Discussion How to unify the three models What s the pros and cons of both models How to prevent misspecification Can one of them be more generalizable to the problem, under certain constraints? Can we create a roadmap to solve the problem of validating learning progression? SCM model w low ? CP Validate LP(s) Validity check Data & Hypothesize Between item MIRT model Saltus model High ? How can(if applicable) both model be applied to a longitudinal setting?