Advanced Troubleshooting Solutions for Psychometric Analysis

Troubleshooting Inadmissible & Problematic Solutions HSE Psychometric School August 2019 Prof. dr. Gavin T. L. Brown University of Auckland Ume University

Trouble-Shooting Summary Estimation problems are quite common Small sample size Poor model specification Over-factored constructs Solutions must fit and be theoretically sound Remove factors, items Use modification indices Solutions need to be tested Alternative structures Invariance to other groups And then we can address the substantive So What? question, with good measurements

Inadmissible solutions Multiple causes of inadmissibility Gerbing, D. W., & Anderson, J. C. (1987). Improper solutions in the analysis of covariance structures: Their interpretability and a comparison of alternate respecifications. Psychometrika, 52(1), 99-111. Causes: Specifying a sub-factor of items when participants do not actually make such fine-grained distinctions Chen, F., Bollen, K. A., Paxton, P., Curran, P. J., & Kirby, J. B. (2001). Improper solutions in structural equation models: Causes, consequences, and strategies. Sociological Methods & Research, 29(4), 468-508. doi:10.1177/0049124101029004003 Not having enough items for the factor Marsh, H. W., Hau, K.-T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Research, 33(2), 181-220. 10.1207/s15327906mbr3302_1 Not having enough people to generate stable estimates (ideally n>400) Too much missing data that has been imputed Factors that are too highly inter-correlated

Negative Error Variance Also known as a Heywood case When the minimum of the discrepancy function is obtained with one or more negative values as estimates for the variance of the unique variables. Explaining more than 100% of variance causes the error to be less than zero (negative). NOT logical or acceptable. Solutions: 1. Remove offending factor and have items predicted by higher-order factor 2. Subordinate the factor to the other factor 3. Fix value to small value >0.00 (e.g., .005) if 2*se includes 0.00

A Useful Application of CI logic: Negative error variance What if a point estimate is negative when in reality it should be positive? If the residual is estimated to be -0.10 this is a violation of reality because we cannot explain more than 100% of variance But if the se is 0.06, then it is possible the TRUE value of the residual is NOT negative -.10 + (2*.06)=.02 Hence, it is possible to overcome this problem by fixing the residual to a small positive value (e.g., .005) But do this only if other studies clearly indicate the value is not reported as negative

Fixing a negative error variance If 2 standard errors (se) are greater than observed value, it is highly likely (95% CI) that the TRUE value is not negative. Hence, it can be fixed to .005. If previous studies have shown that the value is normally >0.00 Do structural causes explain why variance might be negative? Was it negative in other study conditions?

Error Variances Fixed @.005 Admissible Solution 28. School e4 1 e36 e37 e38 e39 1 40. e12 1 1 15. e1 e671 1 1 .66 15. e1 e67 e3 .69 28. .69 e3 35. 1 e64 School 35. .75 1 1 1 1 e4 42. e64 e5 8. 27. 32. 55. .58 42. Teaching Improvement e5 .74 8. 27. 32. 55. e36 e37 e38 e39 Accountability -.28 1 .59 .73 Teaching Improvement 1 Accountability 1 17. 25. 40. e9 1 1 .62 .68 17. 25. e9 .80 1.10 e10 Student .78 1 .74 e10 Student e12 e69 e69 e341 56. 1 56. e34 .76 53. .23 e33 1 53. e33 24. .75 .67 1 e32 24. .71 e32 1 .45 1. 1. Learning Improvement e29 Learning Improvement e29 1 .75 43. e71 43. e71 1 52. e43 52. e43 .68 0.005 Describe 1 1 Describe 1.00 .64 41. 1.67 41. .62 e42 1 e42 0.005 1 e68 1 1 1 1 e68 1 1 .66 34. 34. e41 e41 33. 39. 44. 62. 33. 39. 44. 62. e59 e60 e61 e62 e59 e60 e61 e62 e65 e65 .60 .68 1 -.81 Exam-oriented Exam-oriented .76 60. .72 60. e48 e48 1 54. .73 .62 e47 54. -.44 e47 1 .92 Valid 47. .66 Valid e46 1 47. e46 e451 12. 1 e45 .33 12. .54 .57 1 -.32 64. e66 64. e66 e13 e13 e70 .72 e70 1 1 1 1 1 1 1 1 50. 48. 4. e14 e20 50. 48. 4. e14 .75 e20 -.25 10. e15 .33 Nurturance e21 10. e15 1 e23 e24 1 Nurturance .58 e21 1 .52 23. e22 22. Irrelevance 23. e22 e23 e24 e18 .62 22. .63 Irrelevance 26. e18 26. NB NB. When only 1 predictor explained will be .995 ( 1.00) .61 .66 .75 31. 31. .54 1 63. e57 63. e57 .73.74 .59 51. 46. 38. 21. 1 e55 1 Control 51. 46. 38. 21. .56 .54 e55 1 e54 Control e54 e53 1 e53 e52 1 e52

Not Positive Definite When ALL eigenvalues (the diagonals) in a matrix are NOT >0 At least 1 factor is linearly dependent on another (collinearity) This can be extremely difficult to identify and solve Here at least 2 causes exist: over-correlation of factors + negative error variance

Fixes for Nonpositive Definite Covariance Matrices Goal: Keep the same theoretical framework so that the analysis tests your theory Not just dredging through data to discover fixes Too much chance in that process

Resolving Inadmissible Solutions Reduce the number of factors by joining the factors which are linearly dependent 1. Remove a sub-factor and join the items to the first factor The meaning remains the same (just lose some precision)

1 1 1 1 1 1 1 1 1 1 1 1 1 1 Resolving Inadmissible Solutions 35. School 1 1 1 1 1 35. 1 53. e33 School 1 1 1 e5 42. 1 1 56. e34 1 1 53. 57. e33 e6 e5 42. 1 e64 1 2. Make an over-correlated factor into a sub-factor of the other 57. e64 1 e8 1 5. e35 56. 61. e34 1 1 1 e7 1 e6 1 8. e36 5. e35 1 1 1 e7 61. Teaching Improvement 1 27. e37 1 1 8. 3. e36 1 Student 1 1 1 Teaching Improvement 32. e38 1 27. 17. e37 1 1 e9 e8 3. 1 1 Student 1 55. e39 1 32. 25. e38 1 e61 e10 1 e9 17. 1 e61 1 e11 1 e65 55. 30. e39 1 1 18. e40 e10 25. 43. 1 1 1 1 e65 1 34. e41 1 43. e12 18. 40. e40 1 e11 30. 1 1 1 Describe 41. e42 34. e41 1 Improvement e12 40. 1 1 Describe 41. 11. e42 52. e43 Improvement 1 e19 1 52. 22. e43 e66 1 1 1 6. e44 1 e19 11. 1 e18 e66 1 1 1 1 6. e44 1 12. e45 1 1 e18 22. e17 29. 1 1 1 12. 37. e45 1 47. e46 Valid Nurturance 1 e17 29. 1 e16 1 47. e46 Valid 54. e47 Nurturance 1 e16 37. 1 1 1 1 e15 54. 48. e47 60. e48 1 1 e15 48. 1 60. 50. e48 e69 e14 1 2. e49 1 e14 50. 1 1 1 2. e49 e13 After After 64. 9. e50 1 1 Before Before very high correlation 1 1 e13 64. 9. e50 1 e67 13. e51 1 1 a dependent regression 13. 4. e51 1 e20 Irrelevance 1 21. e52 1 1 1 e20 4. 21. 10. e52 1 1 e21 38. e53 1 1 1 e21 10. 38. 23. e53 Irrelevance Control 1 1 46. e54 1 e22 1 46. e54 e22 23. 1 1 51. e55 1 e23 51. 26. 1 e55 1 e23 26. Control 59. e56 1 1 59. 31. e56 e24 1 1 e24 31. 63. e57 1 Instrumental e68 1 63. 36. e57 e68 e25 1 e25 36. 1 1 1 7. e58 1 1 1 1 7. e58 1 45. 1 1 45. 1 1 1 1 1 1 1 1 1

Resolving Inadmissible Solutions .41 e436 Assessment interferes w ith teaching e43 e436 Assessment interferes w ith teaching .58 .31 e434 .47 Assessment is unfair to students Bad e434 Assessment is unfair to students e433 .52 Small Samples problematic Small Samples problematic Admissible Admissible (N=82) Remove sub-factors Join highly correlated factors Keeps interpretation similar .97 Assessment forces teachers to teach in a w ay against their beliefs e442 e433 .31 Assessment forces teachers to teach in a w ay against their beliefs .56 e423 .65 Assessment results are filed & ignored 1.44 Irrelevance e423 .42 F1 Assessment results are filed & ignored Irrelevance e422 .42 .50 Teachers conduct assessments but make little use of the results e422 .66 Teachers conduct assessments but make little use of the results e421 .59 Assessment has little impact on teaching e440 e421 .26 Assessment has little impact on teaching e437 .18 .30 Assessment is an imprecise process e437 .53 .66 Assessment is an imprecise process inaccuracy e438 Assessment results should be treated cautiously because of measurement error .44 e438 Assessment results should be treated cautiously because of measurement error .38 e439 .40 Teachers should take into account the error and imprecision in all assessment .36 e439 Teachers should take into account the error and imprecision in all assessment .24 e344 Assessment is assigning a grade or level to student w ork Accountability Students e344 .36 Assessment is assigning a grade or level to student w ork e314 .34 Assessment determines if students meet qualifications standards This approach necessary especially when N is low. .41 e314 .36 .46 Assessment determines if students meet qualifications standards 1.40 e313 Assessment places students into categories .67 .54 e313 Assessment places students into categories .26 e323 Assessment is a good w ay to evaluate a school Accountability Schools .27 e323 .39 Assessment is a good w ay to evaluate a school e324 Accountability Schools Assessment provides information on how w ell schools are doing .40 .41 e324 Assessment provides information on how w ell schools are doing e325 Assessment is an accurate indicator of a school's quality .92 .39 e325 Assessment is an accurate indicator of a school's quality e441 e111 Assessment results can be depended on .48 .39 e111 Assessment results can be depended on .48 F2 e112 .68 Assessment results are consistent .44 .85 e112 .43 Assessment results are consistent e114 Assessment results are trustw orthy e114 e133 .43 .08 Assessment results are trustw orthy e13 .95 Assessment establishes w hat students have learned e134 .24 e133 4.15 Describe Assessment establishes w hat students have learned Assessment measures students higher order thinking skills .12 e135 .36 e134 Assessment measures students higher order thinking skills .09 Assessment is a w ay to determine how much students have learned from teaching .49 Improvement e135 1.26 e21 Assessment is a w ay to determine how much students have learned from teaching e212 .09 Assessment information modifies ongoing teaching of students .21 Improvement .45 e212 Teaching Assessment information modifies ongoing teaching of students ex3 .55 Assessment is integrated w ith teaching practice Inadmissible Inadmissible N=82 1.62 ex3 .35 Assessment allow s different students to get different instruction error variances <0.00 + correlations >1.00 e216 Assessment is integrated w ith teaching practice .40 e224 .55 e216 Assessment allow s different students to get different instruction .37 .27 Assessment provides feedback to students about their performance .41 e224 F3 e225 Assessment provides feedback to students about their performance Assessment feeds back to students their learning needs .40 e225 Assessment feeds back to students their learning needs .47 e227 e22 Assessment helps students improve their learning e227 Assessment helps students improve their learning

Resolving Inadmissible Solutions Alternative Model Alternative Model 2: 2: Add paths Base Model Base Model (did not fit a new group) Alternate Model 1: Alternate Model 1: Remove problematic item PS PS. Both revisions worked

Resolving Inadmissible Solutions Parcel factor into a single variable and pretend that it was measured directly 8 items, 2 factors becomes 2 correlated variables when n reduces from 86 to 20. Note Note the similar correlation. .30 e3 e2 .55 Moves e1 .37 .16 .49 -.16 .65 .42 PostTraining Structure/Content Course-work essay Structure/Content PreTraining Structure/Content Focus e2 .00 .57 .33 .18 e1 Material .40 Structure Content e3 .83 .68 .59 Authority e4 -.40 .49 .34 Course-work essay Department grade .58 .55 .57 e7 Mechanics e15 .44 .27 .00 .52 Tone e16 .50 .71 -.20 e4 .27 Style Flow e17 .50 PostTraining Style Course-work essay Style PreTraining Style .25 .00 Originality e18 .65 .00 .07 .43 e5 e6

Improving Fit If solution is recursive and admissible, but does not fit well, what to do? Remove paths and items that are not statistically significant Remove items that have high attraction to logically inappropriate factors (use the modification indices to identify those) Remove items with weak loadings on their respective factors Correlate all the factors with each other Collapse factors Parcel factors into scale scores

Modification index Also known as Lagrange multiplier an estimated value in which the model s 2 test statistic would decrease if a fixed parameter were added to the model and freely estimated Statistically significant at value of 3.84 which corresponds with 1 df at alpha=.05 But generally values under 20 make almost no difference to fit indices Use the values to identify paths that could be added to make the fit better Alternately use the MI to identify paths or items that could be removed. Items with large attraction to factors that violate simple structure are candidates for removal Items with large attraction to other items within their own set are candidates for deletion Highly subjective in decision making: I tend to sum the MI for an item to see which are most problematic. Removing a few items in light of theoretical considerations is the goal

Expected parameter change EPC indicates the estimated value of a fixed parameter if it were added to a model and freely estimated a direct estimate of the size of the misspecification for the restricted parameters fixed parameters associated with the largest EPC value indicate the most model misspecification Must be evaluated in terms of theoretical plausibility and could be freely estimated in the model. standardized EPC invariant to the rescaling of observed and latent variables SEPC values for residual covariances between latent variables and between observed variables are standardized using their respective residual variances instead of their variances the SEPC tends to outperform the MI when arriving at the correct confirmatory model.

Tweaking models What to look for? Items that have strong sum of MI/SEPC These are items that are most mis-specified Your theory does not match participant responses Items that are highly attracted to multiple factors These violate simple structure; Remove these IF and ONLY IF there are sufficient items in a factor to keep a valid factor Factors that want to be joined to other factors Consider this IF and ONLY IF the path makes some sort of theoretical sense can you explain it? Make it dependent or correlated Remember Change only if you can explain it with your theory Don t delete items that would destroy a factor

e47 e48 e49 e51 e52 Tweaking models q42 q35 q57 q58 q36 .45 .83 .76 .63 .75 5. Teacher Quality e34 Q15 7. Error .74 e33 .82 .63 Q28 .13 4. School Quality .76 .20 # manifest variables Gamma hat e29 q19 .06 df Model RMSEA .38 .25 .18 Before 3 items removed .14 33 474 0.066 0.889 .77 e8 .42 q1 e38 Q26 .58 e37 .65 Q31 .62 .44 .46 .60 .17 .21.58 e10 e35 After 3 items removed .75 2. Irrelevant q3 Q49 .47 .81 3. Help Students e62 30 384 0.061 0.913 e11 q45 q5 .64 .55 -.31 .66 e63 q10 e68 e13 q23 q11 This change reduced the df considerably while increasing the RMSEA somewhat. Consequence is that the model fit for gamma hat is clearly acceptable. Fit is improved. -.05 .33 .28 .53 e28 Q14 e27 .68 Q20 .69 e26 Q22 .70 .78 .61 e25 1. Personal Development Q24 But does it still mean the same? What did I lose in gaining better fit? 6. Examination .68 .57 .63 .64 .67 .58 .77 .67 e23 q34 e22 q27 Q62 Q39 Q33 q38 q40 e66 q13 e46 e45 e44 e43 e64

Jamovi: SCoA6 Modification Indices Default display >10; change to >20 and you will more easily focus on items with big problems Examples: sq1, sq2 both want to go on lots of factors. It was designed to go with SF, but strongest on TI. So test the alternative models? ce5 wants to go with 5 other factors. There are 6 items in the set. Delete ce5? If so, what impact on fit AND meaning? sf2 also behaves like ce5. test?

Lavaan MI Syntax Syntax #displays MI in decreasing order, and print only those that would substantially improve model fit i.e., changes in x2 square greater than 20.00 with df = 1) modind_scoawls <- modindices(SCoAwls_fit, sort. = TRUE) modind_scoawls[modind_scoawls$mi > 20.00 , ] NB NB: the sort is MI not SEPC, change $mi to $sepc.all I transferred data to Excel, highlighted MI down to 100, re- sorted on SEPC.all and highlighted down to .40 Top problems: IG predicts bd1; bd1 violates simple structure ti4 correlates with ti5; ti4-5 one is redundant? pe1-2 want regress onto ce1-2; subordinate factor? Do 1 at a time; inspect impact

Improving Fit 1 1 What NOT Correlate residuals It will improve fit but what does it mean? Residuals are meant to be random in their patterns but when correlated Everything I can t explain is systematically related to everything I can t explain and I can t explain how or why it is related . Do you believe this? if not don t do it! Plausible in longitudinal studies though NOT to do 1 1 1 1 1 1 1 1 1 1 If these do not produce acceptable fit, rethink your model and theory Use EFA to see how the items in this sample really do aggregate 1

Jamovi: SCoA Residual MI Yes, the info is here but if you cannot give a clear rationale for correlating residuals, please don t do it. It will inflate fit but it violates the assumptions we work with. If items are within a factor, give serious consideration to deleting 1

Summary Estimation problems are quite common Small sample size Poor model specification Over-factored constructs Solutions must fit and be theoretically sound Remove factors, items Use modification indices and SEPC Solutions need to be tested Alternative structures Invariance to other groups And then we can address the substantive So What? question, with good measurements