Hope College Student Responses: Data Insights

Supporting Academic Success with Data from Hope s Incoming Student Survey Frost Center Friday February 25, 2022 Kathy Kremer, Ph.D., Sr. Director of Assessment and Accreditation Eric Jones, Ph.D., Data and Research Coordinator Carly Mursch, Research Assistant hope.edu/frostcenter hope.edu/data

Todays Agenda Overview of the UCLA HERI CIRP Survey* Patterns in incoming Hope students responses from 2014 and 2020 Relationship between Hope responses and student success Discussion, conversation, questions * The University of California, Los Angeles, Higher Education Research Institute Cooperative Institutional Research Program The Freshman Survey (TFS)

The UCLA HERI CIRP Survey Provides data on incoming college students background characteristics, high school experiences, attitudes, behaviors, and expectations for college. Administered since 1966 to over 15 million incoming students from 1,900+ institutions Administered to incoming Hope students in 2014 (n=283) and 2020 (n=506)

CIRP Constructs Habits of Mind Academic Self-Concept Individual survey items combined into 8 global measures (constructs) using Item Response Theory Social Self-Concept Pluralistic Orientation Social Agency Civic Engagement College Reputation Orientation Likelihood of College Involvement

A measure of the behaviors and traits associated with academic success and the foundation for lifelong learning. Habits of Mind Low Low Average Average High High 100% 15.4% 16.4% 18.9% 19.0% 22.6% 24.0% 31.6% 32.7% 34.1% 80% 40.1% 40.6% 42.1% Key Findings: -Hope scores are higher than the comparison group in both 2014 & 2020 60% 52.8% 52.7% 51.1% 51.0% 51.0% 52.6% 48.9% 49.8% 48.1% 48.6% 47.8% 47.6% -Hope scores decreased from 2014 to 2020 for all groups 40% 47.5% 45.6% 50.7% 43.7% 50.1% 49.7% 41.5% 43.3% 44.7% -The standard deviation of Hope scores decreased from 2014 to 2020 for all groups 20% 42.1% 44.5% 36.1% 45.4% 35.4% 33.9% 33.7% 31.0% 26.3% 24.4% 24.0% 23.7% 15.8% 14.8% 14.5% 0% Overall (8.24) Men (8.71) Women (8.09) Overall (7.61) Men (7.18) Women (7.79) Overall (8.77) Men (9.08) Women (8.53) Overall (8.15) Men (8.27) Women (7.99) -Comparison group scores also decreased from 2014 to 2020, but not as much as Hope scores SD 2014 2020 2014 2020 Selective Religious Colleges Selective Religious Colleges Selective Religious Colleges Hope College Hope College Hope College

A measure of students beliefs about their abilities and confidence in academic environments Academic Self Concept Low Low Average Average High High 100% 20.3% 20.6% 22.5% 23.7% 23.8% 27.4% 27.6% 27.8% 30.9% 32.9% 33.6% 80% 40.8% Key Findings: -Hope scores are higher than the comparison group in both 2014 & 2020 60% 52.4% 53.0% 50.7% 50.9% 50.6% 51.7% 51.2% 49.6% 49.4% 49.3% 48.5% 48.5% 40% 47.5% 47.2% 50.0% 49.0% 49.0% 47.5% 47.8% 48.5% -Hope scores had a small decrease from 2014 to 2020 47.3% 49.8% 47.7% 42.1% 20% 32.2% 30.3% 29.4% 27.3% 27.2% 24.9% 24.4% 24.1% 19.8% 19.3% 18.7% 17.1% -The standard deviation for Hope scores increased from 2014 to 2020 0% Overall (8.13) Men (8.52) Women (7.95) Overall (8.55) Men (8.16) Women (8.49) Overall (8.21) Men (8.28) Women (8.04) Overall (8.58) Men (8.56) Women (8.46) SD 2014 2020 2014 2020 Selective Religious Colleges Selective Religious Colleges Selective Religious Colleges Hope College Hope College Hope College -Comparison group scores had no real change from 2014 to 2020

A measure of students beliefs about their abilities and confidence in social situations. Social Self-Concept Low Low Average Average High High 100% Key Findings: -Hope mean scores overall and for women are nearly the same for 2014 & 2020 while slightly decreasing for men 17.7% 17.8% 18.1% 21.8% 21.9% 24.8% 25.1% 27.9% 29.0% 29.3% 33.8% 39.5% 80% 60% 52.2% 51.6% 50.7% 50.5% 49.9% 49.4% 49.8% 48.9% 48.9% 48.6% 48.6% 47.1% 41.4% - The standard deviation for Hope scores decreased from 2014 to 2020 40% 45.9% 46.9% 40.5% 42.0% 45.5% 41.7% 48.9% 44.6% 39.2% 50.3% 40.8% 40.5% 20% 36.2% 35.4% 34.8% 32.9% 32.7% 29.3% 29.2% -Hope mean scores are higher than the comparison group in both 2014 and 2020 27.5% 27.1% 20.4% 19.7% 0% Overall (8.46) Men (8.44) Women (8.31) Overall (8.11) Men (8.14) Women (7.99) Overall (9.20) Men (9.34) Women (8.78) Overall (8.34) Men (8.68) Women (8.02) SD 2014 2020 2014 2020 -The standard deviation for Hope scores is lower in both 2014 and 2020 than the comparison group Selective Religious Colleges Selective Religious Colleges Selective Religious Colleges Hope College Hope College Hope College

Measures skills and dispositions appropriate for living and working in a diverse society. Pluralistic Orientation Low Low Average Average High High 100% 17.9% 19.6% 20.3% 21.1% 22.0% 22.3% 22.7% 23.3% 26.0% 27.0% 28.1% 29.7% 80% Key Findings: -Hope mean scores decreased slightly from 2014 to 2020 with a decrease in standard deviation 60% 49.5% 48.9% 48.7% 48.7% 49.7% 48.8% 48.6% 49.8% 49.6% 48.6% 48.7% 48.7% 40% -Comparison group mean scores are higher than Hope s in 2014 but nearly the same in 2020 45.6% 45.3% 46.4% 47.6% 44.2% 49.0% 41.5% 43.4% 52.1% 51.3% 44.7% 54.9% 20% 32.0% 31.4% 30.9% 30.4% 29.9% 29.9% 28.8% 28.5% 28.3% 28.3% 28.3% 27.2% -The standard deviation of scores for Hope and the comparison group decreased from 2014 to 2020, but more so for Hope 0% Overall (8.62) Men (8.54) Women (8.66) Overall (7.53) Men (7.39) Women (7.63) Overall (8.69) Men (9.06) Women (8.40) Overall (8.00) Men (8.00) Women (7.95) SD 2014 2020 2014 2020 Selective Religious Colleges Selective Religious Colleges Selective Religious Colleges Hope College Hope College Hope College

Measures the extent to which students value political and social involvements as a personal goal. Social Agency Low Low Average Average High High 100% 20.0% 25.6% 26.0% 27.1% 28.0% 28.5% 30.5% 31.5% 35.4% 37.8% 80% 40.9% 42.1% Key Findings: -Hope mean scores were higher than the comparison group in 2014, but lower in 2020 60% 52.6% 52.5% 53.5% 51.6% 50.8% 50.3% 50.2% 49.9% 49.5% 50.9% 48.5% 49.3% 40% 43.8% 40.1% 44.8% 43.8% 45.5% 48.1% 54.3% 46.4% 43.9% 44.5% -Although Hope mean scores increased form 2014 to 2020 they did not increase as much as the comparison group 41.6% 20% 43.9% 30.2% 28.4% 28.1% 27.7% 26.5% 26.3% 25.7% 23.1% 20.7% 17.7% 17.5% 14.0% 0% Overall (8.91) Men (8.25) Women (9.07) Overall (8.15) Men (8.18) Women (7.86) Overall (8.88) Men (9.15) Women (8.64) Overall (8.33) Men (8.35) Women (8.13) SD 2014 2020 2014 2020 -In this construct, scores for women are higher in both 2014 and 2020 Selective Religious Colleges Selective Religious Colleges Selective Religious Colleges Hope College Hope College Hope College

Measures the extent to which students are motivated and involved in civic, electoral, and political activities. Civic Engagement Low Low Average Average High High Key Findings: -Hope mean scores increased from 2014 to 2020 at about double the increases of the comparison group 100% 10.5% 19.9% 23.3% 23.9% 26.4% 28.4% 30.5% 32.2% 80% 40.5% 42.3% 46.2% 48.8% 60% 54.3% 53.2% 54.8% 51.0% 53.8% 50.4% 51.6% 49.7% 51.8% 50.4% -Both Hope and the comparison group standard deviation for scores increased from 2014 to 2020, with larger increases for Hope 49.8% 48.5% 40% 46.8% 48.4% 65.8% 57.4% 45.3% 54.4% 49.6% 51.4% 41.4% 20% 42.9% 39.0% 37.9% 29.4% 25.2% 23.7% 22.7% 22.5% 22.3% 22.1% 18.1% 18.0% 14.8% 14.7% 13.3% -Like Social Agency, scores for women are higher than those for men and among Hope women there is a higher standard deviation 0% Overall (6.89) Men (5.69) Women (7.24) Overall (8.65) Men (7.85) Women (8.90) Overall (7.83) Men (7.99) Women (7.66) Overall (8.70) Men (8.34) Women (8.80) SD 2014 2020 2014 2020 Selective Religious Colleges Selective Religious Colleges Selective Religious Colleges Hope College Hope College Hope College

College Reputation Orientation Measures the degree to which students value academic reputation and future career potential as a reason for choosing this college. Low Low Average Average High High 100% 26.0% 27.8% 28.5% 29.4% 30.5% 30.7% 36.6% 37.3% 37.7% 38.1% 38.1% 80% 41.6% 60% 52.3% 52.1% 50.8% 50.0% 50.4% 51.7% 49.4% 50.8% 50.5% 50.6% 49.1% 48.7% 40% 41.7% 45.6% 42.4% 44.5% 43.5% 38.6% 38.6% Key Findings: -Hope mean scores and comparison group scores all increased from 2014 to 2020 40.3% 37.5% 42.9% 50.6% 45.1% 20% 30.5% 28.4% 28.2% 26.9% 25.8% 23.3% 23.3% 22.0% 21.0% 19.9% 18.9% 18.3% 0% Overall (6.76) Men (6.98) Women (6.71) Overall (7.61) Men (6.87) Women (8.13) Overall (7.07) Men (7.19) Women (6.97) Overall (7.91) Men (7.68) Women (8.00) SD -For both years, all Hope scores are above the comparison group 2014 2020 2014 2020 Selective Religious Colleges Selective Religious Colleges Selective Religious Colleges Hope College Hope College Hope College

Likelihood of College Involvement A measure of students expectations about their involvement in college life generally. Low Low Average Average High High 100% 18.8% 21.2% 21.5% 24.4% 33.7% 34.9% 80% 41.2% 42.9% 44.0% 47.2% 49.5% 57.1% Key Findings: -Hope mean scores decreased from 2014 to 2020, while the comparison group scores increased 60% 55.2% 53.8% 52.7% 53.7% 52.0% 52.4% 50.8% 50.5% 49.8% 48.8% 48.1% 47.9% 40% 44.4% 44.2% 45.0% 53.6% 42.4% 41.9% -COVID impact would have been equally present for both Hope and the comparison group 41.0% 20% 41.1% 40.8% 34.4% 34.3% 30.6% 39.2% 42.3% 27.5% 23.9% 23.2% 17.8% 38.3% 16.1% 15.2% 11.3% 10.6% 4.6% 0% Overall (6.67) Men (7.08) Women (5.93) Overall (7.35) Men (7.97) Women (6.44) Overall (8.16) Men (8.23) Women (7.57) Overall (7.76) Men (7.81) Women (7.16) SD -In this construct, Hope women have a higher mean score than Hope men. 2014 2020 2014 2020 Selective Religious Colleges Selective Religious Colleges Selective Religious Colleges Hope College Hope College Hope College

Predicting Student Success Do the CIRP constructs predict short- and long-term student success outcomes? Short Short- -Term Term Long Long- -Term Term First-Semester GPA Current GPA (3 semesters) Year 1 to Year 2 Retention

Predicting Student Success Sample Students entering Hope in August 2020; Completed or partially completed CIRP Grades and retention pulled from institutional data Number of students in analyses ranges from 459 to 487 How big or meaningful is the relationship? R R2 2 r r Small .10 .01 Medium .30 .09 Large .50 .25 R2 indicates what percentage of variability in one variable (e.g., GPA) can be accounted for by another (academic self-concept). Cohen (1988, 1992)

Predicting Student Success: Short-Term Which CIRP constructs predict first first- -semester GPA semester GPA? R R2 2 r r Medium Medium Academic Self-Concept .36*** .13 Likelihood of College Involvement Small Small .24*** .06 ***p<.001 Key Findings: Incoming Hope students with higher scores on Academic Self-Concept earn higher first-semester GPAs Incoming Hope students with higher scores on Likelihood of College Involvement earn higher first-semester GPAs

Predicting Student Success: Short-Term Does Academic Self-Concept predict first beyond high school GPA? first- -semester GPA semester GPA above and YES! YES! Step 1 (Outcome: First-Semester GPA) Step 2 (Outcome: First-Semester GPA) Predictor Predictor B B t t Predictor Predictor B B t t High School GPA .551 14.13*** High School GPA .491 10.77*** Academic Self Academic Self- -Concept Concept .114 2.49* ***p<.001. *p<.05. ***p<.001. R2 for model = .304*** R2 for model = .313*** R2 change = .009*

Predicting Student Success: Short-Term Does Likelihood of College Involvement predict first above and beyond high school GPA? first- -semester GPA semester GPA YES! YES! Step 1 (Outcome: First-Semester GPA) Step 2 (Outcome: First-Semester GPA) Predictor Predictor B B t t Predictor Predictor B B t t High School GPA .570 14.69*** High School GPA .544 13.53*** Likelihood of College Likelihood of College Involvement Involvement .090 2.24* ***p<.001. *p<.05. ***p<.001. R2 for model = .325*** R2 for model = .332*** R2 change = .007*

Predicting Student Success: Long-Term Which CIRP constructs predict current GPA (3 semesters) current GPA (3 semesters)? R R2 2 r r NEW NEW Small Small Habits of Mind .13** .02 Medium Medium Academic Self-Concept .41*** .17 NEW NEW Small Small Social Agency .11* .01 Likelihood of College Involvement .27*** .07 Small Small- -Medium Medium *p<.05. **p<.01. ***p<.001 Key Finding: Incoming Hope students with higher scores on these CIRP constructs have higher GPAs 3 semesters later

Predicting Student Success: Long-Term Does Habits of Mind predict current GPA school GPA? NO! NO! current GPA above and beyond high Step 1 (Outcome: Current GPA) Step 2 (Outcome: Current GPA) Predictor Predictor B B t t Predictor Predictor B B t t High School GPA .586 15.63*** High School GPA .581 15.23*** Habits of Mind Habits of Mind .030 .77 ***p<.001. ***p<.001. R2 for model = .344*** R2 for model = .345*** R2 change = .001

Predicting Student Success: Long-Term Does Academic Self-Concept predict current GPA high school GPA? current GPA above and beyond YES! YES! Step 1 (Outcome: Current GPA) Step 2 (Outcome: Current GPA) Predictor Predictor B B t t Predictor Predictor B B t t High School GPA .588 15.56*** High School GPA .505 11.50*** Academic Self Academic Self- -Concept Concept .159 3.61*** ***p<.001. ***p<.001. R2 for model = .346*** R2 for model = .364*** R2 change = .018***

Predicting Student Success: Long-Term Does Social Agency predict current GPA school GPA? current GPA above and beyond high YES! YES! Step 1 (Outcome: Current GPA) Step 2 (Outcome: Current GPA) Predictor Predictor B B t t Predictor Predictor B B t t High School GPA .597 15.78*** High School GPA .591 15.60*** Social Agency Social Agency .078 2.05* ***p<.001. *p<.05. ***p<.001. R2 for model = .357*** R2 for model = .363*** R2 change = .006*

Predicting Student Success: Long-Term Does Likelihood of College Involvement predict current GPA and beyond high school GPA? current GPA above YES! YES! Step 1 (Outcome: Current GPA) Step 2 (Outcome: Current GPA) Predictor Predictor B B t t Predictor Predictor B B t t High School GPA .597 15.78*** High School GPA .563 14.42*** Likelihood of College Likelihood of College Involvement Involvement .122 3.12** ***p<.001. **p<.01. ***p<.001. R2 for model = .357*** R2 for model = .371*** R2 change = .014**

Predicting Student Success: Long-Term Do students who stay at Hope from Year 1 to Year 2 score differently on any of the CIRP constructs compared to those who leave? Academic Self Academic Self- -Concept* Concept* Social Agency** Social Agency** Retained Retained Not Retained Not Retained Retained Retained Not Retained Not Retained 51.10 47.31 51.85 47.48 n = 435 n = 35 n = 425 n = 34 (Medium) (Medium) *t(468) = 2.51, p<.05, d = .44 (Small (Small- -Medium) Medium) **t(457) = 3.03, p<.01, d = .54 Key Finding: Students who stay at Hope after their first year have higher scores on Academic Self-Concept and Social Agency when they enter Hope

Predicting Student Success: Long-Term Do students who stay at Hope from Year 1 to Year 2 score differently on any of the CIRP constructs compared to those who leave? College Reputation Orientation** College Reputation Orientation** Likelihood of College Involvement* Likelihood of College Involvement* Retained Retained Not Retained Not Retained Retained Retained Not Retained Not Retained 52.20 48.07 52.31 49.44 n = 429 n = 35 n = 425 n = 34 (Small (Small- -Medium) Medium) **t(462) = 3.07, p<.01, d = .54 (Medium) (Medium) *t(457) = 2.22, p<.05, d = .40 Key Finding: Students who stay at Hope after their first year have higher scores on College Reputation Orientation and Likelihood of College Involvement when they enter Hope

Predicting Student Success: Long-Term Do students who stay at Hope from Year 1 to Year 2 score differently on any of the CIRP constructs compared to those who leave? Civic Engagement* Civic Engagement* Retained Retained Not Retained Not Retained 53.88 50.40 n = 451 n = 36 *t(485) = 2.35, p<.05, d = .41 (Small (Small- -Medium) Medium) Key Finding: Students who stay at Hope after their first year have higher scores on Civic Engagement when they enter Hope

Predicting Student Success: Summary Predicted 1 Predicted 1st st- - Sem. GPA Sem. GPA Predicted 1 Predicted 1st st- - Sem. GPA above Sem. GPA above HS GPA HS GPA Predicted Predicted Current GPA Current GPA Predicted Predicted Current GPA Current GPA above HS GPA above HS GPA Predicted Year 1 Predicted Year 1 to Year 2 to Year 2 Retention Retention Habits of Mind Academic Self-Concept Social Self-Concept Pluralistic Orientation Social Agency College Reputation Orientation Likelihood of College Involvement Civic Engagement = Statistically Significant Predictor

Key observations Hope consistently scores above our comparison group in most CIRP constructs. Overall scores for Hope students increased from 2014 to 2020 in the constructs of Social Agency, Civic Engagement, and College Reputation Orientation. Hope s largest score increase from 2014 to 2020 is in Civic Engagement with the greatest decrease in mean in the Habits of Mind construct. Students completing the CIRP Survey in 2020 were entering college during a period of converging crises and change, including the pandemic, that may or may not have influenced observed increases in the Civic Engagement construct or other construct changes.

Key observations Construct scores that are consistently predictors of Hope student success are Academic Self-Concept and Likelihood of College Involvement, however, the Hope mean scores for both of these constructs decreased from 2014 to 2020. The two constructs with the greatest difference in scores for Hope men and women are also these same two predictors of student success: Academic Self-Concept (men 52.4, women 49.6) and Likelihood of College Involvement (men 48.8 and women 53.7). While the naturally-occurring relationships between the CIRP constructs and the short- and long-term student success measures tended to be small, interventions focused on strengthening construct effect on student success could be a starting point to consider.

Discussion, conversation, questions Who at Hope College (positions or departments) would benefit from the information shared today? Thinking about your own work at Hope, how might you use what you ve learned today? How might data from this survey continuously inform decisions to benefit Hope students?