Key Areas of Student Competency in Undergraduate Education

February 6, 2016 Loyola Core-Common Curriculum Assessment Results

General Education Overview Assessment Package Results* Using the Data Spring Testing* Agenda

General Education The general education component of each undergraduate degree is designed to foster student competency in five key areas: Critical Thinking Communication Quantitative Reasoning Information Literacy Ethical Reasoning

What do we want our students to learn? Core Competencies Student Learning Outcomes 1. Students will demonstrate the ability to think critically in a variety of contexts. A. Evaluate evidence. B. Support a thesis with evidence. C. Identify faulty logic in an argument/text 2. Students will demonstrate the ability to communicate effectively in a variety of contexts. A. Formulate a position. B. Organize information and defend a position. C. Demonstrate a clear grasp of grammar and mechanics 3. Students will demonstrate the ability to use quantitative reasoning in a variety of contexts. A. Perform basic computational operations. B. Interpret summaries of data. C. Demonstrate problem-solving capability 4. Students will demonstrate the ability to use information literacy skills in a variety of contexts. A. Find information relative to a topic. B. Identify relevant sources of information relative to a topic. C. Use information according to standards of academic integrity 5. Students will demonstrate the ability to use ethical reasoning in a variety of contexts and to reflect critically on issues of social justice. A. Identify ethical perspectives and concepts. B. Apply ethical perspectives or concepts to an ethical question. C. Recognize basic ethical issues

How do we ensure that our students achieve these outcomes? Spiritual Intellectual The Loyola Core (previously Common Curriculum) Set of core courses that functions as the academic expression of the university s mission 39-41 credit hours Interdisciplinary approach to learning Moral Ethical Artistic Loyola Core Natural Social Scientific Humanistic

How do we determine the extent to which students achieve these outcomes? Instruments: ETS Proficiency Profile Standardized Assessment of Information Literacy Skills Test Defining Issues Test Loyola Core Assessment Package: Body of assessment methods Measure the competencies identified by university Direct measures of student learning Nationally recognized Detailed data regarding student skill sets Comparison with peer institutional performance

Core Competencies Instruments Description 1. Critical Thinking 2. Communication 3. Quantitative Reasoning ETS PP Measures academic skills, rather than the specificsubject knowledge Online, abbreviated version 36 questions 40 minute test session 4. Information Literacy SAILS Measures how well students can navigate information use Identify strengths and weaknesses Directly based on the Information Literacy Competency Standards for Higher Education identified by the ACRL Online version 45 questions 35 minute test session 5. Ethical Reasoning DIT Designed for assessing moral concepts in terms of their importance for moral judgments Used to evaluate how well developed students moral concepts are and how well they can apply them to concrete moral dilemmas Online version 60items 45 minute test session

ETS PP Participants Pre Test 472/588 (80%) of all first-time, full-time students completed the test Testing scheduled during First-Year Seminar courses in fall 2013 Post Test 215 completed the test, 27/92 LC sections in spring 2016 Testing scheduled by directly soliciting instructors, options included Set up college test session, award extra credit in specific classes: COB (54) In class testing: Psych (25), FAMI (111) Add an assignment outside of class (nonproctored): Mass Comm (25)

ETS PP Test Taker Profile Students Reporting Credit Hour Completion (Fall 2015) Pre Test: Fall 2013 95% identified as not a transfer student 88% reported no credit hours successfully completed Post Test: Fall 2015 85% identified as not a transfer student 69 % upperclassman None Entering Freshman Fewer than 30 30-60 Hours 61-90 Hours More than 90

ETS PP Results Students were more proficient in the skill areas of reading, writing and mathematics as freshman, rather than as juniors. Pre-Post Test Proficiency Comparison 80 70 60 50 40 30 20 10 0 Reading 1 Writing 1 Math 1 Freshman Juniors

ETS PP Theories for pre/post-test performance decrease include: Sampling issue Freshman may be eager to cooperate, Juniors and Seniors may lack motivation if no incentive to do well Majors that require little or no math Instructors may not be able to articulate purpose of test or convey importance Proctors may need to more fully explain purpose and how data are used Outlier scores may pull down means, considering removing, or scatterplot

ETS PP Results Freshman Carnegie Class Proficiency Comparison 80 LU freshman were more proficient in the skill areas of reading, writing and mathematics than freshman at other master s level colleges and universities. 70 60 50 40 30 20 10 0 Reading 1 Writing 1 Math 1 Loyola Carnegie Class

ETS PP Results Junior Carnegie Class Proficiency Comparison 70 LU upperclassman were more proficient in the writing skill area than juniors at other master s level colleges and universities. 60 50 40 30 20 10 0 Reading 1 Writing 1 Math 1 Loyola Carnegie Class

SAILS Participants Sampling 92 sections of Loyola Core courses offered (Spring 2016) Each tagged with one core competency for assessment purposes Information literacy if offering 5+ sections, designate 3 History Music Sociology N = 110

N % Test Taker Profile SAILS 24 21.8 Class Standing Freshman 26 23.6 Sophomore 29 26.4 Junior 31 28.2 Senior Business 6 5.5 Major Communications/Journalism 5 4.5 General Studies 1 0.9 Health Sciences 5 4.5 History 7 6.4 Humanities 13 11.8 Law 3 2.7 Performing & Fine Arts 18 16.4 Science/Math 9 8.2 Social Sciences/Psychology 26 23.6 Other 16 14.5 Undecided 1 0.9

Results SAILS LU students performed better than the institution-type benchmark on these skill sets: Developing a Research Strategy Selecting Finding Tools Searching Retrieving Sources Evaluating Sources Documenting Sources Understanding Economic, Legal, and Social Issues LU students performed about the same as the institution-type benchmark on this skill set: Using Finding Tool Features

SAILS Results Documenting Sources Searching Selecting Finding Tools Evaluating Sources Retrieving Sources Using Finding Tool Features Developing a Research Strategy Understanding Economic, Legal and Social Issues Best Performance Worst

SAILS Skill Sets Loyola University Institution Type: Masters All US Institutions Developing a Research Strategy* 530 10 511 1 510 0 Selecting Finding Tools* 543 12 520 2 509 1 Searching* 523 10 493 1 489 0 Using Finding Tool Features 542 15 534 2 528 1 Retrieving Sources* 553 15 526 2 518 1 Evaluating Sources* 515 11 477 2 474 0 Documenting Sources* 540 15 501 2 480 1 Understanding Economic, Legal, and Social Issues* 502 12 475 2 471 1

DIT Participants Sampling 92 sections of Loyola Core courses offered (Spring 2016) Each tagged with one core competency for assessment purposes Ethical reasoning designate 5 course sections Philosophy Religious Studies N = 182 No demographic items = no test taker profile available

DIT Test Description 5 hypothetical moral dilemmas, each followed by 12 related issues Participants are asked what decision to make which issues are the most important to consider when making that decision

DIT Test Description Evaluates moral schemas that participants use when considering ethical dilemmas and making moral decisions Items cluster around 3 general moral schemas: arguments that appeal to personal interest maintaining social laws and norms moral ideals and/or theoretical frameworks for resolving complex moral issues

represent the proportion of items selected that appeal to considerations that focus on the DIT Schema Scores Personal Interest Direct advantages to the actor and on the fairness of simple exchanges of favor for favor Good or evil intentions of the parties, on the party s concern for maintaining friendships and good relationships, and maintaining approval Maintaining Norms Maintaining the existing legal system, maintaining existing roles and formal organizational structure

represent the proportion of items selected that appeal to considerations that focus on the following DIT Schema Scores Post Conventional Organizing a society by appealing to consensus-producing procedures (such as abiding by majority vote), insisting on due process (giving everyone his day in court), and safeguarding minimal basic rights Organizing social arrangements and relationships in terms of intuitively appealing ideals N2 score is an index with two parts: the degree to which Post Conventional items are prioritized Personal Interest items (lower stage items) receive lower ratings than the ratings given to Post Conventional items (higher stage items)

DIT Results Students perform at or above their educational level as compared to schema scores detailed by level of education and drawn from large sample (N=10,553) Personal Interest Maintain Norms Post Conventional N2 score Mean 27.26 28.99 35.47 33.57 Standard Deviation 12.55 13.26 16.97 15.20 N 182 182 182 182

Indirect Measures Student feedback on select items related to the Loyola Core National Survey of Student Engagement (NSSE) Institutional Trends (2013, 2014, 2015) Comparison Groups (Loyola, Jesuit, Carnegie Class, General Reference Group) Freshman and Seniors (Spring 2015) Student Satisfaction Inventory (SSI) Spring 2014 Course Evaluations AY 2014-15 AY 2015-16

So what? Key implications of results for institutional planning purposes How results can inform future quality enhancement initiatives Recommendations for close-the-loop actions

Recommendations Provide faculty LC assessment results Frame general education as connecting/impacting major programs Set targets or benchmarks for expectations of student performance Focus on testing upperclassman who have completed LC Request rapid response report Drill down on GPAs Scores ~ Student ID ~ Persistence Focus actions on select group of students or courses Survey faculty for input regarding how to improve core competencies ETS PP

Why is this important? Identified key areas, core competencies, student learning outcomes Designed curriculum with required courses Advise students how to best select from menu Approximately 1/3 of each undergraduate degree Professional obligation SACSCOC accreditation CR 2.7.3 General education CS 3.5.1 General education competencies CS 3.5.3 Undergraduate program requirements

Instruments ETS Proficiency Profile http://www.ets.org/proficiencyprofile/about Standardized Assessment of Information Literacy Test https://www.projectsails.org Defining Issues Test http://ethicaldevelopment.ua.edu/dit-and-dit-2/

Department Sections Offered Sections Requested Spring Testing ETS Proficiency Profile (Post Test) Standardized Assessment of Information Literacy Skills Set BIOL 4 1 CHEM 2 1 CLHUN 2 1 CMMN 2 1 ENGL 20 6 HIST 15 6 March 13 - April 7 MATH 3 1 MUGN 3 1 Juniors and Seniors only PHIL 25 6 PHYS 2 1 OIRE will work with instructors: Schedule Reserve computer lab Provide instructions Proctor POLI 2 1 RELS 21 6 SCIE 7 3 SOCI 7 3 VISA 2 1 TOTAL 121 51