Program Review: Utilizing Assessment for Data-Driven Decisions

Make Your Data Talk: Program Review Through the Lens of Assessment Ann S. Knettler, Ed.D. Jill Sieben-Schneider, Ed.D.

We ask you to join us in creating a culture that reflects Access and Inclusion and Civility and Respect this week and in all aspects of our organization.

Agenda Utilizing Professional Standards Overview of AHEAD Standards Overview of CAS Standards (Council for the Advancement of Standards in Higher Education) Why Conduct a Program Review? Benefits of a program review Process for conducting a review Data Sources: Understanding, Creating, and Assessing Student Learning and Program Outcomes

AHEAD & CAS Standards

AHEAD Domains, Standards, and Performance AHEAD Domains, Standards, and Performance Indicators Indicators Designed to: Expand the vision of disability resource offices Guide campus administrators understanding of the breadth disability resource work Direct the development and evaluation of disability resource offices Improve the preparation of professional personnel Updated in 2021 Committee of AHEAD members from various institution types Previous versions developed and revised in 1997 & 2006

AHEAD Domains, Standards, and Performance Indicators (2) AHEAD Domains, Standards, and Performance Indicators (2) 5 Domains: 5 Domains: Domain 1: Leadership & Collaboration Domain 2: Consultation & Information Dissemination Domain 3: Access & Equity Domain 4: Office Administration and Operations Domain 5: Professional Development Available on the AHEAD Website

AHEAD Domains, Standards, and Performance Indicators AHEAD Domains, Standards, and Performance Indicators - - Example Example Domain 4 Domain 4 Office Administration and Operations Domain 4 Standard Domain 4 Standard Disability resource professionals are guided by a program mission, have appropriate resources to fulfill the mission, operate under practices consistent with the mission, and establish an ongoing process of assessment in achieving the mission. Domain 4 Performance Indicators Domain 4 Performance Indicators 4.4: Design and implement a rigorous program of ongoing assessment to improve service delivery and demonstrate institutional impact. 4.4.1: Establish a written assessment plan that aligns with the program mission statement and philosophy and includes qualitative and quantitative data and measurable goals; review and revise the plan as needed.

Founded in 1979 Council for the Advancement of Standards in Higher Education Consortium of 40+ member organizations (CAS1) CAS Board of Directors comprised of representatives from member associations

47 sets of functional area standards and self- assessment guides (SAGs) Council for the Advancement of Standards in Higher Education Designed to be achievable by any program or service, at any institution type Standards = Threshold (MUST) vs. Guidelines = Good practice (SHOULD) (CAS2) Available for free on the AHEAD Website See CAS instructions pdf/site, choose DRS Functional Area Standards, use code LP-AHEA at check-out.

CAS Principles, Parts, and Standards Principle 1: Students and Their Environments Part 1: Mission; Part 2: Programs and Services; Part 3: Student Learning, Development, and Success, and Part 4: Assessment Principle 2: Advocating for Diverse, Equitable, and Inclusive Communities Part 5: Access, Equity, Diversity, and Inclusion

CAS Principles, Parts, and Standards (2) Principle 3: Organization, Leadership, and Human Resources Part 6: Leadership, Management, and Supervision; Part 7: Human Resources; and Part 8: Collaboration and Communication Principle 4: Ethical Considerations Part 9: Ethics, Law, and Policy Principle 5: Learning-Conducive Structures, Resources, and Systems Part 10: Financial Resources; Part 11: Technology; and Part 12: Facilities and Infrastructure

Revised and published in 2022-2023 by CAS AHEAD representatives and AHEAD members Disability Resources & Services (DRS) Standards Continuity with revised AHEAD Standards DRS Functional Area Standards Available for Free Login to AHEAD Site Assessment Resources Page Follow link to PDF & CAS Store Create account with/CAS or login Use code LP-AHEA at check-out

DRS STANDARD 1.1, CRITERION MEASURES: DRS STANDARD 1.1, CRITERION MEASURES: The primary mission of Disability Resources and Services (DRS) is: 1. Provide leadership and facilitate equal access to all institutional opportunities for disabled students 2. Provide institution-wide advisement, consultation, and training on disability-related topics, including legal and regulatory compliance, universal design, and disability scholarship 3. Collaborate with partners to identify and remove barriers to foster an all-inclusive campus 4. Provide individual services and facilitate accommodations to students with disabilities Using CAS Standards to Develop and Evaluate DRS Principle 1: Students and Their Environments Part 1: Mission Standard 1.1: Programs and Services Mission

Education University (EdU) is committed to providing equal access to all students in all programs. Students with disabilities who wish to request reasonable accommodations must register with the AccessAbility Services office by contacting the executive director of student academic support services. Students should register with AccessAbility Services as soon as possible after acceptance to Ed.U. Mission Statements Is Anything Missing? Teach Me College (TMC) is committed to providing students with disabilities the resources and support necessary to assist in their academic success by engaging in an interactive process with each student. By understanding each student's unique learning profile, adjustments can be individualized.

The mission of the Center for Disability Resources (CDR) at Delaware State University is to provide equal opportunities and equal access to education, employment, programs, goods and services for students, faculty, staff, and campus guests with disabilities. In achieving this, the CDR collaborates with and empowers individuals who have documented disabilities by working together proactively to determine reasonable and appropriate accommodative measures under the Americans with Disabilities Act Amendments Act (ADAAA) and other current legislation. Delaware State University Mission: As a community partner, the CDR also provides advisement, consultation, professional development, and training on disability- related topics, including but not limited to, legal and regulatory compliance, access and equity, the proactive application of universal design, physical and electronic accessibility, the high school to college and college to career transition, and disability scholarship. Training, workshops, and resources are available to the University and community. Tag Line: The CDR proudly works with University and Community Partners to foster a welcoming, diverse, engaging, respectful, and inclusive community for all. The CDR proudly works with University and Community Partners to foster a welcoming, diverse, engaging, respectful, and inclusive community for all.

DRS STANDARD 2.2, CRITERION MEASURES: DRS STANDARD 2.2, CRITERION MEASURES: Using CAS Standards to Develop and Evaluate DRS DRS must provide or include Relevant information, services, and resources consistent with its mission and goals. Appropriate and relevant office policies, procedures, and practices Individual consultation, accommodation, and service Proactive dissemination of information Institution-wide education, consultation, and advocacy Guidance and technical assistance to the institution on disability-related laws and regulations Principle 1: Students and Their Environments Part 2: Programs and Services Standard 2.2: Programs and Services Mission

DRS STANDARDS 3 & 4, CRITERION MEASURES: DRS STANDARDS 3 & 4, CRITERION MEASURES: Develop and Evaluate DRS DRS must 3.1: Identify relevant and desirable student learning and development outcomes 3.3: Assess the student learning and development outcome domains and dimensions 3.3: Provide evidence of the extent to which student learning and development outcomes are achieved. 4.1: Develop assessment plans and processes that document progress toward achievement of mission, goals, outcomes, and objectives. 4.2: Identify assessment methods that will allow for the collection of relevant data on its goals, outcomes, and objectives. Principle 1: Students and Their Environments Part 3: Student Learning, Development, and Success Part 4: Assessment

Program Review a formal internal or external evaluation process that assesses programs and identifies necessary improvements

Catalyst for individual, group and organizational learning Reflect the strategic information needs of the organization Why Conduct a Program Review? Designated to inform decision-making and potential actions Accountability, acquisition of knowledge and management and/or development for program improvement Institutions should improve student success by focusing on practices within their control

Design new programs and services Focus time, energy, and resources Outcomes Outcomes for AHEAD & CAS AHEAD & CAS Standards Standards for Devise staff development Guide strategic planning Develop learning and development outcomes Measure program and service effectiveness

Program Review for Ongoing Planning and Assessment Cycle If an action plan already is in place, then standards become part of that process Program Review Using Standards Prep for Strategic Plan/ Program Review Strategic Plan/Program Action Plan Maintaining timely, consistent and accurate data is crucial to having good results Data Gathering Interim Reviews Action Plan Enabled

Evaluating the Program Using the CAS Self-Assessment Guide Part 5: ETHICS Suggested Evidence and Documentation: 1. Program code or statement of ethics 2. Ethics statements from relevant functional area professional associations 3. Personnel policies, procedures and/or handbook 4. Student code of conduct 5. Operating policies and procedures related to human subjects research (Institutional Review Board, IRB) 6. Minutes from meetings during which staff reviewed and discussed ethics Criterion Measures: DNA IE 0 1 2 Rating: Does Not Apply Insufficient Evidence/ Unable to Rate Does Not Meet Partly Meets Meets 5.1 Ethical Standards Disability Resources and Services (DRS) reviews applicable professional ethical standards and adopts or develops and implements appropriate statements of ethical practice. DRS publishes and adheres to statements of ethical practice, ensures their periodic review, and orients new personnel to relevant statements of ethical practice and related institutional policies. DRS personnel who are licensed or certified in other professions recognize and apply the professional ethical standards appropriate to their role and function at the institution.

CAS Methodology (1) Plan the process: map out steps for process, develop timeline, build by in with all stakeholders, and explicitly identify desired outcomes of the self study. Assemble and educate the self assessment team: determine who should be on the team and how to educate the team about the self study process. Identify, collect, and review evidence: define what constitutes evidence; then gather, collect, manage, and review evidence. Conduct and interpret ratings using a evaluative evidence: clarify teams, rating criteria; employee process for rating (small group, individual, staff); negotiate rating differences; and manage group ratings.

CAS Methodology (2) Develop an action plan: identify discrepancies, corrective action, and recommend steps (E. G., Identify strengths, weaknesses, recommendations, benchmarks, for achievement, resources, timeframe, and responsible individuals). Prepare a report: identify audience for report (S); describe the self study process, evidence gathering, rating process, and evaluations; summarize strengths and weaknesses; describe the action plan; and draft and executive summary. Close the loop: put action plans into practice; work to negotiate politics and secure resources; identify barriers to overcome; and build by in to the program review results.

Student and Program Outcomes Creating and assessing outcomes that align with CAS and AHEAD Standards

Student Learning Outcomes (SLOs) Learning outcomes are statements describing the changes in behavior or performance that are the desired outcome of the learning interaction between the students and the facilitator of the learning process. facilitator of the learning process. Learning outcomes are statements describing the changes in behavior or performance that are the desired outcome of the learning interaction between the students and the As a result of interaction with our office and services, SLOs articulate what students are expected to: Value and appreciate (affective) Know (cognitive) Do (behavioral), and

Program vs Student Outcomes Evaluate program outcomes that assess DS mission effectiveness to determine how well DS is: Providing leadership and facilitating access Providing advisement, consultation & training Collaborating to remove barriers Providing individual services & accommodations

PO to SLO Example Program Outcome: Educate students about appropriate application of accommodations. Where/How outcome will be met: Initial meeting with DS Professional DS Website Emails/reach out from DS When/By when should this be achieved? End of student s 1st semester Related Student Learning Outcome: Based on their experience with DS, students with alt format accommodations will understand the process for requesting alternate format materials.

What should be the result of our work? Program outcomes (e.g., percentage of students reached) Operational outcomes (e.g., reduced wait-list time to appointment) Student learning and development outcomes (e.g., intrapersonal development/integrity/ethical decision-making) Integrating SLOs with Program Outcomes Important to balance focus on student learning outcomes with focus on program outcomes Are our programs and services organized and run effectively to achieve the intended outcomes? Are the intended outcomes achieved?

Avoid compound SLOs e.g. Students understand and value the purpose of our services Tips for Developing SLOs and POs As you are developing each SLO, think about: When and where the opportunity for learning may occur for that desired SLO How you might measure whether or not the SLO has been achieved

Specific Measurable SMART SMART Learning and Program Outcomes Aggressive, but achievable Results-oriented Time-bound

SLO Example

Key Takeaways Professional Standards can be used to assess program efficiency and current standing. Using assessment (e.g. program review, outcomes assessment, etc): Provides normative data that other departments can relate to Informs professional development Identifies areas for growth and areas of current success Provides advocacy data Supports student learning outcomes Benefits the DRS office as a whole

Contact Information Ann Knettler, Ed.D. AHEAD Representative to CAS Vice President, Consulting GrackleDocs ADA & 504 Compliance Officer Office of General Counsel Delaware State University Jill Sieben-Schneider, Ed.D. AHEAD, Representative to CAS AHEAD, Treasurer, Board of Directors Associate Dean of Students Dean of Students Office | AccessibleNU Northwestern University Email: Email: jill.schneider@northwestern.edu Email: Email: aknettler@grackledocs.com

Helimsky, E. (2013). Evaluation purposes, perspectives, and practice. In Alkin, M. C. (Ed.), Evaluation roots: A wider perspective of theorists views and influences (p. 267- 282). Thousand Oaks, CA: SAGE Publications, Inc. Preskill, H. (2013). The transformational power of evaluation: Passion, purpose, and practice. In Alkin, M. C. (Ed.), Evaluation roots: A wider perspective of theorists views and influences (p. 323-333). Thousand Oaks, CA: SAGE Publications, Inc. References Sharp, M. D. (2017). Overview of the CAS professional standards. In Gulley, N. Y., Dean, S. R. & Dean, L. A. (Eds.),Using the CAS professional standards (p. 1- 16). Washington, DC: NASPA.

For More Information CAS Visit the CAS website CAS Professional Standards for Higher Education, DRS Functional Area Standards, and DRS Self- Assessment Guide available through CAS store and Anthology AHEAD Visit the AHEAD Assessment Resources and Services page login as member Visit the AHEAD Program and Professional Standards page login as member

LinkedIn LinkedIn https://www.linkedin.com/company/cas-standards/ Facebook Facebook https://www.facebook.com/CAS.Standards/ Connect with CAS Twitter Twitter https://twitter.com/CAS_Standards Instagram Instagram https://www.instagram.com/cas_standards/ YouTube YouTube https://www.youtube.com/user/CASstandards