Insights into Disability Program Evaluation and Review Practices

Program Review & Evaluation Practices of Disability Resources Jean Ashmore, AHEAD Representative to CAS Ann Knettler, AHEAD Rep to CAS, Delaware State University Sally Scott, AHEAD Sr. Research Associate Katy Washington, AHEAD Secretary, University of North Texas AHEAD Conference Boston 2019

AHEAD Survey on Program Review Practices Developed to learn about use of (or not) program review and evaluation in disability resources To inform AHEAD about same, plus needs for member professional development Completed in Spring 2019 Total respondents = 312

Demographics 71% from 4 year colleges (private & public) 47% from schools with 1,500-9,999 enrolled Students registered with DR 49% 100-500 Over 50% of DROs were staffed by 1-2 67% of respondents were Directors; average degree was Master s 50% of all had 11-15 or more years DR exp

Has the DRO conducted a Program Evaluation or Program Review within the past five years? Yes 43% No 57% Yes 43% No 57%

Top 3 Factors why PR not conducted 60.0% Lack of personnel, time and resources to conduct 58.2 % Not required or expected by administration at institutional, system or district level 37.0% Limited experience with program review

Additional factors for not conducting PR 26.1% Staff turnover has precluded a review 20.6% DR program information and data are incorporated in a divisional report so a DR specific review is not needed 7.9% Program review is done during an accreditation cycle (usually this is more than five years) 13.9 % Other - Write In Many cite plans to do review in near future

PR particulars - respondents who completed PR within last five years About half conducted PR at regular intervals Time to complete PR 66% - 2-5 months PR somewhat achieved purposes - 63% How the PR was completed 39% self study by DR staff 28% by DR with institutional colleagues 22% external expert guided

What purposes does a program review serve for the DRO? 83.5 % Is an internal tool for DR planning and to enhance practice 65.1% To evaluate achievement of established DRO outcome goals 63.3% A basis for funding appeal for equipment, staff, space, etc. 59.6% Fosters transparency with administration 48.6% To fulfill administrative expectations 46.8% Formalizes data collection into a reportable format

Use Professional Standards for PR No 33% Yes 67% No 33% Yes 67%

Factors influencing non use of standards for PR Familiarity with disability standards is limited 22.9% The institution has its own set of review criteria 40.0% No current standards reflect our DRO practices 14.3% Standards are not needed when doing a program review 11.4%

Which standards used for PR Other (Please Specify) 7% AHEAD 22% CAS 34% AHEAD Professional Program Standards 22% Combo 37% A combination of AHEAD and CAS standards 37% Other 7% CAS Disability Resources & Services Standards and Guidelines 34%

Standards Sections Most Cited AHEAD Professional Standards Policies and Procedures 80% CAS Disability Resources Svcs Program 64% Faculty/Staff Awareness 60% Mission 55% Diversity, Equity and Access 55% Program Administration and Evaluation 40%

CAS specific questions Was the Self-Assessment Guide (SAG) used? No 57% Yes 43% Was assessment of student learning outcomes included in PR Yes 54% No 46% If used was the SAG useful? Most rated it as very useful

Benefits to standards used AHEAD Professional Standards They assure the review considers all aspects of the program 71% They align campus practices with our profession 71% They provide an objective measure for the program 64% They are accepted/endorsed by our administration 43% CAS Disability Resources Svcs They provide an objective measure for the program 77% They align campus practices with our profession 55% They assure the review considers all aspects of the program 50% They are recognized by our accrediting body 46%

Ways AHEAD and CAS standards used together for PR 37% We used the CAS SAG as our primary set of standards, but also included the AHEAD standards as well. Reviewed both, highlighted areas of weakness for concentration during eval. AHEAD biennial survey was used as well as basic premises from CAS. Information from both sources were used to develop measurements and to develop questionnaires for student feedback.

Is a PR planned in the future? No 43% Yes. In the upcoming academic year 18% Yes 1 yr 18% No 43% Yes 3 yrs 30% Yes. In the next three years 30% Yes accred 9% Yes. Before the next accreditation cycle 9%

Using standards in the future Will you use standards for a PR? 93% Yes 7% No Which standards will you most likely use? 51% AHEAD 22% CAS 27% Don t know

Experience in DR related to PR practices More experience in DR appears to be related to having conducted a recent PR whereas less experience appears be related to having not conducted a PR More experience appears to relate to use of professional standards for a PR More experience has some relation with use of a combination of AHEAD and CAS standards when doing a PR CAS standards used by those with fewer years of experience

Respondents Rate PR uses Data from PR has enhanced DR practices 75% agree or strongly agree Data has been used for accreditation purposes 24% disagree, 21% strongly agree PR has helped communicate practices & needs 67% agree or strongly agree Data helped to advocate for resources & staff 35% strongly agree

Survey Highlights Program review was not completed in last five years by over 50% Standards were used for PR, highest use was a combination of AHEAD and CAS Majority of respondents do plan to complete a PR in the future and will use standards AHEAD standards are cited as those most likely to be used in the future

Useful resources from AHEAD on Program Review Program review resources/tools 82% Training on use of standards 72% Training on assessment and evaluation 67% Webinars 66%

AHEAD Survey citation Should you wish to cite findings from the 2019 AHEAD member survey on practices in program review, please feel free to do so using the following attribution/citation Ashmore, J., Knettler, A., Scott, S., & Washington, K. (2019). Survey ofProgram Evaluation and Review Practices by Disability Resource Professionals. Huntersville, NC. The Association on Higher Education and Disability

Contact Information Jean Ashmore jean@ahead.org Ann Knettler aksmith@desu.edu Sally Scott sally@ahead.org Katy Washington katy.washington@unt.edu