Academic Portfolio Review Task Force Open Forum for Feedback on APR Rubric

Academic Portfolio Review Task Force Open Forum for Feedback on APR Rubric and Contextual Information

Staff Adam Landreth, Director of Online Programs & Advisor, Bryan School of Business and Economics Valeria Caviness, University Program Specialist, Office of the Provost Melissa Skillings, Assistant Director of Research Integrity, Office of Research and Engagement Karen Blackwell, Director, Institutional Research and Enterprise Data Management Professional Track Faculty Kristen Christman, Lecturer, Communication Studies, College of Arts & Sciences Trisha Kemerly, Lecturer, Department of Consumer, Apparel, and Retail Studies, Bryan School of Business and Economics Brad Johnson, Clinical Professor, Teacher Education and Higher Education, School of Education (also rep. the Graduate Council) Wade Maki, Senior Lecturer and BLS Program Director, College of Arts & Sciences Rachel Mills, Genetic Counseling, School of Health & Human Sciences Kevin Wells, Lecturer, Media Studies, College of Arts & Sciences Tenured/Tenure Track Faculty Sarah Daynes, Professor, Sociology, College of Arts & Sciences (also rep. Faculty Senate) Mark Fine, Professor, Human Development and Family Studies, School of Health & Human Sciences Heather Gert, Associate Professor and Department Head, Philosophy, College of Arts & Sciences Justin Harmon, Associate Professor, Community & Therapeutic Recreation, School of Health & Human Sciences John Lepri, Professor, Biology, College of Arts & Sciences Stephanie Pickett, Associate Professor, Adult Health Nursing, School of Nursing Debra Wallace, Professor, Senior Associate Dean of Research & Innovation, Distinguished Professor, School of Nursing (also rep. the Research Advisory Council) Kelly Wester, Professor and Department Chair, Counseling & Educational Development, School of Education (chair of PRTF) Ken White, Associate Professor and Associate Dean for Academic Affairs, College of Visual and Performing Arts PRTF Members

Todays Open Forum 2 hours long Goal is to provide a brief overview of the academic portfolio review timeline and field questions but more so take suggestions and comments PRTF Online Feedback Form Please feel free to come and go as you need to https://go.uncg.edu/prtf_ff

Academic Portfolio Review Task Force PRTF Charge (provided in April 2023): Develop a rubric for program review, including (a) consideration of potential weighting of the various data categories, and (b) the identification of any differences in the evaluation criteria used and/or weighting applied between undergraduate and graduate programs Review and assess metrics and data sources with insights from the data team that has been established Determine what qualitative (contextual) data should be assessed, including the quantity of materials Leave their discipline and unit mindsets at the door and adopt a University mindset. Host open forums to provide faculty and staff engagement and opportunities for input. PRTF is not responsible for making program decisions

Academic Portfolio Review Timeline Description Proposal for Portfolio Review Process 12/2022-4/2023 Process and Timeline shared with others on campus PRTF formation, charge, and development of rubric PRTF shares rubric and receives feedback from campus PRTF refines and clarifies rubric for campus Rubric is publicized and utilized in program evaluation Recommendations regarding program continuation/discontinuation Dates Responsibility Dean s working group/Provost Provost/Faculty/Deans 3/13/2023-3/23 3/28-8/2 Provost/PRTF members 8/2-9/1 PRTF/Faculty/Staff 9/1-9/15 PRTF 9/18-12/15 Deans and Unit level committees established by Deans Deans/Provost/Chancellor Data compilation and reporting 1/8/24-2/1

Data Availability for Rubric - Initial 1. Cost and Revenue of Delivery Faculty and staff FTE Personnel and non-personnel spending Revenue for department (tuition, differential tuition, state appropriation, etc) Credit hour production Cost per credit hour 2. Academic Program Demand & Instructional Productivity Applications, admits, enrolls Trend in headcount enrollment Number of degrees awarded 3. External Grants & Contracts Total grant and contract submissions, awards, and expenditures Grant-related student support 4. Student Success Undergrad First-year course completion rate Graduation rate Degree efficiency (number of hours completed upon graduation) Student Success Graduate Time extensions granted as a percentage of program enrollment Dismissals and probation as percentage of program enrollment Number of extension credits Labor Market & Workforce Reliance Job growth Job openings TBD: Qualitative/Contextual data Contextual information relevant to program and associated with data in items 1 6 5. 6. 7.

Data Availability for Rubric - Revised 1. Cost and Revenue of Delivery Faculty and staff FTE Personnel and non-personnel spending Revenue for department (tuition, differential tuition, state appropriation, etc) Credit hour production Cost per credit hour 2. Academic Program Demand & Instructional Productivity Applications, admits, enrolls Trend in headcount enrollment Number of degrees awarded 3. External Grants & Contracts Total grant and contract submissions, awards, and expenditures Grant-related student support 4. Student Success Undergrad First-year course completion rate Graduation rate Degree efficiency (number of hours completed upon graduation) Student Success Graduate Number of students who complete program within 5 (MS/PBC) or 7 (doctoral) years Percentage of students who graduate within the 5 7 year time frame Number of extension credits Labor Market & Workforce Reliance Job growth Job openings TBD: Qualitative/Contextual data Contextual information relevant to program and associated with data in items 1 6 Comparable to other programs in state or on campus Accreditation Other contextual information (e.g., number might have admitted but couldn t, changes in personnel that impacted costs, efficiency, or admits etc) 5. 6. 7.

Notables from Collaboration between PRTF and Provost/Academic Council of Deans Removal of Category 6 Labor Market Shifting of items Category 5 Graduate Student Success Weighting of individual items Varied weighting of categories across the rubric Contextual data Other important notables for UNCG generally: Inclusive process of faculty, staff, and administration Faculty, staff, and administrators are making recommendations (not external consulting groups)

Access the PRTF Documents View Documents Cover Page for Program Review Draft Rubric for Program Review Contextual/Qualitative Data Guide for Program Review https://go.uncg.edu/prtfrubric

Category 1: Cost and Revenue of Delivery Category weight UG: Category weight GR: (item weights and rubric labels) 39% 41% Item weights Exceeds Expectations (4) Meets Expectations (3) Approaching Expectations (2) Needs Examination (1) Notes and Additional Information 1.a is used to calculate 1f and 1g. The information for this item will be provided for contextual information and calculation, but will not be formally evaluated in this rubric. a. Faculty FTE (total full-time and part-time faculty FTE) and faculty headcount by department/school b. SHRA and EHRA Non-faculty Staff FTE (total full- time and part-time staff FTE) and staff headcount by department/school c. Personnel spending for all faculty, staff, and graduate teaching/research assistants at the department/school level 0% The information for this item will be provided for contextual information and calculation of information for 1.g, but will not be formally evaluated in this rubric. 0% 1.c is used to calculate 1g The information for this item will be provided for contextual information and calculation, but will not be formally evaluated in this rubric. 1.d is used to calculate 1g. The information for this item will be provided for contextual information and calculation, but will not be formally evaluated in this rubric. 0% d. Non-personnel spending for the department/school e. Revenue for the department (tuition, differential tuition, fees, state appropriation) Separate by undergrad-grad f. Credit hour production by full-time and part-time faculty at the department/school level Separate by undergrad-grad 0% The department's revenue is at the 85th percentile. The department's revenue is within the range of 46th to <85th percentile. The department's revenue is within the range of >15th to 45th percentile The department's revenue is in the lower 15th percentile. 30% The department's credit hour production is within the 46th to <85th percentile The department's credit hour production is within the range of >15th to 45th percentile. The department's cost per credit hour is within the range of 46th to <85th percentile The department's credit hour production is in the lower 15th percentile. The department's credit hour production is at the 85th percentile. 40% g. Cost per credit hour by department/school Separate by undergrad-grad The department's cost per credit hour is in the lower 15th percentile The department's cost per credit hour is within the >15th to 45th percentile The department's cost per credit hour is at or higher than the 85th percentile 30% Category 2: Academic Program Demand and Instructional Productivity/Efficiency Category weight UG: Category weight GR: (item weights and rubric labels) 29% 31% Item weights Exceeds Expectations (4) Meets Expectations (3) Approaching Expectations (2) Needs Examination (1) Notes and Additional Information h. Demand and yield applications, admits, enrolls The number of apps for the program are within the range of the 46th to <85th percentile The number of apps for the program are within the range of the >15th to 45th percentile The number of apps for the program are at the 85th percentile or higher The number of apps for the program are in the lower 15th percentile h.1. Demand (number of applications received) 10% Percentiles are calculated separately for UG, PMC/PBC, and graduate (MS/MEd/PhD/etc.) but PBC/PMC and graduate programs will be evaluated on the same rubric GR rubric h.2. Admit (number of applications completed and submitted in Slate) h.3. Yield (percentage of admitted students who enroll) The number of admits for the program are at the 8th percentile or higher The number of admits are within the range of the 46th to <85th percentile The number of admits are iwithin the range of the >15th to 45th percentile The number of admits are in the lower 15th percentile 10% Yield for the program is at the 85th percentile or higher Yield for the program is within the range of the 46th to <85th percentile Enrollment for the program is within the range of the 46th to <85th percentile Enrollment trends are comparable to UNCG total enrollment, by being within the 46th to 85th percentile Yield for the program is within the range of the >15th to 45th percentile Yield for the program is in the lower 15th percentile 10% Enrollment for the program is at the 85th percentile or higher Trend in enrollment outpaces UNCG enrollment trend, by being at the 85th percentile or higher Enrollment for the program is within the range of the 16h to 45th percentile Enrollment trends negatively outpaces UNCG's enrollment trend, by being within the >15th to 45th percentile Enrollment for the program is in the lower 15th percentile Trend in enrollment negatively outpaces UNCG's enrollment trend, by being in the lower 15th percentile i. Headcount enrollment for majors in the program 20% j. Trend in headcount enrollment growth k. Number of degrees awarded over a three-year time period, AY 2020-21, AY2021-22, and AY2022-23 25% Number of degrees awarded is in the top 85th percentile Number of degres awarded within the range of 46tyh to 85th percentile Number of degrees awarded is within the >15th to 45th percentile Number of degrees awarded is in the lower 15th percentile 25%

Category 3: External Grants and Contracts Category weight UG: Category weight GR: (item weights and rubric labels) l. Total annual grant and contract submissions and awards 10% 10% Item weights Exceeds Expectations (4) Meets Expectations (3) Approaching Expectations (2) Needs Examination (1) Notes and Additional Information The total amount of the department's grant and contract submissions is at the 85th percentile or above The total amount of the department's grant and contract awards is at the 85th percentile or above The total amount of the department's grant and contract submissions is within the 46th to <85th percentile The total amount of the department's grant and contract awards is within the 46th to <85th percentile The total amount of the department's grant and contract submissions is within the >15th to 45th percentile The total amount of the department's grant and contract awards is within the >15th to 45th percentile The total amount of the department's grant and contract submissions is in the lower 15th percentile The total amount of the department's grant and contract awards is in the lower 15th percentile l.1 total grant and contract submissions 10% l.2 total grant and contract awards m. Total annual grant and contract expenditures 20% The amount of the department's grant and contract expenditures, by dollar amount, is at the 85th percentile or higher The percentage of the department's grant and contract expenditures is at the 85th percentile or higher The amount of the department's annual salary savings by grant contract awards is in the 85th or higher percentile The total amount of the department's indirect-costs recovery is in the 85th or higher percentile The amount of the department's grant and contract expenditures, by dollar amount, is within the 46th to <85th percentile The percentage of the department's grant and contract expenditures is within the 46th to <85th percentile The amount of the department's annual salary savings by grant and contract awards is within the 46th to <85th percentile The amount of the department's grant and contract expenditures, by dollar amount, is within the >15th to 45th percentile The percentage of the department's grant and contract expenditures is within the >15th to 45th percentile The amount of the department's annual salary savings by grant awards and contract is within the >15th to 45th percentile The amount of the department's grant and contract expenditures, by dollar amount, is in the lower15th percentile The percentage of the department's grant and contract expenditures is in the lower15th percentile The amount of the department's annual salary savings by grant and contract awards is within the lower 15th percentile m.1 total annual grant and contract expenditures 20% m.2 total percentage of annual grant and contract expenditures Percentage is calculated by taking the amoung expended by the amount received (e.g., 100,000 spent of 200,000 equates 50%) 10% n. Salary savings from grants and contracts 13.30% The amount of the department's indirect-costs recovery is within the 46th to <85th percentile The total amount of the department's student support is within the 46th to <85th percentile The amount of the department's indirect-costs recovery is within the >15th to 45th percentile The total amount of the department's student support is within the >15th to 45th percentile. The amount of the department's indirect-costs recovery is in the lower 15th percentile o. Indirect cost recovery from grants and contracts 13.30% The total amount of the department's student support is in the 85th or higher percentile The total amount of the department's student support is in the lower 15th percentile p. Grant related graduate support and undergraduate support (GRA/GTA & waivers) 13.30%

Category 4: Student Success for Undergraduate Programs Category weight UG: Category weight GR: 22% NA Item weights - with s not included (in parentheses is the weights when item s is included) (item weights and rubric labels) Exceeds Expectations (4) Meets Expectations (3) Approaching Expectations (2) Needs Examination (1) Notes and Additional Information The department's first-year course completion rate is at the 85th percentile or higher The department's four-year graduation rate is at the 85th percentile or higher The department's first-year course completion rate is in the 46th to <85th percentile The department's four-year graduation rate is in the 46th to <85th percentile The department's first-year course completion rate is in the >15th to 45th percentile The department's four-year graduation rate is in the >15th to 45th percentile The department's first-year course completion rate is in the 0 to 15th percentile The department's four-year graduation rate is in the lower 15th percentile q. First-year course completion rate for cohort students by course department 25% (10%) r. Four-year graduation rate 65% (20%) 6-year graduation rates are not being examined or collected by rpk for the current dashboards. This is not being used in the calculation of the rubric/evaluation, however, descriptions have been provided if it is used in the future. The department's six-year graduation rate is at the 85th percentile or higher The department's total number of student credit hours completed upon graduation upon graduation is in the lower 15th percentile The difference between the total number of hours attempted vs. completed by graduates is in the lower 15th percentile comparatively to other programs The department's six-year graduation rate is in the 46th to <85th percentile The department's six-year graduation rate is in the >15th to 45th percentile The department's six-year graduation rate is in the lower 15th percentile s. Six-year graduation rate for first-time students 0% (60%) The department's total number of student credit hours completed upon graduation is in the >15th to 55th percentile The difference between the total number of hours attempted vs. completed upon graduation is within the >15th to 55th percentile The department's total number of student credit hours completed upon graduation is in the 56th to <85th percentile The department's total number of student credit hours completed upon graduation is in the 85 to 100th percentile t1. Degree efficiency, measured as total number of hours completed upon graduation 5% (5%) The difference between the total number of hours attempted vs. completed upon graduation is in the 56th to <85th percentile The difference between the total number of hours attempted vs. completed upon graduation is in the 85th to 100th percentile t2. Degree efficiency, measured by the difference between number of hours attempted vs. completed by graduates Category 5: Student Success for Graduate Programs Category weight UG: Category weight GR: (item weights and rubric labels) 5% (5%) NA 18% Item weights Exceeds Expectations (4) Number of students graduating from the program within the graduate school's allotted time frame is at the 85th percentile or higher Meets Expectations (3) Number of students graduating from the program within the graduate school's alloted time frame is within the 46th to <85th percentile Approaching Expectations (2) Number of students graduating from the program within the graduate school's alloted time frame is within the >15th to 45th percentile Needs Examination (1) Number of students graduating from the program within the graduate school's alloted time frame is in the lower 15th percentile Notes and Additional Information u. Number of students who complete their program within the University-established timeframes (5 years for master's degrees; 7 years for doctoral degrees) 40% Percentage is the # of students who graduated within the 5-7 year time frame compared to the number of students began the degree program during the same time frame.Obtaining data for this item will not be possible for programs that are newer, as a minimum of 8 years are reuqired to examine master's programs and 10 years to examine doctoral programs. The percentage of students graduating within the allotted time frame is at the top 85th percentile compared to other programs Number of extension credits used/taken by students in the program are in the lower 15th percentile The percentage of students graduating from the program within the allotted time frame is within the 46th to <85th percentile Number of extension credits used/taken by students in the program is within the >15th to 55th percentile range The percentage of students graduating from the program within the allotted time frame is within the >15th to 45th percentile Number of extension credits used/taken by students in the program is within the 56th to <85th percentil range The percentage of students graduating from the program within the allotted time frame is in the lower 15th percentile v. Percentage of graduates who completed their program within the graduate school's standard time frame (5 years for master's; 7 years for doctoral) 40% Number of extension credits used by students in the program is in the upper 85th percentile w. Number of sections of extension credits 20%

Qualitative/Contextual Data Category 1: COST AND REVENUE OF DELIVERY What specialized roles do staff play in the department? What kinds of non-personnel spending is the department responsible for that impacts its overall operation? What requirements limit credit hour production or cost per credit hour (e.g., secondary accreditation, safety concerns, etc.)? (This item also may impact category 2, 4, and 5) Provision of cognate requirements for other departments and programs Cross listed faculty that may impact FTE in one department and SCH generation in a second department What, if any, specialized pedagogy is required for the program that may affect cost and revenue of delivery (e.g., individualized instruction, clinical supervision or oversight, field based experiences)? Category 2: ACADEMIC PROGRAM DEMAND & INSTRUCTIONAL PRODUCTIVITY/EFFICIENCY Recent internal and/or external changes in departments or programs (e.g., new programs, faculty transitions, program closures) that have impacted applications, enrollment and/or demand? What factors impact students completion of degree? Category 3: EXTERNAL CONTRACTS AND GRANTS What are the opportunities for grants and contracts within the discipline? Is there community engagement of staff, faculty, and students that is a result of grants and contracts? Do the grants/contracts produce community engagement hours for staff/students? Are there departmental employment expectations to obtain grants and contracts? Describe any inter-institutional partnership grants and contracts and/or training grants and the impact these have. What factors inhibited the expenditure of grant funds, if any? Is the department primarily undergraduate, graduate or a mix of both, and how does this impact grant funding or contracts received by the department? Category 4: STUDENT SUCCESS FOR UNDERGRADUATE PROGRAMS What factors commonly impact the academic success of the department s first-year students (class size, preparedness of UG student population, progression through tiered coursework)? What program-based factors commonly impact UG students graduation rates specific to the department (programs of discovery)? What UG student-based factors commonly impact students graduation rates specific to the department (skill proficiency, population served, etc.)? Do departments provide opportunities for UG students beyond coursework that impact quality of education (undergraduate research, research assistantships, student University service positions, internships, community service/engagement, etc.) Category 5: GRADUATE STUDENT PROGRAM SUCCESS Number of graduate students teaching courses on campus, and/or SCH generation by graduate student Graduate students in service or support positions on campus (GAs that serve the institution or undergraduate student needs on campus) Additional impact of the program on graduate student success (e.g., research, publications, community engagement, conference proposals, securing of research funding, creative activities) Do departments provide opportunities for graduate students beyond coursework that impact quality of education (undergraduate research, research assistantships, student University service positions, internships, community service/engagement) Part II 1) Other factors of the program (not listed above) impact aspects of the criteria on the rubric during the review period. 2) Does the department rely on other departments or programs for courses? This may include specific tiered or pre-requisite courses, cross-listed courses, program minors, or certifications. 3) Does the department contribute to the success of the student population beyond its own department (i.e., MAC-serving coursework)? 4) Comparison of program to other programs (e.g., number of similar programs in the state, state or national ranking of program) 5) How does the program tie into the mission of the University 6) Scholarly products and creative contributions of faculty and staff 7) The labor market and employment can be an important component of a program s demand and impact. rpk's labor market data (original category 6) will not be considered in the Program Review process, given the data was problematic. Therefore, if appropriate for your department/program, please indicate what types of jobs and positions your graduates get after they graduate from your program (e.g., please provide SOC codes where appropriate)? Do you have additional data or information to reflect your program s job market value?