Independent Verification of Graduate Employment in Vocational Institutions

Best Practices Best Practices: : Independent Independent Third Verification of Placement Verification of Placement Third- -Party Party Christopher Lambert Associate Executive Director ACCSC

Webinar Agenda Foundational Thoughts Guidelines for Employment Classification: A Documentation Framework The Independent Verification Process First Look: Independent Verification Results Best Practices: Lessons Learned Frequently Asked Questions

Foundational Thoughts Foundational Thoughts

Foundational Thoughts for Todays Presentation Meaningful things are happening every day at ACCSC-accredited institutions. In 2014-2015, ACCSC conducted over 500 on-site evaluations Every single day students are engaged in the classroom/labs, going on externships, taking licensure exams, graduating with skills needed to gain entry into a vocational field, and ultimately starting on their career path. The average graduate placement rate at ACCSC-accredited institutions from 2012 to 2014 was 76%.

Foundational Thoughts for Todays Presentation Building / Reestablishing Confidence ACCSC has always required schools to maintain verifiable records of initial employment and believes that initiatives like the independent verification of graduate employment, unannounced visits, and regular student achievement reporting is building / reestablishing confidence around the success of ACCSC- accredited schools and the students they serve.

Foundational Thoughts for Todays Presentation Challenges Remain Despite the positive trending in student achievement, demonstrating acceptable rates of student graduation and graduate employment in the career field for which the school provided education remains a significant challenge

155 Total Team Findings in 51 Team Summary Reports Considered at the February 2015 Commission Meeting 25 20 15 10 5 0 Student Achievement Admissions Adverstising Management PAC Avg. Slightly More than 3 Team Findings Per TSR Student Achievement Admissions Adverstising Management PAC

237 Total Team Findings in 67 Team Summary Reports Considered at the November 2014 Commission Meeting 30 25 20 15 10 5 0 Student Achievement Management Admissions IAIP PAC Avg. Slightly More than 3 Team Findings Per TSR Student Achievement Management Admissions IAIP PAC

July 1, 2013 June 30, 2014: ACCSC Accredited Institutions on Reporting 60 50 40 30 20 10 0 Refund Reporting Program Advisory Committee Reporting Heightened Monitoring Outcomes Outcomes Warning

Changing the Dialogue From Skepticism to Confidence

Increased Federalization of Higher Education The expectations of accreditors are changing significantly Accreditation as we know it today has been subject to increased scrutiny and criticism by a variety of organizations and policy makers who have valid questions about whether or not accreditors are fulfilling their promise. Fundamentally, does accreditation help ensure: 1) the quality of education provided by its member institutions for students; and 1) the continued integrity of the Title IV federal student financial aid program.

Changing the Dialogue: From Skepticism to Confidence The enterprise of accreditation is evolving and improving. A key example of how national accreditors have embraced the challenge of demonstrating that accreditation is both reliable and dependable without increased involvement by the federal government is in the area of student achievement assessment. In 2011, ACCSC initiated a new process whereas the Commission subjected the reported rates of student achievement to independent verification 10% random sample of Annual Report Data.

Changing the Dialogue: From Skepticism to Confidence In 2014, ACCSC adopted a policy requiring institutions seeking accreditation to engage an independent third-party auditor to verify the most recently reported graduate employment data as a means: to bolster confidence in accreditation and to reinforce a long standing requirement that institutions maintain verifiable records of initial employment. ACCSC believes that the third party independent audit requirement significantly strengthens accreditation and helps to change the dialogue from skepticism to confidence.

Guidelines for Employment Classification A Documentation Framework

POLICIES AND PROCEDURES Section I (A)(1)(d), Substantive Standards, Standards of Accreditation The school has and adheres to policies or protocols in regard to employment classification (reasonable time period, sustainability, relatedness, etc.). Action Items: Are the school s policies in line with ACCSC s Guidelines for Employment Classification? Appendix VII of the Standards of Accreditation Is the school working to update its policies regarding how it ensures that it maintains verifiable records of initial employment? If the school is working with a third-party, is there a signed contract?

Verifiable Employment Record A verifiable employment record includes, at a minimum the following information: Graduate Name and Contact Information; Date of Initial Employment; Place of Employment; Employer Address and Phone Number; Employer Contact Person/Supervisor; and Descriptive Job Title and Duties. Action Item: Does the record include all areas as required?

Justification for Employment Classification Appendix VII Guidelines for Employment Classification The employment classification is appropriate and reasonable based on the educational objectives of the program. Action Items: Does the placement seem sensible based on the program objectives that the graduate completed? If the Third-Party was unable to verify the placement, can the school demonstrate a reason for listing the graduate as employed?

Justification for Employment Classification Appendix VII Guidelines for Employment Classification Employed in Field: I. is for a reasonable period of time, II. is based on program objectives, and III. can be considered sustainable (e.g., not a single day of employment). Action Items: Does the school follow its internal policy regarding employment for a reasonable period of time? When is the school capturing the information, is it the first day the student works? Is there any additional follow-up after a period of time? Is the policy consistently applied to all graduates?

Justification for Employment Classification Appendix VII Guidelines for Employment Classification Regular Employment I. is directly related to the program from which the individual graduated; II. aligns with a majority of the educational and training objectives; and III. is a paid position. Action Item: Does the documentation reference the skills required for the position which match the educational and training objectives of the program? Verifiable Record + Job Description / Announcement

Justification for Employment Classification Appendix VII Guidelines for Employment Classification Self-Employment The school secures written documentation from the self-employed graduate verifying that the employment is valid including, at a minimum, a statement signed by the graduate which includes the following: The graduate s name and contact information; An attestation that the self-employment is aligned with the individual s employment goals, is vocational, and is based on and related to the education and training received; An attestation that the graduate is earning training-related income; and In cases where a licensure is required for employment, an attestation that such licensure has been achieved. Action Items: Does the self-employment statement or form include all areas as required? Is the license current/active? Did the student take the exam but not yet receive a result?

Justification for Employment Classification Appendix VII Guidelines for Employment Classification Career Advancement Students that are already employed at the time of graduation can be considered employed when completing the program of study as follows: The school shows with written documentation from the employer or the graduate that the training allowed the graduate to maintain the employment position due to the training provided by the school; or The school shows with written documentation from the employer or the graduate that the training supported the graduate s ability to be eligible or qualified for advancement due to the training provided by the school. Action Items: Is the hire date prior to the graduation date? If so, did the school provide Career Advancement documentation?

Independent Third-Party Verification of Employment: The Process

Be Sure to visit the Resources / Independent Verification Section of www.accsc.org for more information on The Independent Third-Party Verification of Employment Process

Independent Third-Party Verification of Employment Process: Selecting a Third-Party The auditor does not go through any type of approval process with ACCSC prior to the project To date 21 different third-parties have submitted an independent audit. The auditor is required to attest to the following to verify its independence and third party nature: Not affiliated with the school Does not provide any other service to the school Exception: Certified Public Accountant The auditor must verify employment data in a manner independent from the school

Independent Third-Party Verification of Employment Process: Selecting a Third-Party The auditor must select a minimum of a 50% sample of employed graduates from each program offered The auditor must secure written or verbal verification from the graduate or employer Not just an audit of back-up documentation The auditor must provide a summary of results and a description of the methodology used The auditor must sign a signed certification attesting to the accuracy of the findings and the auditor s independence

Independent Third-Party Verification of Employment Process The school must engage an independent third-party to verify the employment data in the Graduation and Employment ( G&E ) Chart(s) submitted in the most recently submitted Annual Report at the time of the due date of the SER. 2015 ACCSC Annual Report The independent third-party must select minimally a 50% sample of employed graduates (classified as Graduates - Employed in the Field ) and report the results from that sample. 50% sample from each Graduation and Employment Chart filed in the ACCSC Annual Report. Program by Program Once the 50% sample has been selected by the independent third-party, for the purposes of this application, the independent third-party may not alter the sample in an effort to get better results.

Independent Third-Party Verification of Employment Process The report must include a summary of results for each G&E Chart that includes the following (at a minimum): a. The total number of students in the sample; and b. The number of students in each of the categories below and the percentage in each category: Verified as Correct(graduate s employment is verified as listed in the school s record); Not verified (unable to verify after all attempts); Verified as Not Correct (e.g., graduate not found in employer records, position is unpaid/externship, other discrepancies); and Verified but Different (e.g., different start date, job title, employer, etc.). Any additional information or response the school deems appropriate regarding the above results.

Independent Third-Party Verification of Employment Process An on-site evaluation team reviews the verification results from the independent third-party A school may provide the updated information to the on-site evaluation team as long as the updated information was subject to the same methodology identified in the school s SER. If an on-site evaluation team does find that a school has not demonstrated compliance with accrediting standards due to the accuracy or validity of the data, the team will likely include a team finding / request for additional information in the Team Summary Report The school will be required to respond to the Commission with supporting documentation to demonstrate that the student achievement data is accurate and verifiable. The Commission, not the on-site evaluation team, makes the final determination regarding a school s compliance with accrediting standards.

First Look: Independent Verification Results 2011 2014 ACCSC Annual Reports

4 Year Averages 2011 to 2014 Random Sample 303 Schools Randomly Selected for Independent Verification 12,894 Employment Records The percentage of records classified as Verified as Correct 73% (9,348 of 12,894) The percentage of records classified as Unable to Verify 21% (2,706 of 12,894) The percentage of records classified as Verified as Not Correct 4.4% (580 of 12,894)

Random Sampling Results 2011 - 2014 80 70 60 50 40 30 20 10 0 Verified Unable Not Correct

First Look: Independent Verification Results 2014 / 2015 Accreditation Cycle

Independent Verification: 2014/15 Accreditation Cycle 131 Schools with Results 21 Different Independent Third Parties 10,010 records 99% (130 of 131) schools met the required 50% sample. 10% (13 of 131) submitted 100% samples.

Independent Verification: 2014/15 Accreditation Cycle Verified as Not Correct Verified as Correct Mean Verified as Correct Median Not Verified Verified but Different 76% 83% 18% 4% 2% January 2014 84% 94% 11% 4% 1% March 2014 78% 83% 14% 6% 2% July 2014 63% 67% 25% 8% 3% September 2014 Overall 75% 79% 17% 5% 2%

Independent Verification: 2014/15 Accreditation Cycle 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Jan-14 Mar-14 Jul-14 Sep-14 Verified as Correct - Mean Verified as Correct - Median Unverified Verified as Not Correct Verified but Different

Best Practices / Lessons Learned

Best Practices / Lessons Learned Program Viability and Program Viability and Assessment Assessment Accreditors expect schools to conduct a realistic assessment of the viability of program offerings and the factors impacting student achievement. In all cases, accreditors expect that a school will take appropriate and responsible action regarding any program offering that is not meeting expectations. Schools should makes the decision about a program s continued viability, and should not put this decision into the hands of a regulatory body (state or accreditor). Some guiding questions include: At what rate are students graduating? At what rate are graduates passing licensure exams? At what rate are graduates getting a job in a training related field?

Best Practices / Lessons Learned Program Viability Program Viability and Assessment and Assessment Are graduation and employment rates getting better? Getting worse? Each year, accredited institutions are required to submit student achievement data to their accreditor via an annual report How is the data trending year over year? Are there any mitigating factors impacting the school s ability to demonstrate successful student achievement? Economic Conditions, Unemployment, Student Population Served, etc. Did the school examine the rates of student graduation in relation to admissions standards? Is the school open enrollment? What changes has the school made with respect to the program curriculum or career service initiatives?

Best Practices / Lessons Learned Working with an Independent Third-Party Keep in mind that the results that are provided by the independent third party should be: Useful The report should provide specific details in each of the categories identified that presents the results in an easy to understand format for the school. Usable Does the report provide the necessary information, in an organized, detailed fashion to afford the school an opportunity to explore any unable to verify or verified as not correct ? For any graduate found to be verified as not correct the report should be specific as to why.

Best Practices / Lessons Learned Sortable the school should be able to easily sort through the data, for example by category or by employer, to gain additional information on any trends in the results of the audit. The school is responsible for making sure third-party results are submitted on- time and organized in accordance with the guidelines. The third-party should be able to meet the institution s timelines and needs. Be specific with the due date of results as part of the contract with a third- party.

Best Practices / Lessons Learned Start Early. Real Time Verifications Do not wait fox six months to begin the verification process. Older employment data proves to be more difficult to verify. Batching Verifications - Weekly? Monthly? Quarterly? Verify the employment AFTER the graduate started work. This is the most common reason for an invalid placement Many times a graduate is offered a position (after externship) and the school counts that student as employed in field; however, the school did not verify that the person actually started. Verify 5, 7, or 14 days after the graduate started, began working, and received pay.

Best Practices / Lessons Learned Incorporate Third-Party Verification into Regular On-going Processes This requires budgetary / financial support For programs that have stable/reliable/consistent employment results, consider having an independent third-party conduct a 10% / 25% sampling each year. Consider having an independent third-party conduct a 50% to 100% sampling every year for programs with: 1. A significant number of self-employed graduates (prove more difficult to verify), 2. Programs that have historically had more challenges with employment, or 3. Programs that have not faired well during a previous verification process (i.e. high number of unable to verify / verified as incorrect)

Best Practices / Lessons Learned Set Expectations with Students Begin on the First Day of Orientation and continue throughout the program Help Students to Understand Their Role in the Accountability Framework Accreditation Resources: New Student Letter / Graduation Letter Establish protocols for post-graduation activities that can be managed and tracked. Admissions Protocols as Template Have students sign a release form that permits the school to obtain employment information Signed and dated upon graduation, not start, as frequently employers that require a signed release want it to be signed within 12 months of the request.

Best Practices / Lessons Learned Set expectations with Career Services Random, Unannounced, In-House Review of Graduate Employment BEFORE subjected to Third- Party Audit. Ensure Complete, Descriptive, Clear, Legible, Graduate Employment Records Maintain up-to-date employment records in the school s database or in another centralized manner - Not in individual files or post-it notes Sloppy record keeping is another common reason why a record is not verifiable Include:

Best Practices / Lessons Learned Set expectations with Employers Work proactively with Employers Set expectations, Get feedback Batching Verification When we contact employers who hire multiple graduates, have the third-party do it as one call/email/fax with the request for multiple graduates. Avoid calling employers to scrub data in advance of the third-party verification process, only to have your 3rd party call them. Employers and Graduates get frustrated that they just gave that information

Best Practices / Lessons Learned Maximize Your Relationship with Your Designated Third-Party Get weekly reports while the project is underway Ask for the option to update incorrect emails or wrong telephone numbers while the project is underway Require the third-party to use the revised contact information Ask the third-party about the number of calls/emails and the pacing of outreach as to not frustrate your employers/graduates

F Frequently requently A Asked sked Q Questions uestions

Frequently Asked Questions Q: Does my school have to have an independent third-party verification of employment data every year? A: No. ACCSC requires independent third-party verification only at the time of the Application for Renewal or Initial Accreditation and results are due at the same time as the Self-Evaluation Report ( SER ). A school may opt to have employment information verified annually or on-going if it chooses to.

Frequently Asked Questions Q: Which Annual Report will my school need to have verified by the independent third-party as part of my application for accreditation and submission of my SER? A: Schools must have independently verified the employment data from the Graduation and Employment Chart(s) from the most recently submitted Annual Report on file with the Commission at the time of the due date of the SER.

Frequently Asked Questions Q: In order to be considered independent, the third-party cannot be one that provides other services to the school (with the exception of a Certified Public Accountant). What does other services mean? A: Other services means that the school must choose a third-party that the school does not have any other contract or arrangement with to provide services or support to the school. Other services are those such as, but not limited to: Recruiting, advertising, and admissions; Admissions documentation authentication or translation; Education Credit Assessment; Background checks and drug screening services; Administrative services (e.g., payroll); Student support or help desk services; Library resources; Information technology support; Employment placement services or research to find where graduates are working; Student loan repayment support or management;