Performance Accountability in the World of WIOA
WIOA brings significant changes in key accountability concepts, participant exits, and reporting requirements, requiring a structured target-setting approach. Implementation is ongoing, impacting Texas and beyond.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Performance Accountability in the World of WIOA
Introduction WIOA brings Massive Accountability Changes Especially for Texas Changes in Key Accountability Concepts Participant Exit Exclusions Replaces Existing Common Measures Creates Significant new Reporting Requirements Creates Highly Structured Process for Target Setting & Negotiations Implementation of WIOA s Accountability Provisions is a Work-in-Progress 2
Overview of Topics WIOA Performance Accountability Background Performance Accountability Constructs Participant / Exit / Exclusions Introduction to the New Common Measures Employment/Earnings Outcomes Definitions / Calculations Comparison of Old & New Data Education Outcomes Definitions / Calculations Comparison of Old & New Data Effectiveness Serving Employers Data is the Key Required Data & Allowable Data Sources Target-Setting & Statistical Modeling Implementation 3
Key Terms & Abbreviations Title I / Adult, Dislocated Worker, Youth (Adult, DW, Youth) Title II / Adult Education & Family Literacy (AEL) Title III / Wagner-Peyser (WP) Title IV / Vocational Rehabilitation (VR) Temporary Assistance to Needy Families (TANF) Supplemental Nutrition Assistance Program Employment & Training (SNAP E&T) Career & Training Programs (C&T) Primarily Adult, DW, Youth, WP, TANF, SNAP E&T Generally also includes special initiatives Essentially All but AEL, VR, and Child Care Jobs for Veterans State Grants (JVSG) 4
Key Terms & Abbreviations (continued) Department of Labor (DOL) Employment Training Administration (ETA) Department of Education (ED) Office of Career, Technical, & Adult Education (OCTAE) Rehabilitation Services Administration (RSA) The Departments DOL & ED The Secretaries The Secretaries of DOL & ED Participant Individual Record Layout (PIRL) OCTAE s National Reporting System (NRS) Notice of Proposed Rule-Making (NPRM) Information Collection Request (ICR) Quarter (Qtr or Q) YYYYQX Qtr X in Calendar Year YYYY (e.g. 2004Q1) 5
Key Terms & Abbreviations (continued) WorkInTexas.com (WIT) TWC s job matching system; also used as the primary system for recording Wagner-Peyser services, included by TVC staff TWIST TWC s integrated case-management system used by most programs other than VR & AEL TEAMS TWC s AEL case-management system ReHabWorks (RHW) TWC s VR case-management system On-the-Job Training (OJT) Reemployment & Employer Engagement Measures (REEMs) Measurable Skills Gain (MSG) Effectiveness Serving Employers (ESE) Educational Functioning Level (EFL) 6
WIOA Performance Accountability Background
WIOA WIOA Signed into law July 2014 WIOA Replaces Workforce Investment Act & Includes 6 Programs Title I Adult, Dislocated Worker, Youth (Adult, DW, Youth) Title II Adult Education & Family Literacy (AEL) Title III Wagner-Peyser (WP) Title IV Vocational Rehabilitation (VR) Common Performance Accountability & Reporting 116(a) establishes performance accountability measures that apply across the core programs to assess the effectiveness of States & local areas . . . in achieving positive outcomes for individuals served by those programs Other Federal Programs Impacted by WIOA SNAP E&T now requires use of WIOA-esque measures Draft TANF Bills proposing WIOA or WIOA-esque measures 8
Where We Are: WIOA Regulations WIOA Required Draft Regulations by April 2015 DOL & ED timely published 2,400+ pages of proposed regulations for 60 Day Comment Period Nearly 2,900 sets of comments submitted Final Regulations Issued Required January 2016, but not issued until June 2016 Final Package 4,100 pages in length Preamble Estimated Impact/Cost Comments & Responses Regulations 298 pages on Performance Accountability Complete/Final? Numerous holes to be filled by Further Guidance 9
Where We Are: Reporting Secretaries supposed to develop Joint Reporting Specifications by July 2015 July 2015 Draft Information Collection Request (ICR) published for 60 Day Comment Period, receiving 112 Sets of Comments April 2016 2nd Draft ICR published for 30 Day Comment Period, receiving unknown # of comments June 2016 Final ICR with Joint Specs published Numerous References to more Guidance to Come Apparently the Departments are already working on amendments Joint Specs only include the elements jointly in common across all 6 WIOA programs DOL, OCTAE, RSA each require additional data WIOA didn t specify a due date for these items Requirements issued under separate means 10
Where We Are: Reporting DOL-specific ICR September 2015 Draft published for 60 Day Comment Period, receiving 68 sets of comments April 2016 2nd Draft published April 2016 for 30 Day Comment Period receiving unknown sets of comments June 2016 Final DOL Specifications Published Numerous References to more Guidance to Come RSA-specific ICR April 2015 Draft RSA-911 ICR published for 60 Day Comment Period June 2016 Somehow RSA finalized the RSA-911 without going through a 2nd draft ICR with Comment Period OCTAE-specific ICR September 2016 Finally proposed a new version their National Reporting System for public comment 11
Performance Accountability Constructs & Key Reporting Issues
Reportable Individual vs Participant Most important definition in WIOA Accountability? Definition of Participant Nearly all Participants are included in WIOA measures Not Everyone? No Some People are only Reportable Individuals Others are excluded for factors outside the system s control WIOA Rule 677.150(b) Reportable Individual is an individual who has taken action that demonstrates an intent to use program services and who meets specific reporting criteria of the program including: Individuals who provide identifying information; Individuals who only use the self-service system; or Individuals who receive information-only services or activities 13
Participant WIOA Rule 677.150(a) defines as a Participant as a person determined eligible for service AND met specific other programmatic requirements: AEL completed 12+ contact hours EVEN IF THEY CROSS PYs TWC will not apply the Cross PY provision to AEL grantees this year VR has an approved/signed Individual Plan for Employment (IPE) & has begun to receive services Very different from old State Measure Not all Consumers are Participants No IPE? Not a Participant Adult/DW/WP has received Staff-Assisted qualifying services other than Information-Only Services VERY different than old Customer definition excludes those receiving only Self-Service & Information-only Service (even by staff) TWC is applying this definition for other C&T Programs Youth has received an Objective Assessment, development of an Individual Service Strategy (EDP) & any of the 14 WIOA Youth program elements identified in WIOA 129(c)(2) 14
Information-Only Services? Information-Only Services are not Specific to the Person being Assisted They are not dependent on assessing or understanding a person s: Skills Education Career Objectives Barriers to Employment Other specific needs General List of Information-Only Services: All Orientations Computer Work Station Usage (22) LMI (29/LMI, 191/SLM) WOTC Eligibility (139/TCE) Referral to Educational Service (141/RES) Referral to WIOA or Other Training note that this is the REFERRAL only, not the actual provision of training (the training itself would be Qualifying) 15
14 Youth Elements Tutoring, Study Skills Training, Instruction, & Evidence-based dropout prevention & recovery strategies Alternate Secondary School Service/Dropout Recovery Services Paid & Unpaid work experience including employment, pre-apprenticeship, internships/job shadowing, & OJT Occupational Skill Training Education offered Concurrent & in the Context of Workforce Preparation activities and training for a specific occupation or occupational cluster Leadership Development Supportive Services Adult Mentoring (which must last at least 12 months) Follow-up Services (which again, must last at least 12 months) Comprehensive Guidance/Counselling Financial Literacy Entrepreneurial Skills Training Career Awareness, Career Counseling, Career Exploration, etc. Activities to help prepare for/transition to Postsecondary Education & Training 16
Customer/Consumer vs Participant Data reported will look very different for VR and C&T VR served 102.9K Consumers in SFY16 but only 75.3K (73.2%) are Participants under the new definition 11,104 Consumers received Blind Services of whom 7,219 (65.0%) are considered Participants 91.7K Consumers received General VR Services of whom 68.3K (74.4%) are considered Participants C&T Programs served 1.27M Customers in SFY16, but only 830K (65.44%) are Participants under the new definition Youth Program served 8,166 Customers in SFY16, but only 5,799 (71.01%) are Participants under the new definition JVSG data were generally not affected because JVSG- services involves staff-assistance (generally beyond Information-Only service) 17
When Does Exit Occur? Departments Developed Separate Standards VR Exit is upon Closure of File in accordance with RSA regulations (Unless Closed due to Supported Employment in an Integrated Environment, but NOT in Competitive Integrated Employment) AEL/C&T Exit is on Date of last Qualifying Service, but is determined based on lack of Qualifying Service for 90 days Self-Service & Information Service do not Extend Participation Planned Gap in Services can prevent exit IF the Participant is expected to return on a specific date for specific services If Participant does not return as planned Exit is retroactive back to the date of the last Qualifying Service Common Exit not allowed across OCTAE, RSA, & DOL DOL requires Common Exit across their programs TWC intends to use Common Exit across all C&T programs ED Claims to be evaluating Common Exit & TWC would likely want to Pilot such a system 18
Impact of Lack of Common Exit? Undercuts the Idea of an Integrated System WIOA Envisions Stronger Integration & Coordination of Services Focus should be on the PARTICIPANTS & their outcomes Siloed Exits & Siloed Reporting puts the Focus on the Program Has the Potential to Distort Outcomes Most WIOA measures focus on specific Post-Exit Qtrs If a Participant is coenrolled across AEL, C&T, and/or VR with exits in different quarters, the Participant could be reported differently for each program Biggest Potential Impact is probably on Earnings Outcomes Example: AEL Participant Exits late 2015Q2 continuing Work Search in C&T becoming employed mid 2015Q4; Wages in 2015Q4 are for a partial quarter. Wages for 2016Q2 (2nd qtr after C&T Exit) are more likely to be a full quarter 19
Who is Included in Performance? ALL Participants without a Valid Exclusion: Institutionalization (criminal or health) Participants Receiving AEL while Incarcerated are included in Measurable Skills Gain, but not employment, earnings, and credential outcomes Health/Medical condition expected to last at least 90 days that precludes continued participation in the program or employment Death Reserve Forces called to Active Duty Determination of Ineligibility (correcting an earlier mistake) Those in Foster Care who exited due to mandated move from the area (only applies to those 14-21) BIG CHANGE - Lack of a Valid SSN No Longer Excludes Participants 40,118 out of 94,391 PY16 AEL Participants, did not have SSNs recorded in TEAMS (42.5%) Difference between 55% and 31% Employed Q2 Post-Exit 20
Consequences of Failure Greater Consequences for Failure under WIOA 2nd Consecutive Year of Failure by a State results in a 5% Reduction to the Governor s 15% Statewide Funds 3rd Consecutive Year of Failure by a local Board results in: Appointment of a New Board Prohibition in the use of certain Providers & Partners found to have achieved a poor level of performance Other Significant Actions as Deemed Appropriate WIOA Definitions of Failure State is Considered to Have Failed if Average % of Target is <90% for all Measures in any PROGRAM Average % of Target is <90% for all Programs in any MEASURE % of Target on any single Measure is <50% States have Flexibility in Setting Standards for Locals & other Subrecipients Legislature sets Failure at Missing Target by >5% 21
Introduction to the New Common Measures
Origin of New Common Measures Integrated Performance Initiative DOL-funded grant from 2004 led by Washington State with Goal to Develop Measures Of Importance to Elected Officials, Policy-Makers, & Stakeholders Acceptable to WF, K-12, & Post-Secondary Education Initial Workgroup of 7 States produced 1st Proposal Proposal validated/revised with 10 other States Proposal validated/revised AGAIN thru discussions with Think-tanks & other Experts Results Championed by National Governors Association in all WIA Reauthorization drafts Ultimately adopted in WIOA with few changes Measurable Skills Gains added in by OCTAE 23
New Federal Common Measures New Common Measures in WIOA Statute 1. Employed Q2 Post-Exit (EmpQ2) 2. Employed Q4 Post-Exit (EmpQ4) 3. Median Earnings Q2 Post-Exit (MedEarnQ2) 4. Measurable Skills Gain (MSG) 5. Credential Rate 6. Effectiveness in Serving Employers (ESE) Totally Undefined in Statute MSG & Credential Rate don t apply to Wagner-Peyser Not a Training Program Youth versions of Employed Q2 & Q4 also count Enrolled in Education (Emp/Ed Q2, Emp/Ed Q4) Changes are most substantial for VR 24
New State Common Measures Federal Measures were a Missed Opportunity Difference between Youth & NonYouth Measures is counting Enrollment in Education Why is going back to school Good for a 22 or 24 year old, but not a 25 or 30 or 50 year old? Federal Measures also don t include Retention Employment in Q4 is not the same as retaining employment State WIOA measures correct these issues by: Focusing on Employment or Enrollment in School/Training Dropping the Q4 measure in favor of a Retention measure 25
Post-Exit Outcomes Employment/Earnings/ Educational Enrollment
Employed Q2 Post-Exit DENOMINATOR All Exiters without a valid Exclusion Not just those Unemployed at Participation NUMERATOR Employed Q2 Post-Exit Unsubsidized Employment Only Good Picks up instances where we help an employed person get a 2nd Job Bad Same Pre- & Post-Exit status considered successful (still employed in original job) Adds 1 Quarter of Lag to the Measurement Period Minimize Affect of Added Lag by Running Data on Employed Q1 ~90% of those Employed Q1 are also Employed Q2 Reported to OCTAE, RSA, & DOL (all but Youth) 27
Lets Look at Some Data!!! Program Entered Employment or % Rehabilitated Employed Q2 Post-Exit C&T 78.4% 69.9% JVSG 72.9% 65.3% AEL 45.5% 31.7% VR (different periods) 59.0% 57.9% C&T differences not obvious Entered Employment was 741.2K out of 945K Exiters Employed Q2 Post-Exit was 565K out of 808K Exiters Difference is in definition of Participant & new measure includes employed OR unemployed at Participation AEL differences mostly due to no Lack of SSN exclusion Excluding non-SSN Participants raises performance to 55% VR used to focus on status at Exit vs. Post-Exit Differences may also be due to lack of way to record Post-Exit employment in RHW 28
Employed Q4 Post-Exit DENOMINATOR All Exiters without a valid Exclusion NUMERATOR Employed Q4 Post-Exit Unsubsidized Employment Only Not a Retention Measure Not a Subset of those Employed Q2 Good? Nothing Good about it Bad Meaningless Measure Counts those not employed Q1-3 Post Exit if Employed Q4 Same Pre- & Post-Exit status considered successful Adds 1 Quarter of Lag to the Measurement Period Reported to OCTAE, RSA, & DOL (all but Youth) 29
Lets Look at Some More Data!!! Program Employment Retention Employed Q2 Post-Exit Employed Q4 Post-Exit C&T 84.6% 69.9% 69.9% JVSG 84.1% 65.3% 65.3% AEL 58.3% 31.7% NA VR 88% 57.9% 56.1% REMEMBER Employed Q4 is NOT a Retention Measure It is essentially the same population as the Employed Q2 measure followed-up upon 2 quarters later 30
Median Earnings Q2 Post-Exit Median Earnings is the Earnings Level at which half of those employed earn MORE & half earn LESS $5,000, $5,000, $5,050, $6,000, $6,000 Median is $5,050 $4,900, $5,000, $6,000, $6,266 Median is $5,500 Good Shorter Lag than old Average Earnings (Q2 vs. Q3) Median reduces influence of outliers 5 Exiters with $5,000, $5,000, $6,000, $6,500, $150K Earnings Average is $34,500 Median is $6,000 Bad Performance tends to be about 1/3rd the old measure Smaller Numbers are less meaningful to Stakeholders Reported to OCTAE, RSA, & DOL (including Youth) 31
Lets Look at Some More Data!!! Program Average Earnings (Average Earnings in Q2/Q3 of those employed in Q1/Q2/Q3) Median Earnings C&T $15,895 $4,956 JVSG $18,253 $6,616 AEL NA $4160 VR $12,468 $4,319 Old Measure Only included those employed in Q1, Q2, & Q3 Tended to include more people with Stable Employment For JVSG this was 16,782 Exiters New Measure includes anyone with any Wages in Q2 Post-Exit For JVSG this was 18,643 Exiters 32
Employed/Enrolled Q2 Post-Exit DENOMINATOR All Exiters without a valid Exclusion Not just those Unemployed or Not in School at Participation NUMERATOR Employed or Enrolled in Education Q2 Post-Exit Unsubsidized Employment Only Education or Advanced Training Good Recognizes Enrollment in Education as a Positive Outcome Bad Same Pre- & Post-Exit status considered successful We don t have much post-exit Enrollment Data Yet Adds 1 Quarter of Lag to the Measurement Period Reported to LBB (VR, AEL, C&T) & DOL (Youth Only) 33
Employed/Enrolled Q2-4 Post-Exit (Retention) DENOMINATOR All Exiters without a valid Exclusion Employed or Enrolled in Education in Q2 Post-Exit NUMERATOR Employed or Enrolled in Education both Q3 & Q4 Post-Exit Unsubsidized Employment Only Education or Advanced Training Can be any combination of Employment & Education Good RETENTION!!! Deeper look at Success Bad Same Pre- & Post-Exit status considered successful Adds 1 Quarter of Lag to the Measurement Period Reported to LBB Only (VR, AEL, C&T) 34
Lets Look at Some More Data!!! Program Employment Retention Employed Q4 Post-Exit Employed/ Enrolled Q2-4 Post-Exit C&T 84.6% 69.9% 84.7% JVSG 84.1% 65.3% 81.9% AEL 58.3% NA 83.5% VR 88% 56.1% 85.7% 35
Credential Rate DENOMINATOR Exiters without valid Exclusion who: Were in Training excluding OJT or Customized Training; OR Were in Education at or above Secondary School Level at Date of Participation or During Period of Participation What does that mean for English as a Second Language in AEL? We don t know OCTAE never seemed to anticipate it NUMERATOR Within 1 Year of Exit Achieved: Recognized Post-Secondary Credential; or High School Diploma or Equivalent AND was also either: Employed in any of the 4 Quarters Post-Exit employed; or Enrolled in Post-Secondary Education/Training at some point during the Year following Exit Does not appear to consider moving from Secondary to Post- Secondary Enrollment or Enrollment in Integrated Education & Training during Period of Participation OCTAE seems to be trying to limit Numerator 37
Credential Rate (continued) Examples of Recognized Post-Secondary Credentials Associate s, Bachelor s, Graduate or Post-Graduate Degree Occupational Certificates (awarded by either educational institution or independent education or training provider upon successful completion of a course of study & exam) Occupational Licensure (generally awarded by a governmental entity) Occupational Certification (often issued by 3rd party organization based on demonstrated competencies) Good Having a NonYouth Credential measure Bad Lots of Lag One Year Post Exit to Achieve Credential Diploma Requires Employed or in Post-Secondary in Year after Exit Reported to RSA, OCTAE, DOL & LBB (VR, AEL, C&T) 38
Lets Look at Some More Data!!! Program Educational Achievement (C&T) / Achievement of GED (AEL) Credential Rate C&T 80.0% 55.4% (10,440 Exiters) (9,513 Exiters) AEL 52.0% 22.1% VR NA 30.0% C&T has Big Changes Failure to Achieve due to Entered Employment no longer an exclusion OJT & Customized Training no longer put Participant in Denominator AEL now looks at those who reached ASE level rather than just those who took all 4 Parts of the GED test VR doesn t have a good means to track Credentials, especially Post-Exit 39
Measureable Skills Gain DENOMINATOR All PARTICIPANTS without a valid Exclusion in Education or Training during the POP Even if Education/Training not paid for by the program (other than perhaps AEL we re still looking at that) NUMERATOR Achievement of a MSG during the PROGRAM YEAR (July-June) Good Having an interim measure of progress Bad Completely Arbitrary Period Harder to Attain MSG if enrolled in education/training late in the Program Year YOU DON T WORRY ABOUT THAT YOU FOCUS ON NEEDS OF PARTICIPANT WE WILL FIGURE OUT A WAY TO ADJUST TARGETS 40
Measureable Skills Gain Measurable Skills Gains Include: Educational Functioning Level (EFL) Gain Comparison of Pre-Test & Post-Test Scores Exit from Program with enrollment in Post-Secondary Education/Training during the SAME PROGRAM YEAR Attainment of Credential (including Diploma/Equivalent) Transcript/Report Card: Sufficient # of Credit Hours & meeting State s Academic Standards 12 Hours in a Semester or 12 hours over 2 consecutive Semesters Training Milestone Satisfactory or better progress report towards established milestones such as completion of OJT or One Year of Apprenticeship Skills Progression: Successful Completion of Exam required for a particular Occupation Progress in attaining Technical/Occupational Skills as evidenced by trade-related benchmarks such as knowledge-based exams 41
Common or Uncommon? WIOA requires Common Measures OCTAE is proposing different standards for Credential Rate & Measurable Skills Gains using NRS Only want to count non-Diploma/Equivalent Credential if in Integrated Education & Training Only want to count MSG if achieved thru EFL or achievement of Diploma/Equivalent (not other Credential) RSA is proposing different standard for MSG by redefining the Joint PIRL data elements in the RSA- 911 Changes relatively mild (referencing VR Program Policies) but this reduces commonality 42
Effectiveness Serving Employers (ESE) Undefined in Statute Secretaries required to consult with States, Locals, Employers, Stakeholders to develop Secretaries couldn t settle on anything Proposed 3 measures & directed states to select 2 to pilot Good Having National Measures of Effectiveness Serving Employers Applying measures across programs rather than separately Bad DOL & ED also referring to this as Employer Satisfaction Problems with all 3 proposed measures Not clear how they can be applied for LVER even though DOL wants an LVER-specific set of measures 44
ESE #1 Market Penetration % of Employers Served Denominator Total # of Establishments Numerator # of Establishments that received service Employer Information & Support Services Workforce Recruitment Assistance Strategic Planning/Economic Development Activities Untapped Labor Pool Activities Training Services (including Incumbent Worker Training) Rapid Response/Business Downsizing Assistance Problems Which Services are Most Meaningful? Which are Easiest to Get Credit For? 45
ESE #2 Repeat Business % of Employers Served that are Repeat Customers Denominator Total # of Establishments Receiving Service at any time in last 3 years Numerator # of Establishments that received service this year AND at any time in the last 3 years Same Services as Market Penetration Problems Creates a Disincentive to Serve Smaller Employers >93% of Employer Served with 1000+ Employees were repeat customers >75% of Employers Served with 100-249 Employees were repeat customers <60% of Employers Served with <10 Employees were repeat customers Creates a Disincentive to Serve New Employers Denominator includes anyone served this year, even if for 1st time Numerator might not include those served twice in the current year 46
ESE#3 Retention with Same Employer Looks at Employers in Q2 and Q4 to confirm match Denominator Those Employed Q2 Numerator Those Employed Q4 at Same Employer as Q2 What if multiple Employers in Q2 and Q4? Any match counts Good Could have been an amazing measure Would have allowed system to compare retention of our job seekers with all employees at an employer or industry Bad Counts people who don t get new jobs Exit-based instead of based on quarter of new hire Effectively anywhere from 3 Quarters to 5 Quarters of Retention 47
WIOA is Highly Data Intensive States Required to Capture & Report Far More Data Statute Lays Out Data Gathering Requirements Dozens of Specific Data Elements Applies to Formula and Statewide Funded Participants Statute Lays Out how States Report Data Not just Common Measures, Common Reports Statute Prescribes how the Departments will Use Data to Set Targets & Evaluate Performance The Departments have to Develop Statistical Models to Set Targets during Negotiations and RESET Targets at the end of the Year (Down or UP) The Models are built with & run on Data We are not going to be judged against an Arbitrary Target that OCTAE, RSA, or DOL create Success will be based on the results we achieve given WHO we serve & the conditions in which we serve them 49
Data Requirements The Departments Developed a Set of Joint Performance Reporting Specs Participant Individual Record Layout (PIRL) The PIRL contains the basic Elements required to be reported on EVERY Participant 76 Different Elements (though some are conditional) The Departments also Developed Specs for the Common Performance Reports The Departments each ADDED to the Joint PIRL DOL Created a DOL-PIRL that adds several hundred elements RSA incorporated the 76 Joint PIRL elements into an updated RSA-911 OCTAE is largely doing their own thing thru their NRS 50
