Maximizing ROI through Program Evaluation & Financial Analysis

Show Me the Money: Moving From Impact to ROI Patti Phillips CPLP, Ph.D. patti@roiinstitute.net

Objectives Isolate the effects of a program Convert benefits to monetary value Identify the intangible benefits Tabulate the fully-loaded costs Calculate ROI

The ROI calculation is simple. Program Benefits Program Costs BCR = Net Program Benefits Program Costs X 100 ROI =

Try it! $750,000 $425,000 BCR = $750,000 - $425,000 $425,000 X 100 ROI =

Try it! $750,000 $425,000 = 1.76:1 BCR = $750,000 - $425,000 $425,000 = 76% X 100 ROI =

What Makes a Good ROI? 1. Set the value at the same level as other investments 15% Set slightly above other investments 25% Set at break even 0% Set at client expectations While we don t strive for a negative ROI, negative is not always bad! 2. 3. 4.

Levels of Levels of Evaluation Evaluation Measurement Focus Measurement Focus Typical Measures Typical Measures 0. Inputs and Indicators The input into the project in terms of scope, volume, efficiencies, costs Participants, Hours, Costs, Timing Relevance, Importance, Usefulness, Appropriateness, Intent to use, Motivation to take action 1. Reaction & Perceived Value Reaction to the project or program, including the perceived value Learning to use the content and materials, including the confidence to use what was learned Skills, Knowledge, Capacity, Competencies, Confidence, Contacts 2. Learning Extent of use, Task completion, Frequency of use, Actions completed, Success with use, Barriers to use, Enablers to use Use of content and materials in the work environment, including progress with actual items and implementation 3. Application & Implementation The consequences of the use of the content and materials expressed as business impact measures Productivity, Revenue, Quality, Time, Efficiency, Customer Satisfaction, Employee Engagement 4. Business Impact Benefit-Cost Ratio (BCR), ROI%, Payback Period Comparison of monetary benefits from program to program costs 5. ROI

ROI Methodology Stage 1 Stage 2 Data Collection Evaluation Planning Develop Evaluation Plans and Baseline Data Collect Data After Solution Implementation Collect Data During Solution Implementation Develop Objectives Of Solution Level 1 Level 3 Level 2 Level 4

Capture Costs Of Solution Stage 4 Communicate Results Stage 3 Data Analysis Convert Data to Monetary Value Calculate the Return On Investment Isolate the Effects of Solution Generate Impact Study Report Level 5 Identify Intangibles Measures Intangible Benefits

Operating Standards 1. Report the complete story 2. Conserve resources 3. Use the most credible sources 4. Choose the most conservative alternatives 5. Isolate the effects of the program 6. No data no improvement 7. Adjust estimates for error 8. Throw out the extreme and unsupported claims 9. Use first year benefits for short-term programs 10.Include fully-loaded costs 11.Report intangible benefits 12.Communicate results to all stakeholders

Inputs Reaction Learning Application Isolate the Effects of the Program Impact ROI Intangible Benefits

Only a select few programs are evaluated to ROI. Consider the following: Life cycle of the program Linkage of program to operational goals and issues Importance of program to strategic objectives Top administrator interest in the evaluation Cost of the program Visibility of the program Size of target audience Investment of time required

OBJECTIVES LEAD THE EVALUATION An Exercise

To what level would you evaluate the following objective? 100 customers will request more information from the website within two weeks of the promotional briefing. 3

To what level would you evaluate the following objective? Given five steps to communicate effectively, participants will rank them in order of application. 2

To what level would you evaluate the following objective? Participants will implement the change process within 3 months after completing the course. 3

To what level would you evaluate the following objective? Within two weeks of the conference, exhibitors will follow up with ten qualified prospects who visited their booth. 3

Given the objective below, how many months after the program will you collect Level 3 data? Three months after the workshop, participants will report a 1 hour per week time savings due to shorter meetings. 4

To what level would you evaluate the following objective? Six months after the program, grievances will go down on average 7 per month due to the program. 4

To what level would you evaluate the following objective? Given benefits of $250,000 and costs of $150,000, the ROI will be 67% . 5

To what level would you evaluate the following objective? Program results will show a BCR of 1.67:1. 5

Collecting Post Program Data Level 3 Level 4 Follow-Up Surveys Follow-Up Questionnaires Observation On the Job Interviews with Participants Follow-Up Focus Groups Program Assignments Action Planning Performance Contracting Project Follow-Up Session Performance Monitoring

Factors to consider When selecting methods, consider: Time required for participants Time required for supervisors Costs of methods Amount of disruption Accuracy Utility Culture/Philosophy When selecting sources of data, consider: Participants Supervisors Direct reports Peer groups Internal staff External sources Organizational records When determining timing, consider: Availability of data Ideal time for behavior change/application Ideal time for business impact Convenience of data collection Constraints on data collection

Methods to Isolate Program Effects Use of a control group arrangement Trend line analysis of performance data Use of forecasting methods of performance data Participant s estimate of program impact (percent) Supervisor s estimate of program impact (percent) Manager s estimate of program impact Use of expert/previous studies Calculate/estimate the impact of other factors Customer input

Classic Control Group Design What is the difference in improvement? Experimental Group Pre-Measure Program Post-Measure Control Group Post-Measure Pre-Measure

Post-Program Only Design What is the difference? Experimental Group Program Post-Measure Control Group Post-Measure

Example of Trend Line Analysis 1.85% Pre Program Average CPI Program Conducted 2% 1.45% Projected Average ERROR RATE .7% Post Program Average 1% J F M A M J J A S O N D J MONTHS

Example of Estimation Monthly increase in credit card accounts: 175 (fact) Contributing Factors Consensus Impact (%) 32% Average Confidence (%) Sales Training 83% Incentives 41% 87% Management Reinforcement Market Fluctuations 14% 62% 11% 91% Other _________ 2% 100% 91%

Example of Estimations Influence Fact % Contrib. Est. Impact Confidence Adjusted Impact Sales Training 175 32% 56 83% 46.48 Fact: 175 New Credit Card Accounts 65.52 % Contribution: 32% 56 Est. Impact: 56 Uncertainty 17% 46.48 Margin of Error: +/- 9.52

Remember the ROI Calculation? Program Benefits Program Costs BCR = Net Program Benefits Program Costs X 100 ROI =

Data are converted by: Converting output to contribution standard value Converting the cost of quality standard value Converting employee s time standard value Using historical costs Using internal and external experts Using data from external databases Linking with other measures Using participants estimates Using supervisors and managers estimates Using staff estimates

5 Steps to Convert Impact to Money Step 1: Focus on a unit of measure Step 2: Determine the value (V) of each unit Step 3: Calculate the change in performance ( P) Step 4: Determine the annual amount of the change (A P) Step 5: Calculate the total annual value of the improvement (A P x V)

Example Using Internal Experts Step 1: One grievance Step 2: V = $6,500 (from Director of nursing and HR Experts) Step 3: P = average of 7 out of 10 grievances prevented per month Step 4: Annual P = Step 5: A P x V =

Example Using Internal Experts Step 1: One grievance Step 2: V = $6,500 (from Director of nursing and HR Experts) Step 3: P = average of 7 out of 10 grievances prevented per month Step 4: Annual P = 84 Step 5: A P x V = $546,000

Example Using Standard Values Step 1: One sale Step 2: V = 30% profit margin Step 3: P = $20,000 revenue per month Step 4: Annual P = Step 5: A P x V =

Example Using Standard Values Step 1: One sale Step 2: V = 30% profit margin Step 3: P = $20,000 revenue per month Step 4: Annual P = $240,000 Step 5: A P x V = $240,000 x .30 = $72,000

Fully-Loaded Cost Profile Assessment Costs (Prorated) Development Costs (Prorated) Program Materials Instructor/Facilitator Costs Facilities Costs Travel/Lodging/Meals Participant Salaries and Benefits Administrative/Overhead Costs Evaluation Costs

Financial Program Skills Based Pay Systems Business Impact Employee Turnover Staffing Level Customer and Job Satisfaction Product Sales Cross Selling ROI = 258%

Telecommunications Program All-Inclusive Workforce Program (AIW) Business Impact Attrition Rate Employee Satisfaction Communication Cooperation & Teamwork Diversity Mix ROI = 163%

Health Systems Program Sexual Harassment Prevention Workshop Business Impact Turnover Reduction Complaint Reduction Job Satisfaction Absenteeism Stress Reduction Recruiting ROI = 1,051%

Technology Program Customer Service Training ROI = -85% Business Impact (Target) Customer Complaint Escalation

Why do people like ROI? The people who like ROI do so because the ROI Methodology: Generates a balanced set of measures Employs a step-by-step process Bridges evaluation disciplines Balances research and reality Offers a flexible evaluation solution Provides a credible approach to measurement and evaluation

Next Steps Assess your readiness for ROI Identify stakeholders and their data needs Determine the purpose of your evaluation practice Identify programs suitable for ROI Develop capability in the ROI Methodology

Success with measurement begins with your worldview If you are measuring, you are estimating. There are no absolutes. Be able to explain what you did, how you did it, and why you did it that way. Sometimes the crowd knows best. Process without standards is no process. A statistic is an estimate of what probably is maybe.

The rest is merely a balancing act. Benefits Costs 47

48

Take Your Learning to the Next Level Measuring the Success of Organization Development, Measuring the Success of Sales Training, and Survey Basics are available in the ASTD Store in Salon GHI and at www.store.astd.org

Thank you! Patti Phillips Patti@roiinstitute.net Visit us online: www.roiinstitute.net Connect with us on: twitter.png YouTube.png Google_Plus_logo.png Linkedin.png facebook_logo_detail.gif 50

Feedback Counts! Your feedback helps ASTD continue to provide top-notch educational programs that help you stay on top of a changing profession. Evaluation forms for this session are available via the mobile app and at the following link: www.astdconference.org/attendees.