Evaluation Activities for Weatherization Programs
In this presentation, Jackie Berger discusses the importance of evaluating weatherization programs, outlining key objectives such as assessing potential improvements, measuring program impacts, and meeting regulatory requirements. The session emphasizes the need to evaluate program effectiveness, efficiency, equity, and client satisfaction to enhance energy efficiency and affordability. Practical recommendations are provided to guide evaluation processes and drive program enhancements.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Evaluating Weatherization Programs Jackie Berger NASCSP Conference September 2015
APPRISE Nonprofit Research Institute Mission Analyze data and information to assess and improve public programs Research Areas Energy Efficiency and Energy Affordability Clients Federal Government (DOE, HHS), State Government, Utility Companies, Nonprofits 2
Session Outline Evaluation Activities Why Evaluate? Recommendations Description Activities Information Objectives Purpose Frequency Options Other Reasons Other Outputs Considerations 3
Evaluation Objectives Assess Potential Improvements Measure Program Impacts Meet Regulatory Requirements 5
Assess Potential Program Improvements Goals Is the program meeting its goals? Efficiency Same impacts at a lower cost? Effectiveness Increased impacts? Equity Geographic, renters/owners Targeting High users, vulnerable groups, other Client Satisfaction 6
Measure Program Impacts Energy Bill Affordability Health, Safety, and Comfort Energy Usage Economic Impacts Environmental Impacts Cost/Benefit Ratio 7
Meet Regulatory Requirements State Public Utility Commission Other Regulatory 8
Program Design Cycle Design Repeat! Research and Evaluation Implementation 9
Program Design Cycle Prioritize program goals Identify program models Select program design Design Identify low-income households Target high usage (or other) households Delivery energy education and measures Implementation Assess program design and delivery Analyze program impacts Recommend program modifications Evaluation 10
Process Evaluation Questions How is the program designed? Documentation review Interviews with program design and management team How is the program implemented? Interviews with program managers and implementers Analysis of program data On-site observation Surveys with program participants Why is it working or not working? Synthesis of all evaluation data 11
Impact Evaluation Measurement Program data analysis Billing data analysis Change in energy usage Impact on client behavior Customer surveys Billing data analysis Improvement in energy affordability Customer surveys Billing data analysis 12
Other Impacts Assessment Home Comfort Client Surveys Client Surveys Home Testing Results Client Health Economic Development Program Expenditures Job Creation Multipliers Environmental Impacts Usage Reduction Environmental Multipliers 13
Evaluation Activities Select activities based on specific evaluation questions Evaluation plan Background research Spec and procedures review Payment impact analysis Public use data analysis Program data analysis On-site observation Provider interviews/survey Participant survey Usage impact analysis Economic impacts Environmental impacts Cost benefit analysis Final report 15
WAP Evaluation Activities Evaluation Activity Period 2008 ARRA Topic Notes Eligible Population Survey of Program Production Stats and Procedures Public: CPS, ACS, RECS, AHS, SIPP, NHIS Funding, Jobs, Procedures Funding, Jobs, Procedures Details on service delivery procedures Main heating fuel and client characteristics Diagnostics, measures, and costs Data Analysis X All State All Agencies X X X X Agency Sample X X Service Delivery Characteristics (Agency Sample) Listing of All Jobs X X Detail for Sampled Jobs X X 16
WAP Evaluation Activities Period Evaluation Activity Natural Gas and Electric Fuel Oil Metering Notes 2008 ARRA Topic 60 months of data for sampled single family and mobile homes Conducted in 2010-2011. Used for 2008 and ARRA analyses Fuel oil, natural gas, and electricity for large multifamily buildings and units Energy Savings Benefits / Cost of Installed Measures X X Usage Impacts X Large Multifamily NYC National Cost- Effectiveness Data Analysis X X 17
WAP Evaluation Activities Evaluation Activity Indoor Air Quality Field Process Period Notes 2008 ARRA Topic X Pre/Post Air Quality 2010-2011 X Observation of Service Delivery 2011 High and Low Savers from 2008 - Data Collected in 2012 and 2013 Performance X Field Studies Impacts of wx on Air Conditioning Usage - Data Collected in 2013 AC Pilot X Impacts of ventilation on radon levels - Data Collected in 2013 Ventilation X 18
WAP Evaluation Activities Evaluation Activity Baseline Satisfaction Follow-Up Baseline Follow-Up Period ARRA Notes 2008 Topic X X X X X Summer 2011 Fall 2011 / Spring 2012 Summer 2013 Fall 2011 / Spring 2012 Summer 2013 Used state-level energy impact data to model emissions reductions and benefits Client Surveys Wx Staff Surveys Emissions Impacts Data Analysis X X Non-Energy Benefits Analysis of measures, client surveys, and emissions impacts Data Analysis X X 19
Evaluation Activities: Evaluation Plan 20
Evaluation Plan Description Evaluation Goals Key Information Objectives Evaluation Activities Specific Research to be Conducted Methodology Sample Selection, Sample Sizes, Analysis Techniques Research Timeline Inputs and Outputs Deliverables Progress Reports, Frequency of Memos/Reports, Presentation 21
Evaluation Plan Purpose Make Decisions Agree on Expectations Guide Research 22
Evaluation Plan Options Level of Detail Approach Methodology Research Options Additional Research Needed for Clarification Change Research Plan Barriers or Findings Flexibility 23
Evaluation Plan Outputs Indicator and Data Model Description of Research Activities Logic Model Deliverables and Dates 24
Evaluation Plan Outputs NH REACH Furnace Cleaning Project Logic Model Immediate Outcomes Intermediate Outcomes Final Outcomes Assumptions Activities Low-income clients: o Want to reduce their energy bills o Will want the furnace cleaning service o Will be willing to participate in follow-up Local oil vendors and home furnace cleaning companies: o Will want to participate o Will be willing to collect and provide data on each furnace cleaning job o Will provide the service in a timely manner Furnace CT&E will reduce low-income energy consumption. Furnace CT&E will furnish health and safety benefits. Clean furnaces and boilers in 5100 low- income homes in Northern New Hampshire. Identify heating systems needing repair or replacement. Improved furnace operation. 1) Reduced energy consumption and costs. 2) Improved health and safety. Improved energy insecurity index. Red flagged systems are replaced. 25
Evaluation Plan Outputs NH REACH Furnace Cleaning Project Indicator and Data Model Outcomes Performance Indicators Weather normalized, control group adjusted fuel consumption Reduced energy costsControl group adjusted fuel costs Data Reduced energy consumption Data collected from fuel dealers. Client surveys. Pre and post health and safety tests on furnaces. Energy insecurity scale. Improved health and safety Control group adjusted client ratings of comfort and health Improved energy insecurity index Control group adjusted energy insecurity scale 26
Evaluation Activities: Background Research 27
Background Research Description Program Manager and Staff Interviews Document Review Program Plans Management Program Rules Schedulers Statistics Training Materials Auditors Budgets Installers Contracts QC Inspectors Internal monitoring and reviews 28
Background Research Purpose Develop an understanding of program details Program goals Targeting Delivery procedures Challenges and barriers Quality control Data sources and availability Performance measures Program design Eligibility Measures and services Education Service delivery contractors 29
Background Research Purpose Confirm understanding of program Interviews Fill in gaps about operations and procedures Obtain information on program resources Explore ideas for program modification 30
Background Research Purpose What are the program s goals? How are the goals prioritized? Reduce Energy Usage Improve Energy Affordability Reduce Greenhouse Gas Emissions Improve Client Comfort Create Jobs 31
Background Research Purpose How do program managers assess performance? What performance measures are in use? % of Elderly, Disabled, Young Children Households Served % kWh/Therm Reduce in Usage Number of Homes Treated Average Spending Per Home Reduction in Energy Usage 32
Background Research Options Documents to Review Data to Assess Interviews to Conduct 33
Background Research Outputs Program Summary Evaluation Focus Refined Understanding Documentation Goals, Design, Delivery, Challenges Additional Research Questions 34
Background Research Outputs Example: documentation of program goals Production Goals Targeting Goals Saving Goals Serve 2,000 homes in the calendar year 15% gas savings for gas heaters 10% electric savings for electric heaters Serve 25% renters Serve 75% vulnerable 35
Background Research Outputs Documentation of program goals WAP Goals Stated in Code of Federal Regulations (CFR) 10 CFR 440.1 Increase Energy Efficiency Of dwellings owned or occupied by low-income persons Reduce Total Energy Expenditures Improve Health and Safety Especially elderly, disabled, children High energy users High energy burden 36
Background Research Outputs Example of initial evaluation recommendations Targeting Data Coordination Prioritize customers based on usage Track coordination with other programs Provide contractors with incentives to coordinate Collect important data Enter into tablet and upload to database 37
Evaluation Activities: Specifications and Procedures Review 38
Spec and Procedures Review Purpose Assess Procedures & Forms Manuals and Technical Procedures Education Specifications Data Collection Forms Work Orders 39
Spec and Procedures Review Purpose Determine potential effectiveness of: Measure Selection Measure Installation Energy Education 40
Spec and Procedures Review Purpose How comprehensive are the energy services described in the manual? Does the manual describe procedures that can result in high quality service delivery? How do the program procedures compare to low-income energy efficiency programs around the country? Should appliance replacement thresholds be updated? Should additional measures be added to the program? What changes in program procedures would have the greatest impact on the effectiveness and efficiency of service delivery? 41
Spec and Procedures Review Outputs Example of Modifications to Documentation Re-organize program manual Create an electronic version that can be searched and has links to worksheets Provide quick reference for testing procedures Use terminology consistently Combine data collection forms Require use of a work order that provides clear direction on the work to be done 42
Spec and Procedures Review Outputs Example of Technical Recommendations Weather normalize usage prior to assessing appropriate spending level Disaggregate usage by end use (for example heating versus water heating) to better align spending with savings opportunities Review cost-effectiveness calculations in audit decision tree to reflect current estimates of costs and savings (example: refrigerators) Conduct blower door testing periodically during air sealing to find leaks and determine effectiveness Duct sealing in basements should be focused on safety and comfort 43
Spec and Procedures Review Outputs Example of Education Recommendations Education should be provided during the audit The homeowner or primary adult should be home Customer profile should collect potentially large opportunities for saving Dehumidifiers Second refrigerators Windows left open in winter Action form should prioritize actions by potential for saving List top 3-5 actions with estimated $ savings 44
Evaluation Activities: Public Data Analysis 45
Public Data Analysis Description Statewide Analysis Regional Analysis American Community Survey Data Energy Usage Characteristics Demographics 46
Public Data Analysis Purpose Document demographic characteristics of eligible households Analyze energy needs of eligible households Understand the population the program serves 47
Public Data Analysis Options Geographic Detail State Regional Variables Analyzed Number of Eligible Households Main Heating Fuel Languages Spoken at Home Energy Bills Energy Burden Vulnerable Households 48
Public Data Analysis Outputs Varied from 14% in one county to 36% in others Income Eligibility 26% in state were income eligible 61% of low- income in state spoke English at home Varied from 32% in one county to 87% in another Language 62% of low- income in state were program eligible Varied from 30% in one county to 79% in another Program Eligibility 49
Evaluation Activities: Program Data Analysis 50
