Evaluating Partnerships: Importance, Development, and Assessment
Discover the significance of evaluating partnerships and learn how to develop and assess successful collaborative efforts to achieve desired outcomes. Explore the nature and purpose of partnerships, considerations for partnership evaluation, and steps to engage stakeholders effectively in the evaluation process. Gain insights into matching evaluation tools with partnership development stages and accessing resources for informed decision-making.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Evaluating Partnerships: Why, When and How February 17, 2015 B. J. Tatro, PhD ASTDD, bj@bjtatro.com Presented in Collaboration with the Centers for Disease Control and Prevention 1
General Reminders This webinar will be recorded and archived on the ASTDD website (including speaker notes) Questions will be addressed after the speakers are finished. Please type your question into the chatbox that will appear at the end of the webinar and then click on the bubble to the right of where you type your question to send it to the moderator Please respond to the polling questions at the conclusion of the webinar 2
This presentation was supported by Cooperative Agreement 5U58DP004919-02 from CDC, Division of Oral Health. Its contents are solely the responsibility of the authors and do not necessarily represent the official views of CDC 3
Desired Results State Oral Health Programs (SOHPs) have increased awareness of the importance of evaluating their partnerships and of engaging their partners in the evaluation process SOHPs have increased awareness of the importance of matching their partnership evaluation questions and tools to their partnership s stage of development SOHPs are familiar with resources available to them to inform their planning, implementation, and use of partnership evaluation 4
Some Preliminary Thoughts on Partnerships What are they? Nature of the partnership Purpose of the partnership 5
Some Preliminary Thoughts on Partnerships Why do we need them? What does it take to develop, maintain, and grow a successful partnership? 6
Some Preliminary Thoughts on Evaluating Partnerships Why evaluate partnerships? How do we know if we are ready to evaluate a partnership? How do we approach the design and implementation of a partnership evaluation? 7
Step 1: Engage Stakeholders Identify who should be involved in the evaluation Specify the roles they will play Think about how to engage these stakeholders throughout the evaluation process 8
Step 2: Describe the Partnership Develop your logic model 9
State of Hawaii Purpose = Develop & coordinate partnerships to improve oral health Inputs Activities Outputs Outcomes Hawaii Oral Health Team (staff) 1. Identify priorities & establish, enhance, & maintain a collaborative relationship with internal DOH partners, with priority focus on CDC-funded chronic disease programs, CSHCN, & MCH a. Contacts identified, documented Short Term Ongoing communication & collaborative action Internal Partners Family Health Services Division Programs: Office of Primary Care & Rural Health CSHCN MCH (including MIECHV, Parenting Support, Adolescent & Youth, etc.) WIC Other DOH: Chronic Disease Management & Control Branch (Asthma, Cancer, Diabetes, Heart/Stroke, Obesity, Tobacco) Office of Planning, Policy & Program Development (Preventive Health & Health Services Block Grant) Developmental Disabilities b. Communication established, documented c. Information shared, documented Increased awareness of the relevance of oral health to partners target populations 2. Identify priorities & establish, enhance, & maintain a collaborative relationship with external partners, with priority focus on FQHCs & RHCs d. Shared interests & common goals identified, documented Increased awareness of the benefits of prevention, early identification, & treatment to overall health e. Benefits identified, documented For both internal & external partners: a. Identify contacts b. Conduct 1:1 outreach c. Become familiar with each other s mission & activities d. Identify shared interests & common goals e. Identify potential benefits of working together f. Identify options for collaborative efforts g. Make decision about whether & how to collaborate h. Establish partnership objectives & develop work plans i. Implement work plans j. Evaluate collaborative strategies f. Options identified, documented Intermediate Term Oral health addressed in partners burden documents if applicable, activities, messaging, plans, policies, &/or practices g. Decision made, documented h. Objectives established & work plans developed Resources devoted to oral health expanded/ leveraged i. Work plans implemented, documented j. Evaluation conducted, documented External Partners State Medicaid Office Hawaii Federally Qualified Health Centers (FQHC) & Rural Health Clinics (RHC) Hawaii Chapter of the American Academy of Pediatrics Insurance providers Legislators Community representatives Longer Term Increased number & commitment of oral health champions Increased attention to prevention, early identification, & treatment Increased number of children have dental sealants CDC Funding & TA Impact Reduced dental caries, oral cancer, periodontal disease ASTDD TA Improved health equity Improved overall health 10
Step 2: Describe the Partnership (continued) Identify what you need and what you have 11
Partnership Membership Assessment Tool from Fundamentals of Evaluation Partnerships/Evaluation Guide (National Health Disease & Stroke Prevention Program) Want Have Roles Want Have Skills/Expertise Want Have Representation Partnership roles Data analysis, worksites State Emergency Services Leader Data analysis, healthcare State Obesity Program Committee leader Reviewer, medical content State Diabetes Program Task leader Writer State Tobacco Program Meeting planner Advocate for stroke State Epidemiology Meeting facilitator Advocate for heart disease State Office of Minority Health Strategic planner Legislative advocate Hospital Association Communications Medical expert Primary Care Association Training Cardiologist State Legislature/Policy Makers Financial support Neurologist Schools (as worksites) Content reviewer Healthcare quality improvement Community health clinics Budget management Nursing Private insurers Spokesperson Pharmacy Medicaid/Medicare Funder Media communications Prevention Research Center Champion, healthcare Workplace wellness Chambers of Commerce Champion, public health State policy change Unions Champion, worksites Community policy change Business coalition on health Strategic implementer Training for healthcare State American Heart Association Resource linker connection to groups with influence or resources Evaluation Disparate groups (race/ethnicity, geographic, gender. SES. Etc.) Marketing 12
Step 2: Describe the Partnership (continued) Identify the stage of development of your partnership Formation Building Maintenance Engage stakeholders 13
Step 3: Focus the Evaluation Design Develop and prioritize evaluation questions that are matched with your partnership s stage of development Engage stakeholders 14
Step 4: Gather Credible Evidence Complete your evaluation plan: Indicators Data Sources (tools) Methods of data collection Frequency of data collection Study design Implementation roles, responsibilities, and timelines Engage stakeholders 15
State of Minnesota Indicator/ Performance Measure Evaluation Question Method Data Sources Frequency Responsibility How were the opportunities identified and leveraged to integrate oral health efforts with other programs that address related chronic disease/risk factors? # of meetings, campaigns, summits organized Meeting notes, summit reports review Meeting notes and summit reports Program coordinator/ evaluator Annually How many partnerships were developed with a focus on prevention interventions in a year? # of partnerships that focus on prevention intervention and that provided a letter of support or MOA Program reports review Program coordinator/ evaluator Annually Program reports Partnership database Letters of support How was the integration of oral health into overall health encouraged/institutionalized? # of meetings with other chronic disease partners Meeting notes and reports review Meeting notes and reports Program coordinator/ evaluator Biannually How was the documentation provided for collaboration with partners outside state health department? # of meetings, meeting notes, emails, letters indicating agreement to collaborate Meeting notes and reports review Meeting notes and reports Program coordinator/ evaluator Biannually What are types of activities are partners involved in? # of specific tasks accomplished Meeting notes and reports review Meeting notes and reports Program coordinator/ evaluator Biannually What are the reasons for long term sustained partnerships? List of reasons for long term sustained partnerships Once in a grant period Program evaluator Partners survey Survey questionnaire What are the challenges partners feel that hinder intended progress? List of reasons for long term sustained partnerships Once in a grant period Program evaluator Partners survey Survey questionnaire 16
Indicator/ Performance Measure Evaluation Question Method Data Sources Frequency Responsibility How does the coalition focus on the following? Oral health infrastructure development and sustainability State oral health plan revision and implementation School-based/-linked sealant programs Community water fluoridation efforts Coalition initiatives in the areas of Oral health infrastructure development and sustainability State oral health plan revision School-based/-linked sealant programs Community water fluoridation Meeting notes and reports review Meeting notes and reports Program coordinator/ evaluator Biannually What sectors of CDC coalition diversity framework do coalition members represent? # of members from each sector of CDC coalition framework Coalition member roster Coalition member roster Program Coordinator On-going # of members from each sector attending coalition meetings Member group is identified on the membership application Stakeholder database Data Coordinator What were the steps taken to facilitate institutionalization and sustainability of coalition? 501c3 status, written vision/mission statement, priorities /plans/strategies Consistency of membership Review bylaws and coalition work plan Coalition members/ program coordinator By-laws and work plan Annually Which programs and policy changes have resulted from the coalition efforts? # of policy changes as a result of coalition efforts Review policy documents Coalition members/ program coordinator Policy document Annually List of program initiated Review reports Coalition reports Annually 17
Indicator/ Performance Measure Evaluation Question Method Data Sources Frequency Responsibility What types of communication are being used by partners? Types of communications by partners Coalition secretary/ program coordinator Review reports Progress reports On going Member survey and review of agenda, minutes and reports Member survey and Coalition agendas, minutes and reports On going # of member that feel the communication is adequate Use of website for communications Coalition website updated quarterly # of partner involved in initiating new activities outlined in state oral health plan On going Coalition website How do collaborative partners focus on oral health issues and prevention strategies as outlined in the state oral health plan? Review program progress reports Program progress reports Program coordinator/ evaluator Quarterly # of partner participating in activities outlined in state oral health plan What are the forms of documentation of commitment or support available from stakeholders e.g. delineating accountability and clear lines of responsibility for activities? Letters of support from stakeholders Review coalition reports Program coordinator Annually Coalition reports Description of roles and responsibilities of partners for activities Review of coalition manual Coalition manual How did partnerships influence oral health policies, practices and systems? List of policies, practices and systems influence Review report and survey results Annual reports and partners survey Once in the grant period Program evaluator 18
Step 5: Justify Conclusions Ensure accuracy of data and data entry Analyze the data Make sense of what you have learned Develop recommendations Engage stakeholders 19
Step 6: Ensure Use and Share Lessons Learned Develop a communication plan Communicate results Incorporate into ongoing planning Engage stakeholders 20
Resources ASTDD website (slides/speaker notes from this webinar will also be posted on the ASTDD website) and B. J. Tatro (ASTDD evaluation consultant) bj@bjtatro.com There are also a variety of resources related to oral health collaborations on the site. Collaboration: What Makes It Work (2 Monsey (Fieldstone Alliance, 2001) Includes The Wilder Collaboration Factors Inventory. This publication includes a review of the literature on factors influencing successful collaboration and The Wilder Collaboration Factors Inventory TOOL. Collaborative Effectiveness Assessment Activity (Prevention Institute, 2004) This TOOL is designed to stimulate thinking about some elements of effective collaboration. Evaluating Partnerships to Prevent and Manage Chronic Disease, F. Butterfoss, Preventing Chronic Disease, April 2009, Vol. 6, No. 2 This article describes how to develop a comprehensive evaluation strategy based on partnership theory, select outcome indicators, choose methods and tools, and use results for accountability and improving effectiveness. nded), P. Mattessich, M. Murray-Close, & B. 21
Resources Evaluation Guide: Fundamentals of Evaluating Partnerships, National Heart Disease & Stroke Prevention Program (Centers for Disease Control and Prevention, 2008) This is one of a series of evaluation technical assistance tools. The guides supplement the CDC program evaluation framework. This particular guide provides information on how to implement the framework when evaluating partnerships at various levels (annual assessment, basic evaluation, and enhanced evaluation). Appendices include examples of evaluation questions, a variety of TOOLS, and other helpful information. Framework for Evaluation (CDC, 2012) CDC provides a six-step framework for conducting program evaluation. CDC also offers guides for developing logic models and developing evaluation plans; see Developing and Using a Logic Model and Developing an Effective Evaluation Plan Guide to Evaluating Collective Impact, H. Preskill, M. Parkhurst, & J. Splansky Juster (FSG, Collective Impact Forum, 2014) In addition to three guides, this site links the user to a knowledge exchange related to collective impact. 22
Resources Handbook on Planning, Evaluating and Improving Interagency Collaboration in State Oral Health Programs (ASTDD, 2012) The handbook and companion workbook provide a framework for evaluating collaboration between state oral health programs and other entities. A variety of TOOLS, as well as oral health examples, are provided. Measuring Collaboration Among Grant Partners, B. Frey, J. Lohmeier, S. Lee, & N. Tollefson, American Journal of Evaluation, 2006, 27, 383 This TOOL can be used to assess levels of collaboration ranging from networking to collaboration for each partner. PARTNER This is an online social network analysis tool designed to collect, analyze, and interpret data to improve collaboration within community networks. 23
Resources Partnership Evaluation: Guidebook and Resources, National Center for Chronic Disease Prevention & Health Promotion, Division of Nutrition, Physical Activity & Obesity (Centers for Disease Control and Prevention, 2011) This is one of a series of evaluation technical assistance tools. The guides supplement the CDC program evaluation framework. This particular guide provides information on how to implement the framework when evaluating partnerships. Appendices include a variety of TOOLS and other helpful information. Partnership Self-Assessment Tool (Center for the Advancement of Collaborative Strategies in Health, 2002) This TOOL can be used to assess how well a collaborative process is working and to identify specific areas for improvement. It is designed for partnerships of five or more that have been in existence for at least six months. Tools for Partnership Evaluation (CDC, 2012) This coffee break webinar was presented by S. Ladd, CDC Division for Heart Disease and Stroke Prevention Applied Research and Evaluation Branch. It covers three types of TOOLS evaluability assessment, membership and roles, and process and functioning. 24
Thank You! Thank you to all who provided information for and input into this webinar, including: Harder+Company, for sharing a matrix of partnership/collaboration evaluation tools that they researched and prepared John Shoemaker, for sharing a variety of partnership/collaboration tools and other resources The many CDC-funded states that provided copies of their partnership/collaboration evaluation logic models and evaluation plans, especially those who allowed us to draw examples from their work And thanks to all of you who participated today! 25
Polling Questions Are you planning to do a partnership evaluation within the next few years? Yes, No, Not Sure Have you developed your partnership evaluation plan yet? Yes, No, Not Applicable (no study planned) What could ASTDD do to help you plan/conduct your partnership evaluation? 26
