Data for Impact: Enhancing Research and Evaluation Capacity Assessment

Research and Evaluation Capacity Assessment Tool and Package (RECAP)

Day 1 Agenda Time Topic Facilitator Welcome and introductions Review slide deck 1. Introduction to RECAP and its components 2. Process overview 3. Structure of assessment tool (domain, subdomains and competencies) 4. Creating the institutional strengthening plan Self-assessment of domains x x (to be selected by the organization completing the self-assessment) Break Self-assessment of domains x x (to be selected by the organization completing the self-assessment) Day 1 Review and Planning for Day 2

Day 2 Agenda Time Topic Facilitator Review day 1 and agenda for day 2 Discuss any updates or revisions for Day 2 session Self-assessment of domains x x (to be selected by the organization completing the self-assessment) Break Self-assessment of domains x x (to be selected by the organization completing the self-assessment) Day 2 review and next steps Items for follow-up, action planning, and next steps

Day 3 Agenda Time Topic Facilitator Review days 1-2 and day 3 agenda Discuss any updates or revisions for Day 3 session Self-Assessment of remaining domains (x x), as needed Break Review scores from days 1 2 (All domains) Preliminary institutional strengthening planning Review assessment results in RECAP workbook and prioritize capacity needs Define strategic responses to identified needs; use the RECAP Resource Guide to propose and map capacity building resources and opportunities to identified needs by domain/subdomain Discuss and define logistical considerations: target group, resources needed, roles/responsibilities, timeline, and monitoring plan Review and next steps Items for follow-up, action planning, and next steps

Background and Overview of Data for Impact (D4I) and RECAP

Data for Impact (D4I) supports countries to Data for Impact (D4I) supports countries to realize the power of data as evidence realize the power of data as evidence Generate strong evidence for program and policy decision making Strengthen individual and organizational capacity to generate and use data Facilitate and enhance the use of data for global health programs and policies

What is RECAP? Research and Evaluation Capacity Assessment Tool and Resource Package Guides organizations to rapidly assess their technical and management capabilities for evaluation and research RECAP assessment results are used to create an actionable plan for institutional strengthening Accompanied by resource guide with free/low-cost resources within each domain

RECAP Package Components Resource Brief Description Provides step-by-step instructions for planning and implementing the capacity assessment and for developing an institutional strengthening plan User Guide Describes domains relevant to research and evaluation, subdomains, and core competencies, with defined performance ideals for each Assessment Tool Supports data entry and visualization and includes suggested discussion questions and probes for each subdomain and a template for planning action steps that map to gaps identified in the assessment Excel Workbook Institutional Strengthening Resource Guide Provides a list of free and low-cost resources mapped to domain

How does it work? The RECAP assessment tool defines critical elements for implementing effective research and evaluation activities in six domains A RECAP assessment workshop brings team members together to discuss capacity within each domain and identify strengths and opportunities RECAP tool scores are then used to design actionable plans for institutional strengthening

The RECAP Assessment Tool

Assessment Framework Six domains aligned to what is needed to undertake an evaluation or other non-clinical research activities Each domain has 2 3 subdomains Each subdomain has multiple core competencies Gender is a cross-cutting theme for the tool The assessment can be adapted based on the organization s needs by focusing on specific domains

Assessment Framework Broad area of expertise, comparable to the phase of the evaluation or study Domain (6) Breaks each domain into parts, i.e., more specialized skills and capacities Subdomain (17) Breaks each subdomain into parts Performance ideal and discussion probes provided for each core competency Core Competencies (67)

The RECAP Assessment Tool: Individual Domains

Domain 1: Research and Evaluation Design Subdomains Core Competencies Needs Identification; Evaluation Questions; Methods Selection; Gender Integration; Research and Evaluation Ethics; Protocol Writing 1.1 Overall Development Quantitative Sites; Quantitative Participants; Qualitative Sites; Qualitative - Participants 1.2 Sampling 1.3 Tools Quantitative; Qualitative

Domain 2: Fieldwork Subdomains Core Competencies 2.1 Team Mobilization Recruitment; Training; Pilot; Relationships; Planning 2.2 Interviewing Techniques Quantitative; Qualitative Individual Interviews; Qualitative Group Facilitation; Gender Integration 2.3 Data Collection Management Gender Integration; Safety; Procedures; Supervision; Quality Assurance

Domain 3: Data Management Subdomains Core Competencies 3.1 Primary Quantitative Setup; Security and Quality; IT Support; Data Entry and Transfer; Analysis Files 3.2 Secondary Quantitative Quality; Comparability; Data Structure Technology; Naming Conventions; Treatment; Datasets 3.3 Qualitative

Domain 4: Data Analysis Subdomains Core Competencies 4.1 Primary Quantitative Analysis Plan; Gender Integration 4.2 Secondary Quantitative Analytic Insightfulness; Analysis Plan; Gender Integration 4.3 Qualitative Organization; Integration; Output; Gender Integration

Domain 5: Information Sharing Subdomains Core Competencies Interpretation; Stakeholder Engagement; Gender Integration 5.1 Translation Strategy; Writing; Visualizations; Production; Facilitation; Action Planning 5.2 Dissemination

Domain 6: Organizational Capacity Subdomains Core Competencies 6.1 Organizational Management 6.2 Human Resources Management Action and Commitment; Organizational Objectives; Relationships; Adaptation and Self-Renewal Personnel Policies; Job Descriptions; Staffing Levels and Retention; Performance Management 6.3 Financial Management Financial Policies and Procedures; Accounting System; Budgeting

Assessment Scoring 1. Each core competency will receive a score 2. Scoring will use the following measurement scale: LEVEL 1 Nascent LEVEL 2 Emerging LEVEL 3 Advancing LEVEL 4 Expert The organization has very limited experience and knowledge. The organization has limited overall experience and knowledge but is aware of gaps and interested in strengthening its capacity and has sought opportunities for institutional strengthening. The organization has demonstrated sufficient knowledge and experience but lacks experience with more complex situations. The organization can problem solve and adapt as necessary and knows how to access resources to fill gaps. The organization demonstrates advanced skills and considerable experience. The organization can anticipate problems and has steps in place to mitigate those problems. This organization is sought out for input.

Assessment Scoring: Purpose Scores will only be used to set priorities for action planning WHAT ARE RECAP SCORES NOT USED FOR? WHAT ARE RECAP SCORES USED FOR? To provide a snapshot of an organization's technical and management capacity to conduct research and evaluation For funders to judge performance or compare organizations To inform discussion around an organization's research and evaluation capacities To measure progress on specific projects or activities To visualize capacity gaps and help prioritize institutional strengthening activities To identify where an organization may need to allocate additional resources

Assessment Scoring: Subjective Process Performance ideal Facilitator guides group towards score consensus Core Discussion notes competency score Dedicated notetaker documents process and evidence in workbook Evidence

Assessment Scoring (continued) Everyone will be invited to propose scores for each core competency Discussion will be held to try and reach consensus If consensus cannot be reached, there will be a simple majority vote The notetaker will capture comments during the discussion and record final assessment scores Note: the assessment format can be adjusted to break a large group up into smaller groups to assess individual domains. In this case, each domain should have a dedicated notetaker.

Planning for Institutional Strengthening

Developing an Institutional Strengthening Plan 1. Review assessment results using the RECAP assessment workbook 2. Prioritize capacity needs Prioritization will be decided by consensus If consensus cannot be achieved, the decision will be made by a simple majority vote 3. Design an institutional strengthening plan 4. Plan for follow-up

Review Assessment Results Using the RECAP assessment workbook, consider: What are the organization s strengths? Where are the opportunities for the organization to improve? What area(s) is the organization interested in improving?

Prioritize Capacity Needs Factors to consider include, but are not limited to: Upcoming activities Funding Strategic positioning Professional development for staff Feasibility

Design an Institutional Strengthening Plan 1. 2. What is the identified need (at core competency level)? What capacity strengthening activities will be implemented to address those needs? Who will participate in the intervention? Are funds and/or other resources needed to implement the intervention? Who is responsible for the day-to-day management of the intervention? What is the timeline? How will you know the intervention happened and if it was successful? 3. 4. 5. 6. 7.

Plan for Follow-Up Once the institutional strengthening plan has been developed, determine follow-up actions for implementation: 1. Identify who will finalize and manage the institutional strengthening plan 2. Determine how and when the organization will review progress against the plan 3. Determine the timing of the next assessment

This presentation was produced with the support of the United States Agency for International Development (USAID) under the terms of the Data for Impact (D4I) associate award 7200AA18LA00008, which is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Palladium International, LLC; ICF Macro, Inc.; John Snow, Inc.; and Tulane University. The views expressed in this publication do not necessarily reflect the views of USAID or the United States government. www.data4impactproject.org www.data4impactproject.org