Quantification of LEADER/CLLD Contributions and Secondary Effects

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Quantification of LEADER/CLLD contributions and measuring their secondary effects Originally developed by the European Evaluation Helpdesk for Rural Development during the CAP programming period 2014 -2022 in 2020

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Disclaimer The content of this training material has been prepared by the European Evaluation Helpdesk for Rural Development in the programming period 2014-2022. The material aims to support capacity building activities in relation to the monitoring and evaluation . For any use outside of its original intent, or any modifications made to the content of the presentation, prior written permission must be obtained from the European Evaluation Helpdesk for the CAP. Commercial use of this material is strictly prohibited. Please contact: evaluation@eucapnetwork.eu for any questions in this respect. The European Evaluation Helpdesk for the CAP makes no claims, promises, or guarantees about the accuracy, completeness, or adequacy of the content of the presentation and expressly disclaims liability for any errors or omissions in such content.

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Learning objectives Reminder of the evaluation frame for the quantification of LEADER/CLLD contributions and their secondary effects Review and resolve doubts on the evaluation steps for the quantification of LEADER/CLLD Acquire a good understanding of key issues that may affect the quantification of LEADER/CLLD contributions and secondary effects Learn new tools to facilitate the quantification of LEADER/CLLD contributions and secondary effects

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Content What are we analysing Reminder of the frame and of the evaluation steps Key facts and issues encountered for quantification of LEADER/CLLD contributions and measuring their secondary effects Solutions identified by RDPs Where to find further examples and guidance Discussion

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Reminder of the frame What you need to know about secondary contributions of LEADER/CLLD Secondary contributions of LEADER/CLLD are contributions of operations implemented via CLLD strategies to any other FA than FA 6B Measures which are programmed under LEADER/CLLD can have secondary contributions to other focus areas than 6B and may be taken into consideration in the assessment of RDP results. This can show a more complete picture of the achievements of LEADER/CLLD Secondary contributions might be quantified when calculating values of results indicators or assessed qualitatively The quantification of secondary contributions is not legally required, but it is considered as good practice

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Reminder of the evaluation steps (1) Step 1: Check the consistency of the common evaluation elements with the LEADER/CLLD intervention logic One CEQ associated with FA 6B (CEQ17): To what extent have RDP interventions supported local development in rural areas? This CEQ has six proposed judgement criteria Which are linked to three common result/target indicators In addition, all other CEQs linked to other FAs to which LEADER/CLLD contributes should also be taken into consideration (secondary contributions) Step 2: Develop programme-specific evaluation elements (e.g. judgment criteria and indicators) For capturing programme-specific effects of LEADER/CLLD or To break down the rather general CEQs For developing a better understanding of the local level effects and their contributions to the RDP level To make the LEADER/CLLD intervention logic more explicit and to show the importance of the approach and its added value

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Reminder of the evaluation steps (2) Step 3: Identify evaluation approaches and methods for LEADER/CLLD Quantitative methods In order to capture the primary and secondary contributions of the LEADER/CLLD measure to the achievement of the objectives of the FAs. Qualitative methods When to apply? For the initial analysis and the triangulation of quantitative findings. A sampling of beneficiaries which carry on operations under the CLLD strategy, and which have flagged secondary contributions to specific FAs. Data collected via sampling are used to calculate contributions of LEADER/CLLD operations to values of indicators linked to the FAs, where contributions are expected. Use the tools presented in the Guidelines: Assessment of RDP Results: how to prepare for reporting on evaluation in 2017, PART III, Annex 10, e.g. interviews with stakeholders and beneficiaries, focus groups, case studies. How? Further informat ion Guidelines Assessment of RDP Results: how to prepare for reporting on evaluation in 2017 , PART III, Annex 11. Guidelines Assessment of RDP Results: how to prepare for reporting on evaluation in 2017 , PART III, Annex 10.

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Reminder of the evaluation steps (3) Step 4: Collect data and information Output and result/target indicators for FA 6B Result/target indicators for the FAs to which LEADER/CLLD shows secondary contributions Common context indicators (these are pertinent for several FAs) Additional and programme-specific indicators as well as impact indicators related to LEADER/CLLD Additional quantitative information needed for answering the evaluation questions Step 5: Analyse information and answer evaluation questions

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Discussion: Evaluation steps Are there any doubts on the evaluation steps? Do you have or foresee any issues in relation to these steps? Discuss if and how such issues can be resolved

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N What are the key facts? The analysis of the evaluations conducted so far shows . Low level of quantification of LEADER/CLLD contributions Low level of reporting on secondary effects of LEADER/CLLD

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N What are the key issues for quantifying LEADER/CLLD contributions and measuring secondary effects? Existing common target/result indicators insufficient for capturing the full effect of LEADER/CLLD, including to FAs other than FA 6B Improving the evaluation capacity at the LAG level Governance issues

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Key issues Issue: Existing common target/result indicators insufficient for capturing the full effect of LEADER/CLLD, including to FAs other than FA 6B Do you share this issue? What have you done to address it?

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Proposed tools and approaches to collect additional information to complement existing common indicators and answer CEQs Additional judgement criteria / indicators (link to Toolbox) Questionnaire to LAGs for collecting additional information (link to Toolbox) Assessing value added of LEADER/CLLD to capture LEADER/CLLD effects on local development in rural areas Although not compulsory, the collection of information on other elements of LEADER/CLLD may cover aspects related to the actual delivery method of LEADER/CLLD (e.g. social capital, implementation mechanism and governance) Using additional evaluation questions, judgement criteria and indicators Applying innovative methods (e.g. social networking approach)

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Key issues Issue: There is a need to improve the evaluation capacity at the LAG level LAGs are at the centre of information collection for assessing the effects of LEADER/CLLD but several LAGs have limited knowledge or familiarity with data collection concepts and methods interpretation of data and the rationale for data collection Specific areas where LAGs need to improve their capacities how to communicate with beneficiaries on data requirements and key evaluation concepts reduce the risk of errors like the overestimation of expected results or double counting of certain data items

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Key issues . Issue: There is a need to improve the evaluation capacity at the LAG level Do you share this issue? What have you done to address it?

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Proposed content for capacity building of LAGs Go through the following tool (link)

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Key issues Issue: There have been unclear and un-systematic flows of information amongst the different stakeholders Do you share this issue? What have you done to address it?

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Example of flows of information amongst the different stakeholders (Finland) Scores the application, recommends decision Recommends to make a payment Several stakeholders are involved along the implementation process, from the application period to the monitoring and evaluation, notably the beneficiaries, LAGs, MA and PA Need for a clear description of who does what at each step of the process in terms of data and information collection and processing LAG, MA, PA MANAGING AUTHORITY PAYING AGENCY LAG LAG EU + National Municipal Validates the monitoring indicators Receives the application, ensures completeness Receives the application Makes the payments Receives and handles payment claims Run reports on applications, payments, monitoring Ensures compliance, makes decision Decision Payment Monitoring Application Makes a payment claim with required attachments Informs of decision and conditions for support Receives payment Completes the monitoring indicators Sends the electronic application in the Hyrra system using secure credentials BENEFICIARY

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Relevant guidance Assessment of RDP Results: How to Prepare for Reporting on Evaluation in 2017: https://eu-cap-network.ec.europa.eu/publications/assessment-rdp-results-how- prepare-reporting-evaluation-2017_en Evaluation of LEADER/CLLD Explains how to assess the primary and secondary contributions of LEADER/CLLD towards the rural development focus area: https://eu- cap-network.ec.europa.eu/publications/evaluation-leaderclld_en Working document for the Rural Development Committee, Rural Development Annual Implementation Report, Monitoring Tables 2014-2020: https://agriculture.ec.europa.eu/document/download/5a2ada56-3505-48b4-aff1- 5871e9172c9c_en?filename=output-indicator-fiches-pillar-ii_en.pdf Evaluating CLLD Handbook for LAGs and FLAGs: https://www.rndr.ro/documente/Evaluare_CLLD_-_GALuri_si_FLAGuri.pdf

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Questions for discussion 1. Developing your own additional judgement criteria and indicators Based on the examples (Tool 1 given as Handout 1): What additional JC and indicators that you consider useful to capture the full effects of LEADER/CLLD in your programme can be developed?

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Discussion 2a. Analysing the evaluation capacity of LAGs Analyse the evaluation capacity of LAGs in your RDP: identify areas of improvement What options can be considered for improving the evaluation capacity of LAGs?

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Discussion 2b. Developing training content for LAGs 1. If you identified training as a means to improve the evaluation capacity of LAGs, what potential types of training are possible? 2. What content of training is needed? 3. What changes could be made so that the Tool (handout) is usable in your RDP?

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Exercise example 2b: Optional training content for your LAGs Content Objective Priority Planning the evaluation Preparing the evaluation Other Other

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Discussion 3. Identifying the main challenges affecting clear information flows amongst stakeholders and how they can be overcome Departing from the information flow in your context: Whare is the information flow lie the main challenges? What could be potential improvements?

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N LAG, MA, PA Describe action Describe action MANAGING AUTHORITY PAYING AGENCY LAG LAG EU + National Describe action Municipal Describe action Describe action Describe action Describe action Describe action Describe action Decision Payment Monitoring Application Describe action Describe action Describe action Describe action Describe action BENEFICIARY

E U C A P N E T W O R K E U C A P N E T W O R K P R E S E N T A T I O N Get in touch European Evaluation Helpdesk for the CAP evaluation@eucapnetwork.eu Rue Belliard 12 Brussels, Belgium Tel. +32 2 808 10 24 https://eu-cap- network.ec.europa.eu/support/evaluation_en