The Annual Implementation Report (AIR) Guidelines

The Annual Implementation Report (AIR) provides key information on the implementation of a program, financial data, indicators, and targets. It outlines activities related to the evaluation plan and program priorities. The report structure includes chapters on program information, progress in implementation, performance issues, and measures taken.

• Evaluation
• Annual Report
• Program Implementation
• Guidelines
• Indicators

hitchings_i Follow

Uploaded on Feb 24, 2025 | 0 Views

Download Presentation

Please find below an Image/Link to download the presentation.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.

You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.

The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.

Presentation Transcript

1. Getting prepared for reporting on evaluation in the AIR submitted in 2017 EvaluationWORKS! 2016

2. Content What does the legal framework specify? What is the Annual Implementation Report (AIR)? Which is the task and which guidance is provided? What are the major issues addressed in the guidelines? How should the evaluation of 2017 be o conducted? o reported? SFC template of the AIR 2017 - point 7 2

3. Legal framework 3

4. What does the legal framework specify? Common M&E system (CMES) Demonstrate achievements, to better targeted support, support a common learning AIR 2017:common and PS indicators and quantified targets, changes in values of result indicators Improve the design and implementation of programmes, assess effectiveness and efficiency towards EU2020 targets M&E system Common indicators Article 14, Annex IV, V, VI and VII of Commission implementing regulation (EU) No 808/2014 Article 50.2, 54.1, 54.3, 56.3 of Regulation (EU) No 1303/2013 Article 67, 68, 69, 70, 75.3 of Regulation (EU) No 1305/2013 AIR 2017 Evaluation during PP: assessment of effectiveness and efficiency based on the evaluation plan Common indicators Electronic information system Common evaluation questions for RD Technical support for CMES (guidance) Evaluation is carried out by internal or external experts independent of programme authorities Structure and content of the AIR incl.point 7 4 4

5. Annual Implementation Report 5

6. What is the content of the Annual Implementation Report (AIR)? AIR shall set out key information on implementation of the programme and its priorities by reference to financial data, common and programme specific indicators and quantified target values (Article 50 of Regulation (EU) No 1303/2013) AIR shall include information interalia on financial commitments and expenditures by measures and a summary of activities undertaken in relation to the evaluation plan (Article 75 of Regulation (EU) No 1305/2013) 6

7. What is the structure of the AIR? Chapter 1 Key information about the programme and its priorities Chapter 2 the progress in implementing the evaluation plan Chapter 3 issues which affect the performance of the programme and measures taken Chapter 4 - steps taken to implement TA and programme publicity Chapter 5 - actions taken to fill ex ante conditionality Chapter 6 description of implementation of sub-programmes Chapter 7 - assessment of information and progress towards achieving objectives of the programme Chapter 8 implementation of actions to take into account principles set out in Articles 6, 7 and 8 of regulation (EU) no1305/2013 Chapter 9 progress made in ensuring the integrated approach Chapter 10 report on implementation of FI The content of the AIR (Annex VII of Regulation (EU) No 808/2014) 7

8. Guidance provided 8

9. What is the task and which guidance is provided? CMES AND EXISTING GUIDANCE Programming and target setting Evaluation plan Indicator fiches: Target indicator fiches for Pillar II Complementary result indicator fiches for Pillar II WD: Common evaluation questions for RDPs 2014-2020 Monitoring and operations database for Pillar II TASK In the AIR submitted in 2017, provide the information resulting from evaluation: Quantification of programme achievements, in particular through the assessment of complementary result indicators and answering relevant evaluation questions + GUIDELINES Assessment of RDP Results: how to report on evaluation in 2017 9

10. Guidelines Assessment of RDP results 10

11. How do the guidelines Assessment of RDP results help to conduct the evaluation and report on it? Download Here! 11

12. Outline of the guidelines Part II (mainly for evaluators) Part III Annexes Introduction (for everyone) Part I (mainly for MA) Provides Example of SFC template Explains Linking RDP intervention logic to RD policy and evaluation elements (structuring) Explains Common reporting requirements, reporting on RDP specific elements Introduces The evaluation of RDP achievements in 2017 Practical tools for preparing, implementation and reporting the evaluation in 2017 Discuss Discusses Data collection (observing) evaluation methods (analysing, Conclusion and recommendations (judging) What is important to prepare evaluation, how to ensure its quality and how to disseminate and communicate evaluation results Examples of qualitative assessment of secondary contributions 12

13. PART I focusing, managing and reporting on evaluation in 2017 What is reported in the AIR 2017 and which information will the SFC template of the AIR 2017 for point 7 ask for? What are the links between information on evaluation in AIR 2017 and other reporting requirements? How to ensure the quality of reporting on evaluation in the SFC template? How to disseminate, communicate and follow up on evaluation findings? 13 13

14. What needs to be reported on evaluation in 2017? Quantification of RDP achievements with result indicators, taking into consideration the primary and secondary contributions of operations to focus areas Validation of values of target indicators which have been set up at the time of programme design Answers to common evaluation questions (no. 1 21) and programme specific evaluation questions, and Conclusions and recommendations for improvement of programme design and implementation 14

15. What are the specific issues addressed in PART I? - 1 Use the qualitative assessment and theory of change (CH 1.3) What to report in case of low or no up take of the RDP in 2017? Use the entire population of beneficiaries in calculation, beneficiaries in calculation Use the entire population of How to report on RDP achievements in case of small programmes? Build up the M&E system at early stage of programme implementation.This serves also for reporting in 2019 and ex post evaluation (CH 5) How to link reporting in 2017 with the overall reporting requirements on RD in 2014-2020? 15

16. What are the specific issues addressed in PART I? - 2 Complementary result indicators (CRI) R2 change in Agricultural output per AWU (FA 2A) R13 increase in efficiency of water use in agriculture in RDP supported projects (FA 5A) R14 increase in efficiency of energy use in agriculture and food- processing in RDP supported projects (FA 5B) R15 renewable energy produced from supported projects (FA 5C) R18 reduced emissions of methane and nitrous oxide (FA 5D) R19 - reduced ammonia emissions (FA 5D) WD: Complementary Result Indicator fiches for Pillar II, June 2014 CRI R2 compulsory for AIR 2017, for each CRI whenever possible (data on beneficiaries) (CH 7) In which case to provide net values of indicators besides gross values? 16

17. PART I: Specific issues addressed (3) Definition of programme specific elements to address specific issues in MS (CH 5.2.1) SFC template point 7 table for reporting on programme specific evaluation questions How to define and report on programme specific elements? How and when to ensure the quality of evaluation? Invest in planning and preparing the evaluation including the Terms of Reference Ensure ongoing quality control in line with quality criteria throughout the evaluation How to fill the SFC template? Example in Annex 1 of the guidelines 17

18. PART I - transition arrangements How to report on transitions arrangements in the AIR 2017? 2007 2013 discontinuing measures paid with 2014-2020 budget 2007-2013 measures paid with 2007-2013 budget 2007 2013 continuing measures paid with 2014-2020 budget are outside the targets/outputs. They are reported in table 6 of Chapter 7 if the 2014- 2020 budget is not significant. Otherwise, the RDP authorities should develop additional indicator and report it also in table 4 of Chapter 7 are included in all sections of Chapter 7 (no distinction 2007-2013 and 2014-2020 measures even only 1 operation is partially paid with 2014- 2020 budget, it counts for 2014-2020 target value) are not reported at all in the AIR submitted in 2017 18

19. PART II the system to assess RDP achievements Guide stakeholders through the path in accomplishing the evaluation task for 2017 (and beyond): Set up the system for answering the relevant evaluation questions (1 21) with the means of indicators Conduct the evaluation Answer evaluation questions 19

20. PART II setting up the system to answer the evaluation questions and assessing the RDP achievements Preparing Conducting Re-visit the RDP intervention logic Observing Link intervention logic to evaluation elements Analysing Structuring Set up a consistent evaluation approach Judging Establish the evidence for evaluation 20

21. PART III Annexes Proposed SFC template in the AIR submitted in 2017, point 7 for CEQ 4 and filled example CMES elements and their use in evaluation Content of the ToR and how the guidelines could help to design it Overview of reporting requirements on the RDP implementation and evaluation in 2014-2020 Tool for qualitative appraisal of RDP interventions effects (measure, FA and RD priorities Other useful documents, e.g. checklists 21

22. PART III, Annex 11: Fiches for answering the CEQ 1 - 21 Each fiche contains: Intervention logic of the FA: primary and secondary contributions Evaluation elements linked to the EQ: Data needs and sources, Methods used to calculate indicator values Challenges and proposed solutions Download Here! 22

23. Content of the fiche - example Evaluation question: Title of the evaluation question and its related Focus Area How to answer the evaluation question (Key steps to answer the evaluation question) List of measures contrbuting to the FA: Primarily programmed measures Measures with secondary contributions Consistency between CEQ, judgement criteria and result indicators Judgment criteria Result indicators: common and additional (if needed) Other indicators used to answer the CEQ, e.g. output and common context indicators Qualitative information Data needs and data sources for calculation of indicators Indicators Data needs Data sources Common Indicators Additional indicators Proposed methodology to collect and analyse data and information: Quantitative Qualitative Challenges/Risks/Issues: Proposed solutions Answer to common evaluation question Conclusions Recommendations 23

24. How should the evaluation results be conducted and reported in 2017? 24

25. PART II: Setting up the system to answer EQs and assess RDP achievements Preparing Conducting Re-visit the RDP intervention logic Observing Link intervention logic to evaluation elements Analysing Structuring Set up a consistent evaluation approach Judging Establish the evidence for evaluation 25

26. Major issues in preparing evaluation Identification of primary and secondary contributions of operations to FA and synergies Preparing Re-visit the RDP intervention logic Use of CMES elements Link intervention logic to evaluation elements Define the additional and programme specific elements Checking consistency between intervention logic and evaluation elements Taking into account the specifities of LEADER, NRN and TA 8

27. Major issues in structuring the evaluation Choosing adequate evaluation approaches and their adequateness for RDP assessment Taking into account the specificities of LEADER, NRN and TA Evaluation approaches in case of low/no up take and for small RDPs Approaches to assessment of secondary contributions Data needs for assessment of results Structuring Set up the consistent evaluation approach Approaches to address the data gaps Establish the evidence for evaluation Operations database including data for evaluation 27

28. Major issues in conducting the evaluation Conducting Quantitative and qualitative data Assessment of RDP results taking in consideration primary and secondary contributions Observing Net assessment of RDP results, when possible Analysing Answering CEQ 1 21 and programme specific EQ Judging Drafting evidence based conclusions and recommendations 10

29. Who are the recipients of the information on evaluation and in which format? Recipient European Commission Responsible MA Format AIR 2017 via SFC template (System for fund management in the EU) Detailed evaluation report Programme administration MA and PA Rural development stakeholders in MS: MC, professional associations, beneficiaries etc. Evaluators MA and evaluators Summary of evaluation report Scientific society Evaluators Detailed evaluation report General public MA Short summary of evaluation finding focused on major issues 29

30. SFC template of the AIR 2017 - point 7 30

31. Reporting on evaluation findings in AIR 2017 SFC template point 7 EU - tool to collect the information on evaluation in MS in a structured and comparable manner: quantification of programme achievements through the assessment of result indicators and answers to evaluation questions The SFC template for the AIR 2017 contains 21 tables for CEQ and 2 tables for programme specific evaluation question (to be multiplied by MS) 31

32. Common evaluation questions (CEQ) (808/2014, Art. 14.1c) To be answered and reported in AIR 2017, 2019, ex post To be answered and reported in AIR 2019, ex post CEQ related to other specific aspects (19-21) CEQ related to EU level objectives (22-30) Focus area related CEQ (1-18) Linked to the FA objectives Linked to EU level objectives Linked to specific policy aspects, such as synergies among priorities and focus areas, TA and NRN Capture the achievements of Focus area related objectives Capture the contribution of the programme towards the overall policy objectives Answered with the means of additional and programme specific indicators Answered with the means of result indicators (and additional indicators/information when necessary) Answered with the means of common impact indicators, context indicators and result indicators (and additional indicators/information) 32

33. Structure of AIR 2017 SFC template point 7 - CEQ Common evaluation question (number and title) 1. List of measures contributing to the FA: primary and secondary programmed measures 2. Link between judgment criteria, result indicators used to answer CEQ 3. Methods applied 4. Quantitative values of indicators and data sources 5. Problems encountered influencing the validity and reliability of evaluation findings 6. Answer to evaluation question 7. Conclusions and recommendations 33

34. Common evaluation question (title and number): No 4 To what extent have RDP interventions contributed to improving the economic performance, restructuring and modernization of supported farms in particular through increasing their market participation and agricultural diversification? 1. List of measures contributing to the FA 2A: Primarily programmed measures Measures programmed under the other FA which show secondary contributions to FA 2. Link between judgment criteria, common and additional result indicators used to answer the CEQ Judgment criteria Common result indicator Additional result indicator Agriculture output per AWU of supported agriculture holdings has increased Farms have been modernised and restructured R1/T4 - % of agriculture holdings with RDP support for investments in restructuring and modernisation CRI R2 - Change in agriculture output/AWU in RDP supported projects 3. Methods (quantitative, qualitative) 4. Quantitative values of indicators and data collection Indicator Absolute value Ratio value Calculated gross value Calculated net value Data and information sources Primary Seconda ry contrib., if relevant LEADER/ Primary Seconda ry contrib., including Total value Primary Seconda ry contrib., including Total value Contrib. CLLD contrib. Contrib. Contrib. LEADER/ LEADER/ CLLD contrib. CLLD contrib. Output Common result R1/T4(%) Common complementary result R2 Additional 5. Problems encountered influencing the validity and reliability of evaluation findings 6. Answer to CEQ 7. Conclusions and recommendations 34 Conclusions Recommendations

35. Part III Annex 11: List of measures contributing to the Focus Area 6. Social inclusion, poverty reduction and economic development in rural areas 5. Resource efficiency and shift towards a low carbon and climate resilient economy 4. Restoring preserving and enhancing ecosystems 2. Competitiveness of all types of agriculture and farm viability 3. Food chain organization and risk management 6A 2A 3A agri- food chains 5B diversifi cation Competi tiveness energy efficiency M6.4 support for investments in creation and development of agriculture activities M4.2 support for investments in processing/mark eting and development of agri-products M6.1 busines start ups for non-agriculture activities Measures/sub-measures programmed under the other FAs which show secondary contributions to FA 1A contributions to FA 2A Measures/sub-measures programmed under the other FAs which show secondary Primarily programmed measures/sub-measures directly linked to the given FA directly linked to the given FA Primarily programmed measures/sub-measures 35

36. Part III Annex 11: Consistency btw. CEQ, judgment criteria, result indicators Judgement criteria Specify the EQ and define the success of RD interventions Evaluation questions Reflect achievements against RD objectives Indicators Measure the success of RD interventions Mutual Consistency between judgment criteria and indicators must be ensured 36

37. SFC template point 7: 3. Methods applied Target indicators values are obtained directly from operation database Applied for complementary result indicators and additional indicators, if used to answer the CEQ Quantitative Rational Description Challenges In case the same method was used for several indicators, the list of these indicators should be provided Qualitative Method is described only ones in the chapter 7 37

38. SFC template point 7: 4. Quantitative values of indicators and data sources All indicators used to answer the CEQ are listed: Output, Common results, Additional and Common context Values: Absolute (output from operation database, some context from EU or national statistics) Ratio (most of target indicators from operation database) Calculated values for complementary result indicators and additional indicators) Net/gross depended on uptake, data sources and method Compulsory R2 change in agriculture output/AWU 38

39. SFC template point 7: 5. Problems encountered influencing the validity and reliability of evaluation findings Explain problems faced by the evaluator during the assessment, which might have threatened the reliability and validity of evaluation findings e.g.: Programme up take Data and information availability Timing and coordination issues, etc. 39

40. SFC template point 7: 6. Answer to evaluation question... ....can be provided based on: Expert judgment (e.g. using the theory of change) Qualitative research (e.g. asking stakeholders) Quantitative and qualitative evidence (collecting data and information from beneficiaries and non-beneficiaries) taking in consideration what would happen without the programme. Robust answer can only be based on the evidence! 40

41. SFC template point 7: 7. Conclusions and recommendations Answer to evaluation question Recommen dations Evaluation findings Conclusions Not every conclusions from evaluation are the basis for recommendations 41

42. Programme specific evaluation questions (PSEQ) ...are formulated in Member States, for: Programe specific focus area: Programme specific evaluation topic: National (territorial) priorities are not covered by existing FA and CEQs Specific evaluation topics are included in the evaluation plan (e.g. RDP delivery, administration etc.) CEQs do not capture the full range of achievements of objectives of the programme or focus area Programme shows potential indirect, secondary, unexpected or negative effects and imply new evaluation topic 42

43. Structure of AIR 2017 SFC template point 7 PSEQ related to FA Programme specific evaluation question - title 1. Rational 2. List of measures contributing to the FA (this step is excluded for PSEQ related to evaluatioin topic) 3. Link between judgment criteria, indicators used to answer PSEQ 4. Methods applied 5. Quantitative values of indicators and data sources 6. Problems encountered influencing the validity and reliability of evaluation findings 7. Answer to evaluation question 8. Conclusions and recommendations 43

44. Programme specific evaluation question (title and number): 1. Rational of the programme specific FA 2. Intervention logic of the programme specific FA: Primarily programmed measures Measures programmed under the other FA which show secondary contributions to FA 3. Link between judgment criteria, common and/or programme specific result indicators used to answer the CEQ Judgment criteria Common result indicator Programme specific result indicator 4. Methods (quantitative, qualitative) 5. Quantitative values of indicators and data collection Indicator Absolute value Ratio value Primary Contrib. Calculated gross value Primary Contrib. ary contrib., includin g LEADE R/ CLLD contrib. Calculated net value Primary Contrib. ary contrib., includin g LEADE R/ CLLD contrib. Data and information sources Second ary contrib., if relevant LEADER / CLLD contrib. Second Total value Second Total value Common output Programme specific output Common result Programme specific result 6. Problems encountered influencing the validity and reliability of evaluation findings 7. Answer to CEQ 44 8. Conclusions and recommendations Conclusions Recommendations

45. Useful links Guidelines for evaluation of rural development programmes in 2014-2020 https://enrd.ec.europa.eu/evaluation/european-evaluation-helpdesk- rural-development/evaluation-helpdesks-publications/guidance_en Member States questions and answers http://enrd.ec.europa.eu/evaluation/publications/working-document- evaluation-related-queries_en Guidelines for evaluation of rural development programmes in 2007-2013 https://enrd.ec.europa.eu/enrd-static/evaluation/library/evaluation- helpdesk-publications/en/evaluation-helpdesk-publications_en.html 45

46. Thank you for your attention! European Evaluation Helpdesk for Rural Development Boulevard Saint Michel 77-79 B-1040 Brussels Tel. +32 2 7375130 E-mail info@ruralevaluation.eu http://enrd.ec.europa.eu/evaluation 46