Evaluating SDGs Follow-Up and Review Processes by Stefano D. Errico

Evaluation in follow up and review processes of the SDGs Stefano D Errico July 2017 Author name Date Stefano D Errico Future vision and current practice GEN Geneva Evaluation Network 31st October 2017 Zenda Ofir & Stefano D Errico EVALSDGs 1

VNRs and Evaluation Stefano D Errico July 2017 The EVALSDGs papers: an international collaboration The 6th and 8thEVALSDGS briefing papers are co-authored by 15 evaluation professionals. The papers have analysed all 65 Reviews submitted to the HLPF in 2016, and 2017. The full series can be downloaded at: https://www.iied.org/effective-evaluation- for-sustainable-development-goals 2

Voluntary National Reviews Stefano D Errico July 2017 VNR, one piece of the jigsaw Every country has to present a Voluntary National Review to the High Level Political Forum However, the VNR to the HLPF is only one piece of the jigsaw - Agenda 2030 encourages all countries to conduct regular reviews, assessments and evaluations to accelerate implementation of the SDGs 3

Voluntary National Reviews Stefano D Errico July 2017 Voluntary National Reviews, the other pieces of the jigsaw The real change will happen only when reviews will be used locally to accelerate the implementation of the SDGs. Data should be co-produced and used by different stakeholders at the national level. But data and evidence are very difficult to be handled. Without proper processes they could lead to inconsistent or conflicting findings and judgements. Source: https://grassrootsfund.org/groups/gala-global- awareness-local-action 4

Evaluation and VNRs Stefano D Errico July 2017 The role of evaluation in VNRs Evaluation can serve Agenda2030 by facilitating the co-generation of value judgements. These must be based on varied sources of rigorous evidence. Equally, they must be based on processes that enable different stakeholders to interpret data and unpack the complexity of the real world. 5

Evaluation and VNRs Stefano D Errico July 2017 The role of evaluators in VNRs Evaluators have a responsibility to challenge monopoliesof problem definition, of issue formulation, of data control, of information utilisation. For example, the interaction of the SDGs often implies clashes between conservation policies and economic growth. Evaluators must mediate different interests in the processes of value definition. 6

Agenda 2030 and complexity Stefano D Errico July 2017 Digesting complexity Agenda 2030 is a massive step forward because it recognises the inter-relationships between human, economic development and the environment. However, in its complexity, it can become a huge burden for national governments. 7

Agenda 2030 and complexity Stefano D Errico July 2017 Digesting complexity We need to find ways to digest the Agenda by developing simple messages around its key principles: - Country ownership - Universality - Sustainability - Partnerships, and - No one left behind 8

Stefano DErrico July 2017 Assessment of the role of evaluation in the 65 VNRs submitted between 2016 (22) and 2017 (43) 9

Use of evaluation and M&E in VNRs Stefano D Errico July 2017 The role of evaluation In almost all national reports, it was not entirely clear how the proposed M&E system would support implementation of the 2030 Agenda. There appears to be very little awareness about just what evaluation is and how it could be used to support the 2030 Agenda Most of the Reviews lack detail about how evaluation could be used to inform the reporting processes 10

Country ownership Stefano D Errico July 2017 Reporting countries are taking ownership Almost all 64 reporting countries implemented a governance structure at the top level of government. Countries with Voluntary National Reviews 2016, 2017 and 2018 M&E responsibilities are usually located in: Presidential/prime minister offices National planning ministry, The environmental ministry The budget/finance ministry. Ministry of foreign affairs 11

Universality Stefano D Errico July 2017 Agenda 2030 is everybody s concern! Though not yet Some countries are reporting more than once in four years: Benin, Qatar, Uruguay, Colombia, Egypt, Mexico, Switzerland and Togo (three times). While, some important states, including Russia, South Africa, UK and the USA, have not yet announced any plans for reporting on their VNRs. Reflection on evaluation as a tool for the 2030 Agenda is poor in the reporting countries from Europe and Asia, apart from some exceptions 12

Sustainability Stefano D Errico July 2017 Adapting M&E systems without forgetting sustainability Voluntary reports and national M&E systems should assess the interactions and interconnectedness of the 17 goals in 2017 almost all countries (41 out of 43) have conducted or are planning to conduct a gap analysis of their current monitoring systems to assess how they can be adapted to the SDGs. 13

Partnerships Stefano D Errico July 2017 Partnerships to oversee the agenda at national level Many of the 64 countries have implemented Special Committees to consult, guide or oversee their SDGs national processes. These institutions often include civil society organisations and people from the private sector. 14

No one left behind Stefano D Errico July 2017 Which data to understand equity and equality? Only 10 countries out of 64 have mentioned the inclusion of qualitative data in their reports. Lack of other sources of evidence is a challenge for assessing no-one left behind because the more data get disaggregated the more it becomes unreliable Made by Andrew a facebook user in Canada 15

Good evaluative practice from VNRs Stefano D Errico July 2017 Examples from Asia In Nepal, the National Parliamentarian Forum has developed National Evaluation Policy and an action-plan for 2016-20. Indonesia has presented clear evaluation guidelines for different moments of the programs (ex-ante, mid-term, ex-post) that mention the DAC criteria. Afghanistanputs significant emphasison multi-stakeholder approaches to review the SDGs. The Maldives intends to use data triangulation by cross-referencing qualitative and quantitative data in reporting against the SDGs. 16

Good evaluative practice from VNRs Stefano D Errico July 2017 Examples from Europe Czech Republic stresses that monitoring must be accompanied by evaluation to address the complexity of the SDGs. Cyprus mentions a few casesof evaluation in the education sector that have been used for programme adaptation and improvement. Portugalstresses thatIndicators per se cannot be the driving force for development. The achievements they indicate must be framed in some form of a causal or contributory relationship to SDGs targets. In Sweden, the municipality of Malmo has adopted the SDGs as their own goal, and in two pilots they have assessed benefits of the municipality s long-term investments by using the three sustainability dimensions. 17

Good evaluative practice from VNRs Stefano D Errico July 2017 Examples from Africa In Kenya, a review of the MDGs has informed the new Medium-Term Plan and the Road Map to implement the SDGs. The VNR has been validated by different stakeholders, and they are willing to tap into potential of new and non-traditional data sources to complement statistics. Botswana highlighted the need of multi-sectoral involvement in the development of the reporting and national indicators system which provides good basis for participatory evaluation. Ethiopia has conducted a national review on their performance against the MDGs which has fed into the preparation of their national agenda for addressing the SDGs. 18

Good evaluative practice from VNRs Stefano D Errico July 2017 Examples from Latin America Costa Rica acknowledges the importance of evaluation in terms of policy review, effectiveness, efficiency, and accountability; Guatemala is planning to conduct periodic reviews that should be carried out at the national, regional and local levels. Belize s VNR utilized evaluation findings in the assessment of the current situation in relation to the cash transfer programmes, gender equality, and fisheries/marine management. Panama has established a multidimensional approach to measure welfare, poverty and discrimination/segregation which requires the adoption/adaptation of policies, monitoring and evaluation mechanisms. 19

Recommendations Stefano D Errico July 2017 UNEG to provide leadership, review and contribute to revision of the existing UN guidelines for reporting for the VNRs.The additions/revisions should clarify: what evaluation is and how it differs from monitoring; how evaluation integrates and builds on monitoring and statistics; how qualitative information can be used in national follow-up and review ; what specific functions evaluation could have in national M&E systems for the SDGs. 20

Recommendations Stefano D Errico July 2017 Countries submitting VNR in 2018 to engage with evaluators, regional and national Voluntary Organisations for Professional Evaluation (VOPEs) early in the process of developing their report, as well as for implementation of evaluations. They should also consider developing national evaluation policies to harmonise practices from across different levels of governments and authorities. 21

Recommendations Stefano D Errico July 2017 The evaluation community to engage in VNRs processes at both global and national levels: At global level, Evalpartners, IOCE, IDEAS, and evaluation offices of inter-governmental institutions to support UNEG in providing guidance. At national level, VOPEs should engage with their government to start high-level dialogues about the role of evaluation in national follow-up and report of the SDGs. They should also support government to better understand the function that evaluation could play in these processes. UNEG, IOCE, and IDEAS should support the effort of VOPE made at country level. 22

Recommendations Stefano D Errico July 2017 Bilateral and multilateral funders to support countries to further develop their capacities in terms of: o Adopting and sustaining participatory processes; o Conducting and using evaluation, which includes development of analytical skills. 23

Recommendations Stefano D Errico July 2017 In terms of content VNRs should: Make clear that macro indicators have limits. VNRs should also be clear on evaluation s role as an essential complement to indicators when judging the effects of policies and attributing these to implemented measures. Complement the focus on quantitative indicators with more qualitative methodologies. This is an opportunity to include different data sources and tools for analysing and reporting progress. strongly focus on building an interlinked picture at national and local levels because the SDGS are strongly interlinked. 24

Stefano DErrico July 2017 Recommendations In terms of content VNRs should: Focus on three main aspects: a) Assessing causes and effects in complex and interdependent systems by looking at what has changed, for whom, under what conditions, how and why b) Paying attention to vulnerable populations andt he environment c) Assessing the roles and responsibilities of different stakeholders through a co-responsibility lens , i.e. taking an approach that expects sustainable development strategies to be co-designed and co-implemented by all stakeholders (public, private, NGOs and citizens). 25