Quest for Better Impact Evaluation: The QuIP Journey
This content delves into the QuIP quest for improved impact evaluation in development, focusing on attributing social impact credibly and cost-effectively. It covers key aspects like motivations, evidence gaps, methodological options, the QuIP story, and BSDR QuIP studies across various sectors and countries.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Attributing development impact: the QuIP quest for better impact evaluation James Copestake j.g.copestake@bath.ac.uk IDS 20 March 2018
1. Motivation: starting premises Public investors should be able to demonstrate social impact credibly and cost-effectively. Intended beneficiaries should have a strong and credible voice in providing such feedback. Evidence should be timely enough to influence future actions, building on what is known already. It should reflect the complexity and diversity of context, interventions and outcomes. It could be based on better methodological standards.
Filling an evidence gap? Donors and investors Other knowledge communities Contractual relationship Development Management Commissioned evaluator Independent researcher Hierarchical relationship Performance assessment (short feedback loop) Impact evaluation (intermediate feedback loop) Applied research (long feedback loop) Implementing staff Service relationship Intended beneficiaries MID-RANGE THEORY GENERAL THEORY PROJECT THEORY
Intermediate methodological options Rely on But Mostly change data not impact Cognitive biases and vested interests 1. Operational data and management feedback Limited generalisability; Limited credibility to non-participants Costly and indivisible Narrowly framed (spurious precision) 2. Participatory approaches 3. Quantitative counter-factual approaches Akerlof s lemon problem 4. Qualitative social research methods
The QuIP - story so far 2012-2015 Design and piloting Assessing Rural Transformations - ESRC/DFID project: goal to design and test a new approach (QuIP). Joint QuIP design with Farm Africa, Self Help Africa and Universities in Malawi, Ethiopia and UK. 8 pilot studies (2 countries x 2 projects x 2 years) 2016-2017 Proof of concept Set up BSDR Ltd as a social enterprise. Twenty further QuIP across eleven countries.
BSDR QuIP studies 2016-17 Activities Child nutrition Climate change adaptation Community mobilisation Drought response Factory working conditions Housing improvement Medical training Microfinance Rural livelihoods Value chain improvement Countries Burkina Faso Ethiopia Ghana Kenya India Malawi Mexico Tanzania Uganda UK Zambia Commissioned by Acumen Bristol City Council C&A Foundation Concern Worldwide Diageo Ltd Gorta Self Help Africa Habitat for Humanity Oxfam Save the Children Seed Global Health Tearfund TreeAid www.bathsdr.org
QuIP design features 1. A standard to be jointly adapted with the commissioner for each new assignment. 2. Sample from operational, monitoring data. 3. Benchmark of 24 semi-structured narrative interviews + 4 focus groups collected in a week. 4. Conducted by independent field researchers without knowledge of the project (blindfolding). 5. Data entry from field notes using pre-formatted Excel sheets to facilitate coding and analysis.
continued 6. Deductive coding of attribution claims as explicit, implicit or incidental (confirmatory). 7. Additional inductive thematic coding of drivers, outcomes and drivers of change (exploratory) 8. Semi-automated generation of tables and charts 9. Use of summary report as the starting point for joint sensemaking 10.Transparency: drill down from summary evidence to raw data for QA and deeper learning
Example of thematic coding Driver of Change (P/N) Primary Outcomes (OP/ON) Secondary Outcomes (OP/ON) Attribution (1-9) Full Answer Broken Answer There have been negative changes in the food we eat during the last six-months. This is related to both increase in price of food items and loss of income sources. There have been negative changes in the food we eat during the last six-months. This is related to both increase in price of food items and loss of income sources. There have been negative changes in the food we eat during the last six-months. This is related to both increase in price of food items and loss of income sources. Contrary to this, the clean water we drink has improved during this period as *** drilled a bore hole in our village which made it possible to survive here. N1 ON1 ON3 6 N3 ON3 6 the clean water we drink has improved during this period as *** drilled a bore hole in our village which made it possible to survive here. P8 OP2 OP3 1 ON1 No or decreased income ON3 Decreased food OP2 Clean drinking water OP3 Improved living conditions N1 N3 P8 Loss of income source Increase in food prices rehabilitated water sources https://www.youtube.com/wat ch?v=3VPzj1KaCXs&feature=e m-share_video_user
How can a QuIP be used? As a stand-alone exploratory approach for needs assessment and programme design As a stand-alone confirmatory evaluation approach (better alongside quantitative monitoring data, but doesn t require a baseline or control group). Mixed method combinations 1. Before other studies: scoping 2. In parallel: triangulation 3. Follow-up: drill down into specific issues 10
Illustrative case study Irish Aid Local Area Development Programme in Luwingu and Mbala districts. Implemented by Self Help Africa (2013-17). Goals: improved livelihood, health, food and nutrition security for 16,000 households. Evaluation combined one QuIP in each district with livelihood zoning, a baseline survey, follow-up nutrition surveys and a farm livelihood survey.
Example of closed question responses Income from farming Quantity of food consumed Variety of food consumed Family health Home food production Gender Age Children's diet CH 13 F 39 + = = = = + CH 6 F 40 = = - = - - CH 7 F 60 = = - = - - CH1 F 46 - = = = = = CH10 F 30 = = - - - = CH11 F 41 + = - = + + CH14 M 67 = - - = = = CH15 M 21 - - = - = = CH3 M 53 + - + + = = CH4 M 40 + + + + = + CH8 F 49 - = = - = = CH9 F 40 + = = + = =
Example of positive attribution table Domains SHA explicit SHA implicit Other factors NS6 NS16 NS18 NS 05 NS10 IY2 IY6 IY 21 IY 13 IY 22 IY 11 NS7 CH 7 CH10 CH 13 PA21 PA11 PA1 FGNS1 FGIY1 FGPA1 FGNS2 FGIY2 FGPA2 NS6 IY 21 NS7 IY26 IY19 CH11 CH 6 PA 3 PA5 PA11 CH4 CH3 CH14 FGCH2 NS2 NS6 NS 05 IY 22 NS11 IY15 CH 7 CH10 CH15 PA2 FGNS1 FGNS2 FGIY2 FGCH2 C. Health NS2 NS6 NS16 IY 21 NS11 IY26 IY17 CH 7 CH11 CH8 CH10 CH 13 PA 3 PA21 CH4 CH3 CH1 CH9 PA19 PA16 PA18 P12 FGNS1 TFGIY1 FGIY2 FGCH2 NS2 NS6 NS16 NS18 NS 05 NS12 NS10 IY6 IY 21 IY 13 IY 22 IY 11 NS7 NS4 NS11 IY9 PA 3 PA5 PA11 PA2 FGNS1 FGNS2 FGIY2 NS2 NS6 NS16 NS10 IY2 IY6 IY 21 IY 11 NS7 NS11 CH8 PA 3 PA5 PA16 FGNS1 D. Farming and income NS2 NS6 NS18 NS 05 NS10 IY2 IY 21 IY13 IY22 IY 11 NS7 NS11 IY19 IY15 PA5 PA1 CH4 PA20 PA2 P12 FGNS1 FGIY1 FGNS2 FGIY2 FGCH2 NS16 NS12 IY2 IY 21 IY22 NS7 NS4 NS11 IY3 IY26 PA 3 PA5 CH9 PA18 PA2 E. Food consumption IY3 CH4 CH9 FGNS1 NS6 NS16 NS10 IY2 IY6 IY 21 IY 13 IY 22 IY 11 NS7 IY9 IY26 IY19 IY15 CH 7 CH11 CH8 CH10 CH 13 PA 3 PA11 PA16 PA20 PA2 FGCH1 FGNS2 FGCH2 F. Who eats what and when? NS12 NS10 NS7 NS11 PA18 FGNS2 FGIY2 IY15 CH 13 CH4 PA18
Example of a positive drivers-outcomes table OUTCOMES understanding of storage methods health/nutrition income/ savings Increased yield Improved food Increased food Increased crop Improved IYCF development Increased hh spending on Varied diet agricultural community community Increase in Increase in education nutrition nutrition Improved Improved Improved Improved Increased Improved Improved antenatal cohesion practices numbers livestock security hygiene variety inputs DRIVERS Access to fertiliser/seeds/chemicals (SHA provided or purchased from SHA-related income) Provision of information on nutrition Government (social cash transfer) 18 (18) 27 (27) 5 (5) - 2 (2) - 1 (1) 4 (4) 1 (1) - 1 (1) 9 (6) 6 (6) - 1 (1) - 26 (23) 17 (14) - - - - - - - 17 (15) - 2 (2) - - - 8 (6) 17 (14) 1 (1) 2 (1) - 2 (1) - 1 (1) 6 (5) - - - - - 2 (2) - - WASH information 29 (22) 1 (1) - - - - - - - - - - - - - - Provision of livestock/poultry (SHA) 6 (5) - 1 (1) - - - 14 (14) 2 (2) - - - - - - - - Joined savings group 10 (7) 3 (3) - - - - - 1 (1) - - - - - 1 (1) - - Training on post-harvest processing and handling - - - - - - - - - 20 (16) - - - - - - Joined cooperative/s 2 (2) 8 (7) 2 (2) - - - - 3 (3) - - - - - - 2 (1) - Providing information on better farming methods 1 (1) - 6 (5) 1 (1) 1 (1) - - - - - - 1 (1) 1 (1) - 1 (1) - More land cultivated 3 (3) - 8 (6) 1 (1) 1 (1) - - - 1 (1) - - - 2 (2) - - - Increase in attendance at community meetings/training - 11 (10) - - - - - - - - 1 (1) - - - 2 (2) -
Example of dashboard based causal chain analysis with drill down
Comparison with other appraoches Group 1. Approaches with specific features that the QuIP also incorporates Appreciative Enquiry; Case Studies; Causal Link Monitoring; Collaborative Outcome Reporting; Critical Systems Heuristics; Goal Free Evaluation; Outcome Mapping; Positive Deviance; Success Case Method; Utilisation Focused Evaluation. Group 2. The QuIP is a more narrowly defined version of a broader approach. Beneficiary Assessment; Contribution Analysis; Developmental Evaluation; Innovation History; Institutional Histories; Outcome Harvesting; Process Tracing; Realist Evaluation. Group 3. More quantitative approaches Cost Benefit Analysis; Difference-in-Difference Evaluation; Qualitative Comparative Analysis; Randomized Control Trials; Social Return on Investment. Group 4. Approaches with stronger participatory and formative goals. Democratic Evaluation; Empowerment Evaluation; Horizontal Evaluation; Most Significant Change; Participatory Assessment of Development; Participatory Impact Assessment for Learning and Accountability; Participatory Evaluation and Participatory Rural Appraisal. See also www.betterevaluation.org
Caveat: what QuIP does not do Does not provide Responses Use as one input into microsimulation Run alongside a quantitative impact evaluation. Estimates of the magnitude of average treatment effects Reveals scope and range of responses Combine with Bayesian updating Use to design or follow-up on quantitative surveys. Statistically representative frequency counts Triangulate Perceptions matter! Objective facts Combine QuIP with process evaluation and follow-up stakeholder engagement. Recommendations for action 18
Ongoing agenda To conduct more studies in different contexts. To promote further research into qualitative impact evaluation approaches and standards. To build stronger communities of good enough evaluative practice To research the political economy of evaluative practice. [CEDIL = DFID Centre for Development Impact and Learning]
BSDR theory of change UoB support for BSDR and QuIP related research QuIP related research outputs by UoB and/or BSDR More and better understand ing about producing good social impact evidence More inclusive and sustainable development More and better QuIP studies conducted by BSDR and by other organisations Internal capacity of BSDR (staff, networks, identity, systems, finance etc.) support for BSDR and/or promotion of QuIP Wider support for producing good social impact evidence (demand) More and better evidence of social impact Increased capacity to produce good impact evidence (supply) Other external Capacity building work (including training, networking and dissemination) to build a QuIP community of practice (dashed lines suggest feedback loops)
Blindfolding Why do it? To enhance credibility of evidence by reducing the risk of pro-project and confirmation bias (of intended beneficiaries and field investigators). To give equal weight to all possible drivers of change. Can be combined with unblindfolded data collection: e.g. through joint follow-up interpretation of findings. Blindfolding is not always possible (e.g. in very low trust contexts). The QuIP does not require it. For confirmatory research the analyst cannot be blindfolded and this may also be a source of bias. However, their work can be audited. 22
The ethics of blindfolding The positives of blindfolding (more credible evidence) should outweigh the negatives (the greater good argument). Negatives include restricting field researchers ability to probe project impact in more detail. The rationale for blindfolding should be explained prior to participants in advance otherwise it is not informed consent. Blindfolding can be partial e.g. by revealing the identity of the commissioner, but not detailed project activities. Blindfolding can be temporary e.g. by holding unblindfolded joint follow-up meetings or sensemaking workshops. 23
Rethinking the qual/quant distinction 1. Two epistemological traditions (facts/numbers vs words/meaning). 2. Different methods: that can be nested but remain distinct. 3. Quantification as the process of codification that can be utilised withinall research traditions 24
Research as (de)codificatiion Evaluation process based on artful simplification of reality (time, space, ontology) Framing + Codifying Decodifying + reframing Design, data collection, analysis, and reporting through time Meshing feedback with follow-up actions Meshing recall with intervention periods Complex reality
Positionality of the analyst ? Intended beneficiaries Commissioner of study Study findings Selected respondents Project Theory of Change Photos and field report Project staff Field QuIP Analyst researchers Transcripts Project context Inner ring: data and documents QuIP guidelines Other Lead evaluator Outer ring: people Others Bath SDR Arrows indicate selected flows of information and influence
Timeliness Decision to fund Agree informally Select lead evaluator Identify field team Agree informally Ethical approval Sign contract Sign contract Sign contract Sign contract Supply sample frame data and approve data collection instruments Organise field access, pilot data instruments and train staff Finalise data collection design and line up trained analyst Clarify theory of change Data analysis and quality checks Collect and transfer data TIMELY ACTION Written and verbal feedback Client Lead Field research organisation(s) evaluator team
Attribution How do the actions of development agencies (X) contribute to improving people s wellbeing (Y) in diverse, complex and risky contexts (Z)? Strategies Statistical inference based on variable exposure to X controlling for Z: Confirmatory - starting with X Rely on intended beneficiaries own self-reported account of causal mechanisms linking X to Y and to Z: Exploratory start with Y and work back to X and Z
Credibility Good enough argument and evidence to enable A to convince B that X caused Y, subject to Z, on the basis of reasonable assumptions. Sufficiently transparent to be open to peer review or audit. Credibility threshold dependent on the prior knowledge and triangulation opportunities of B Credibility is closer in meaning to reasonable than rational
A credible causal claim (that X caused Y) Evidence that X and Y happened; Independent assertions by several stakeholders that X was an INUS cause of Y (with limited prompting); The absence of any more credible counter-explanation for why they might have said this; Their account of how X caused Y is consistent with a plausible theory. INUS- a possibly Insufficient but Necessary part of a causal package that is an Unnecessary but Sufficient cause of Y.
A complex context A setting in which The influence of X on Y is confounded by factors Z that are: impossible to fully identify hard to measure accurately interactive and cumulative in their influence on Y impossible fully to control
