Importance of Evidence-Based Decision Making in Policy Analysis

Evidence-Based Decision Making: The Contribution of Systematic Reviews in Synthesizing Evidence

General introduction to systematic reviews 1. What is a systematic review? 2. Why do/fund one? 3. For which policy questions is SR useful? 4. What are the main international bodies?

Systematic reviews as defined by the experts A systematic, up-to-date summary of reliable evidence of the benefits intervention (The Cochrane Collaboration) and risks of an Reviews sum up the best available research on a specific question (The Campbell Collaboration) Defines a process, not a product. The process is a systematic and transparent approach to all phases of a literature review (Jeff Valentine).

Systematic reviews are research Systematic reviews are a form of research Secondary observations In which studies are the unit of analysis Follow basic steps in the research process Aim to minimize bias and error

Example: microcredit Evidence used to justify development policy is often anecdotal: Small loans enable people in poverty to earn an income and provide for their families Each successful business feeds a family, employs more people and eventually helps empower a whole community (Opportunity International) Where evidence is used it is often based on single studies: Impact studies done on the Grameen Bank by independent researchers find that 5% of borrowers come out of poverty every year the status of women has been enhanced (Yunus, 2005) Where a range of studies are used, not clear how representative: Women participants in microcredit programs often experience important self-empowerment there is a strong indication from borrowers that microcredit improves their lives (Microfinance Gateway)

So what do reviews synthesizing all the existing high quality evidence say? Duvendack et al. (2011): All impact evaluations of microfinance suffer from weak methodologies and inadequate data [which] can lead to misconceptions about the actual effects of a microfinance programme It remains unclear under what circumstances, and for whom, microfinance has been and could be of real, rather than imagined, benefit to poor people Vaessen et al. (2012): From those studies deemed comparable and of minimum acceptable quality, we can conclude that overall the effect of microcredit on women s control over household spending is weak .

Why do we need systematic reviews? Sheer amount and flow of information/ research Variable quality of research outputs Need to separate the wheat from the chaff Problems of publication bias Limitations of single studies

What makes a systematic review systematic? 1. Scoping: defining answerable question, methods set out in study protocol 2. Rigorous search to identify published and unpublished sources, in any language 3. Application of study inclusion criteria (PICOS) 4. Critical appraisal of study quality, to assess how reliable is the evidence 5. Data extraction and organisation 6. Synthesis of evidence (outcomes along causal chain) 7. Interpreting results (policy and practice, research recommendations) 8. Improving and updating reviews as new evidence emerges

Types of (answerable) questions Policy area Example question Effectiveness: what difference does an intervention make? What are the impacts of daycare on child health, nutrition and cognitive development? What is the effectiveness of water, sanitation and hygiene interventions in terms of reducing child diarrhoea in LMICs? What is the comparative effectiveness of multiple interventions in attaining an outcome? Drivers of change: determinants of compliance or adherence What are the barriers and facilitators of patient adherence to tuberculosis treatment in Africa? What are the barriers to discontinuation of female genital mutilation/ cutting in Africa? Views: What are stakeholders experiences or preferences? What is the willingness to pay for clean water in developing countries? Cost-effectiveness or cost-benefit What is the effectiveness and cost-effectiveness of conditional and unconditional cash transfers in improving education outcomes? Risk factors associated with a condition or problem? What are the predictive factors for youth involvement in gang violence?

Resources and international bodies Cochrane Collaboration; 1993; www.cochrane.org Producing high quality information about the effectiveness of health care (> 5000 published online Cochrane library) Campbell Collaboration; 2000; www.campbellcollaboration.org Producing systematic reviews of the effects of social interventions (>200 published online Campbell library) International Development Coordinating Group (IDCG); 2010 www.campbellcollaboration.org/international_development Producing systematic reviews of high policy-relevance focusing on social and economic development interventions in LMICs International Initiative for Impact Evaluation, 3ie; 2008 www.3ieimpact.org/en/evidence/systematic-reviews EPPI Centre - An Institute of Education centre focusing on systematic reviews in education, health and social policy Collaboration for Environmental Evidence producing systematic reviews for environmental management