Research Evaluation Problems and Mechanisms: Insights for Policy Implementation
Discover the challenges in research and evaluation processes, the importance of theories and assumptions, and the role of different mechanisms in policy development. Explore ways to enhance understanding through knowledge repositories for effective implementation.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Wrap up and consequences for TOR s Athens June 2017
Research/ evaluation problems Prevent bags of problems ; having to address 17 questions and subquestions with no logic between them is not doable; An Issue Is Not A Research Question . The rapporteur of the Transport Group Don t try to take on board everything Make sure that the background knowledge you assume is correct and Make that knowledge shareable ( in the form of eg a working paper Decomplexify
Role of theories/ assumptions Basically two routes: A.The program theory /intervention theory / the Policy are Theories Incarnate route and B.The scientific ( explanatory) theory route which can assist in developing and implementing policies
Route 1: the Intervention th/ Program theory route Log Frames are not more than the beginning ; Boxes with colours, graphics, arrows and other nice things for nice people are not `program/ interventions. Only a starter. Labelling the interventions/ measures as Mechanisms is wrong Exercise, when finding M s, in working with policy makers and stakeholders and end groups to find the theory incarnate . O. Swab: A Pr theory is the theory of the programme makers + something else
Types of Mechanisms / Coleman boat model Transformational mechanisms Situational mechanisms Action-formation mechanisms
Summarize what is known about mechanisms by searching knowledge repositories like CAMPBELL, 3IE, COCHRANE, WHAT WORKS SITES and others Dont believe the statement that information on mechanisms in earlier studies is not relevant because contexts always differ . No, they do not , as there are demi-regs and path dependencies Fake handbags: be careful Methods: several. See EVALSED 2012 but also Leeuw (2003): an overview of 3 methods in American Journal of Evaluation . Criteria: logical consistency/ validity and problem relevance
Route 2: explanatory theories In helping developing and implementing policies / measures etc make use of insights from explanatory theories on causes of behavioral changes/ institutional changes Remember this:
EBM: Pawsons medicine development pipeline Phase I Safety and dose-finding Phase II Feasibility Studies Phase III Large-scale RCT Basic Research Therapeutic Discovery Preclinical Development Regulatory Approval Animal Testing typically 10-14 years
EBP: Pawsons Pathway to [social] policy programs RCT Full-scale evaluation ( RCT ) Policy Instigation Programme management Demonstration project 2 to 5 years
The six key principles of a theory- based impact evaluation are: 1. Map out the causal chain (programme theory) 2. Understand context 3. Anticipate heterogeneity 4. Rigorous evaluation of impact using a credible counterfactual 5. Use mixed methods and do rigorous analysis 6. Results visualization & transfer / use focused
THE COMPLEXITY PROBLEM PREVENT DOING SLOPPY TBE S: TILLEY S CASE OF PRODUCING ERROR COSTS POLISHING UP A POLICY THEORY THAT DOES NOT EXIST PREVENT DESIGNED BLINDNESS IS IT A SLOPPY EVALUATION WHEN NO PROGRAM THEORY /TOC HAS BEEN FOUND/ TESTED?
WORLD BANK TEMPLATE When Assumptions/ toc/ program th ~ impl th BIG DA TA Tde VISUALIAZATION, TRANFER AND USE
ROLE OF THEORIES WHAT ELSE CAN BE ADDED (OR DELETED) WITH TDE AS STARTING POINT? ROLE OF MECHANISMS VICTORE
Good luck! Frans.leeuw@maastrichtuniversity.nl flleeuw@cuci.nl
