Explore a novel approach merging Difference-in-Difference evaluation with stochastic dominance to assess changes in child nutritional status in arid and semi-arid regions of Kenya. Discover the unique dataset and empirical results highlighting the impact on welfare over time.
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author. If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
A stochastic dominance approach to program evaluation And an application to child nutritional status in arid and semi-arid Kenya Felix Naschold University of Wyoming Christopher B. Barrett Cornell University May 2012 seminar presentation University of Sydney
Motivation Motivation Program Evaluation Methods By design they focus on mean Ex: average treatment effect (ATE) In practice, often interested in broader distributional impact Limited possibility for doing this by splitting sample Stochastic dominance By design, look at entire distribution Now commonly used in snapshot welfare comparisons But not for program evaluation. Ex: differences-in-differences This paper merges the two Diff-in-Diff (DD) evaluation using stochastic dominance (SD) to compare changes in distributions over time between intervention and control populations 1. 2. 3. 2
Main Contributions Main Contributions 1. Proposes DD-based SD method for program evaluation 2. First application to evaluating welfare changes over time 3. Specific application to new dataset on changes in child nutrition in arid and semi-arid lands (ASAL) of Kenya Unique, large dataset of 600,000+ observations collected by the Arid Lands Resource Management Project (ALRMP II) in Kenya (One of) first to use Z-scores of Mid-upper arm circumference (MUAC) 3
Main Results Main Results Methodology (relatively) straight-forward extension of SD to dynamic context: static SD results carry over Interpretation differs (as based on cdfs) Only feasible up to second order SD Empirical results Child malnutrition in Kenyan ASALs remains dire No average treatment effect of ALRMP expenditures Differential impact with fewer negative changes in treatment sublocations ALRMP a nutritional safety net? 1. 2. 4
Program evaluation Program evaluation (PE) methods (PE) methods Fundamental problem of PE: want to but cannot observe a person s outcomes in treatment and control state x x = i iT iC Solution 1: make treatment and control look the same (randomization) Gives average treatment effect as Solution 2: compare changes across treatment and control (Difference-in-Difference) Gives average treatment effect as: , , 1 T t T t E E x x = = T E x C E x E E x x , , 1 C t C t 5
New PE method New PE method based on SD based on SD Objective: to look beyond the average treatment effect Approach: SD compares entire distributions not just their summary statistics Two advantages 1. Circumvents (highly controversial) cut-off point Examples: poverty line, MUAC Z-score cut-off 2. Unifies analysis for broad classes of welfare indicators 6
Stochastic Dominance Stochastic Dominance iff max , x ( ) F x z min x First order: A FOD B up to ( ) 0 , F x x min x z B A Cumulative % of population FB(x) FA(x) xmax 0 MUAC score Z- Sth order: A sth order dominates B iff ( ) x ( ) x s s 0 , F F x min x z B A 7
SD and single differences SD and single differences These SD dominance criteria Apply directly to single difference evaluation (across time OR across treatment and control groups) Do not directly apply to DD Literature to date: Single paper: Verme (2010) on single differences SD entirely absent from the program evaluation literature (e.g., Handbook of Development Economics) 8
Expanding SD to DD Expanding SD to DD estimation estimation - - Method Method Practical importance: evaluate beyond-mean effect in non- experimental data Let , and G denote the set of probability density functions of , with ?? ,?? ? The respective cdfs of changes are GA( ) and GB( ) Then A FOD B iff A Sth order dominates B iff G = x x 1 t t ( ) ( ) ( ) ( ) A G 0 , G G min 0 max B A s B s , min max 9
Expanding SD to DD: Expanding SD to DD: interpretation differences interpretation differences 1. Cut-off point in terms of changes not levels. Cdf orders change from most negative to most positive initial poverty blind or initial malnutrition blind . (Partial) remedy: run on subset of ever-poor/always-poor 2. Interpretation of dominance orders FOD: differences in distributions of changes between intervention and control sublocations SOD: degree of concentration of these changes at lower end of distributions TOD: additional weight to lower end of distribution. Is there any value to doing this for welfare changes irrespective of absolute welfare? Probably not. 10
Setting and data Setting and data Arid and Semi-arid districts in Kenya Characterized by pastoralism Highest poverty incidences in Kenya, high infant mortality and malnutrition levels above emergency thresholds Data From Arid Lands Resource Management Project (ALRMP) Phase II 28 districts, 128 sublocations, June 05- Aug 09, 602,000 child obs. Welfare Indicator: MUAC Z-scores Severe malnutrition in 2005/6: Median child MUAC z-score -1.22/-1.12 (Intervention/Control) 10 percent of children had Z-scores below -2.31/-2.14 (I/C) 25 percent of children had Z-scores below -1.80/-1.67 (I/C) 11
The pseudo panel The pseudo panel Sublocation-specific pseudo panel 2005/06-2008/09 Why pseudo-panel? 1. Inconsistent child identifiers 2. MUAC data not available for all children in all months 3. Graduation out of and birth into the sample How? 14 summary statistics for annual mean monthly sublocation - specific stats: mean & percentiles and poverty measures Focus on malnourished children Thus, present analysis median MUAC Z-score of children z 0 Control and intervention according to project investment 12
Results: DD Regression Results: DD Regression Pseudo panel regression model where D is the intervention dummy variable of interest NDVI is a control for agrometeorological conditions L are District fixed effects to control for unobservables within major jurisdictions No statistically significant average program impact 13
DD regression panel results DD regression panel results (1) (2) 10th (3) 25th (4) (5) VARIABLES median of MUAC Z <0 median of MUAC Z <-1 median of MUAC Z <-2 percentile percentile intervention dummy 0.0735 0.0832 0.0661 0.0793 0.0531 (0.248) (0.316) (0.371) (0.188) (0.155) change in NDVI 2005/06-08/09 1.308* 2.611*** 2.058*** 0.927* 0.768* (0.0545) (0.00294) (0.00754) (0.0997) (0.0767) (change in NDVI)2 2005/06-08/09 -12.91** (0.0293) -8.672 (0.136) -12.70* (0.0510) -0.954 (0.802) 1.924 (0.479) Constant 0.501*** 0.892*** 0.839*** 0.203*** 0.120*** (2.99e-07) (1.40e-08) (8.70e-09) (0.000133) (0.00114) Observations 114 114 114 114 106 R-squared 0.319 0.299 0.297 0.249 0.280 Robust p-values in parentheses *** p<0.01, ** p<0.05, * p<0.1 District dummy variables included. 14
SD Results SD Results Three steps: Steps 1 & 2: Simple differences SD within control and treatment over time: No difference in trends. Both improved slightly. SD control vs. treatment at beginning and at end: Control sublocations dominate in most cases, intervention never dominates. Step 3: SD on Diff-in-Diff (results focus for today) 15
Expanding SD to DD Expanding SD to DD controlling for covariates controlling for covariates In regression Diff-in-Diff: simply add (linear) controls In SD-DD need a two step method 1. Regress outcome variable on covariates 2. Use residuals (the unexplained variation) in SD-DD In application below, use first stage controls for agro- meteorological conditions (as reflected in remotely-sensed vegetation measure, NDVI). 16
FOSD Difference Intervention vs. Difference Control Median MUAC of obs<0. Categorization by Investment 1 .8 For (drought-adjusted) median MUAC z-scores: Below z=0.2, intervention sites FOD control sites, although not at 5% statistical significance level. ALRMP interventions appear moderately effective in preventing worsening nutritional status among children. % of sublocations .6 .4 .2 0 -1 -.4 .2 .8 1.4 2 difference in median MUAC Z-score of observations with MUAC<0. drought adjusted. 2005/06-2008/09 Control intervention FOSD Difference Intervention vs. Difference Control Median MUAC of obs<0. Categorization by Investment .2 .1 % of sublocations 0 -.1 -.2 -1 -.4 .2 .8 1.4 2 difference in median MUAC Z-score with MUAC<0. drought adjusted. 2005/06-2008/09 17 Confidence interval (95 %) Estimated difference
Similar results at other Similar results at other quantile quantile breaks breaks FOSD Difference Intervention vs. Difference Control 25th percentile MUAC. Categorization by Investment 1 .8 % of sublocations .6 .4 .2 0 -1.5 difference in 25th percentile MUAC Z-score. drought adjusted. 2005/06-2008/09 -.8 -.1 .6 1.3 2 Control intervention 18
Similar results at other Similar results at other quantile quantile breaks breaks FOSD Difference Intervention vs. Difference Control 10th percentile MUAC. Categorization by Investment 1 .8 % of sublocations .6 .4 .2 0 -1.5 difference in 10th percentile MUAC Z-score. drought adjusted. 2005/06-2008/09 -.8 -.1 .6 1.3 2 Control intervention 19
Conclusions Conclusions Existing program evaluation approaches focus on estimating the average treatment effect. In some cases, that is not really the impact statistic of interest. This paper introduces a new SD-based method to evaluate impact across entire distribution for non-experimental data Results show the practical importance of looking beyond averages Standard Diff-in-Diff regressions: no impact at the mean SD DD: intervention locations had fewer negative observations and of smaller magnitude, especially median and below ALRMP II may have functioned as nutritional safety net (though only correlation, there is no way to establish causality) 20
Thank you for your time, Thank you for your time, interest and comments interest and comments 21
SD, poverty & social SD, poverty & social welfare orderings (1) welfare orderings (1) 1. SD and Poverty orderings Let SDs denote stochastic dominance of order s and P stand for poverty ordering ( has less poverty ) Let =s-1 Then A P B iff A SDs B SD and Poverty orderings are nested A SD1 B A SD2 B A SD3B A P1 B A P2 B A P3 B 22
SD, poverty & social SD, poverty & social welfare orderings (2) welfare orderings (2) 2. Poverty and Welfare orderings (Foster and Shorrocks 1988) Let U(F) be the class of symmetric utilitarian welfare functions Then A P B iff A U B Examples: U1 represents the monotonic utilitarian welfare functions such that u >0. Less malnutrition is better, regardless for whom. U2 represents equality preference welfare functions such that u <0. A mean preserving progressive transfer increases U2. U3 represents transfer sensitive social welfare functions such that u >0. A transfer is valued more lower in the distribution Bottom line: For welfare levels tests up to third order make sense 23
DD Regression 2 DD Regression 2 Individual MUAC Z-score regression To test program impact with much larger data set Still no statistically significant average program impact 25
Results Results DD indiv data DD data regression regression indiv Dependent variable: Individual MUAC Z-score VARIABLES time dummy (=1 for 2008/09) 0.0785 (0.290) control - intervention by investment -0.0576 (0.425) Diff in diff 0.0245 (0.782) Normalized Difference Vegetation Index 1.029*** (6.25e-07) Constant -1.391*** (0) Observations 271061 R-squared 0.033 26 Robust p-values in parentheses *** p<0.01, ** p<0.05, * p<0.1 District dummy variables included.
Full SD results Full SD results Sublocation panel Individual data MUAC Z-Score Dominance Median MUAC of obs < 0 Dominance % below -1 SD Which* * Almost Y Y Y Y Y Almost Y Y N Y Y Which* Signif. Dominance Signif. Which* Signif. I.1 Intervention 05/06-08/09 FOSD SOSD TOSD I.2 Control 05/06-08/09 FOSD SOSD TOSD II.1 Intervention vs. Control 05/06 FOSD SOSD TOSD II.2 Intervention vs. Control 08/09 FOSD SOSD TOSD III. Diff Intervention vs Diff. Control FOSD SOSD Y Y Y Y Y Y 08/09 08/09 08/09 08/09 08/09 08/09 Control Control Control - - - NS S S NS NS NS NS NS NS NS NS NS 08/09 08/09 08/09 08/09 08/09 08/09 # Control Control Control Control Control NS NS NS Y Y Y Y Y Y Y Y Y Y Y Y 08/09 08/09 08/09 08/09 08/09 08/09 Control Control Control Control Control Control S S S S S S S S S S S s NS NS NS NS NS NS NS NS NS Y (almost) Y Y N Unclear Unclear N Y? - - NS NS N Y - NS NS Interve ntion * Lower curves to the right are dominate for these indicators for which a greater number indicates better . ** For parts I. and II. higher curves to the left dominate for the proportion of observations below -1SD, as lower proportions are better . In contrast, for changes from 2005/06-2008/09 in part III. larger positive changes are better, so lower curves to the right dominate. # Control sites dominate up to MUAC Z-score of -0.1. Intervention sites dominate for MUAC Z-score > 0. 27
FOSD Difference Intervention vs. Difference Control Median MUAC of obs<0. Categorization by Investment 1 .8 % of sublocations .6 .4 .2 0 -1 -.4 .2 .8 1.4 2 difference in median MUAC Z-score of observations with MUAC<0. drought adjusted. 2005/06-2008/09 Control intervention 28
