Stochastic Dominance Approach to Program Evaluation in Arid Kenya

A stochastic dominance approach to program evaluation And an application to child nutritional status in arid and semi-arid Kenya Felix Naschold Cornell University & University of Wyoming Christopher B. Barrett Cornell University AAEA 27 July 2010

Motivation Program Evaluation Methods By design they focus on mean. Ex: average treatment effect In practice often interested in distributional impact Limited possibility for doing this by splitting sample 2. Stochastic dominance By design look at entire distribution Now commonly used in snapshot welfare comparisons But not for program evaluation. Ex: differences-in-differences 3. This paper merges the two Diff-in-Diff (DD) evaluation using stochastic dominance (SD) 1. 2

Main Contributions of this paper 1. Proposes DD-based SD method for program evaluation 2. First application to evaluating welfare changes over time 3. Specific application to new dataset on changes in child nutrition in arid and semi-arid lands (ASAL) of Kenya Unique, large dataset of 600,000+ observations collected by the Arid Lands Resource Management Project (ALRMP II) (one of) first to use Z-scores of Mid-upper arm circumference (MUAC) 3

Main Results Methodology (relatively) straight-forward extension of SD to dynamic context: static SD results carry over Interpretation differs (as based on cdfs) Only up to second order SD Empirical results Child malnutrition in Kenyan ASALs remains dire No average treatment effect of ALRMP expenditures Differential impact with fewer negative changes in treatment sublocations ALRMP a nutritional safety net? 1. 2. 4

Program evaluation (PE) methods Fundamental problem of PE: want to but cannot observe a person s outcomes in treatment and control state x x = i iT iC Solution 1: make treatment and control look the same (randomization) Gives average treatment effect Solution 2: compare changes across treatment and control (Difference-in-Difference) Gives average treatment effect: , , 1 T t T t E E x x = = T E x E E x C E x x , , 1 C t C t 5

New PE method based on SD Objective: to look beyond the average treatment effect Approach: SD compares entire distributions not just their summary statistics Two advantages 1. Circumvents (highly controversial) cut-off point. Examples: poverty line, MUAC Z-score cut-off 2. Unifies analysis for broad classes of welfare indicators 6

Definition of Stochastic Dominance iff max , x ( ) x F z min x F First order: A FOD B up to ( ) x 0 , x min x z B A Cumulative % of population FB(x) FA(x) xmax 0 MUAC score Z- Sth order: A sth order dominates B iff ( ) x ( ) x s s 0 , F F x min x z B A 7

SD and single differences These SD dominance criteria Apply directly to single difference evaluation (across time OR across treatment and control groups) Do not directly apply to DD Literature to date: Single paper: Verme (2010) on single differences SD entirely absent from PE literature (e.g. Handbook of Development Economics) 8

Expanding SD to DD estimation - Method Practical importance: evaluate beyond-mean effect in non- experimental data Let , denote the set of probability density functions of . and The respective cdfs of changes are GA( ) and GB( ) Then A FOD B iff A Sth order dominates B iff B G = x x 1 t t g ( ) G ( ) G g A B ( ) ( ) ( ) ( ) A G 0 , G G min 0 max B A s s , min max 9

Expanding SD to DD estimation 2 differences in interpretation 1. Cut-off point in terms of changes not levels. Cdf orders changes from most negative to most positive poverty blind or malnutrition blind . (Partial) remedy: run on subset of ever-poor/always-poor 2. Interpretation of dominance orders FOD: differences in distributions of changes between intervention and control sublocations SOD: degree of concentration of these changes at lower end of distributions TOD: additional weight to lower end of distribution. Sense in doing this for welfare changes irrespective of absolute welfare? 10

Setting and data Arid and Semi-arid district in Kenya Characterized by pastoralism Highest poverty incidences in Kenya, high infant mortality and malnutrition levels above emergency thresholds Data From Arid Lands Resource Management Project Phase II 28 districts, 128 sublocations, June 05- Aug 09, 600,000 obs. Welfare Indicator: MUAC Z-scores Severe amount of malnutrition: 10 percent of children have Z-scores below -1.54 and -2.55 25 percent of children have Z-scores below -1.15 and -2.06 11

The pseudo panel used Sublocation-specific pseudo panel 2005/06-2008/09 Why pseudo-panel? 1. Inconsistent child identifiers 2. MUAC data not available for all children in all months 3. Graduation out of and birth into the sample How? 14 summary statistics mean & percentiles and poverty measures Focus on malnourished children Thus, present analysis median MUAC Z-score of children below 0 Control and intervention according to project investment 12

Results: DD Regression Pseudo panel regression model No statistically significant average program impact 13

Results DD regression panel (1) (2) (3) (4) (5) VARIABLES median of MUAC Z <0 10th 25th median of MUAC Z <-1 median of MUAC Z <-2 percentile percentile intervention dummy based on ALRMP investment 0.0735 0.0832 0.0661 0.0793 0.0531 (0.248) (0.316) (0.371) (0.188) (0.155) change in NDVI 2005/06-08/09 1.308* 2.611*** 2.058*** 0.927* 0.768* (0.0545) (0.00294) (0.00754) (0.0997) (0.0767) squared change in NDVI 2005/06-08/09 -12.91** -8.672 -12.70* -0.954 1.924 (0.0293) (0.136) (0.0510) (0.802) (0.479) Constant 0.501*** 0.892*** 0.839*** 0.203*** 0.120*** (2.99e-07) (1.40e-08) (8.70e-09) (0.000133) (0.00114) Observations 114 114 114 114 106 R-squared 0.319 0.299 0.297 0.249 0.280 Robust p-values in parentheses *** p<0.01, ** p<0.05, * p<0.1 District dummy variables included. 14

Stochastic Dominance Results Three steps: Steps 1 & 2: Simple differences SD within control and treatment over time: no difference in trends. Both improved slightly SD control vs. treatment at beginning and at end: control sublocations dominate in most cases, intervention never Step 3: SD on DD (results focus for today) 15

FOSD Difference Intervention vs. Difference Control Median MUAC of obs<0. Categorization by Investment 1 .8 % of sublocations .6 .4 .2 0 -1 -.4 .2 .8 1.4 2 difference in median MUAC Z-score of observations with MUAC<0. drought adjusted. 2005/06-2008/09 Control intervention FOSD Difference Intervention vs. Difference Control Median MUAC of obs<0. Categorization by Investment .2 .1 % of sublocations 0 -.1 -.2 -1 -.4 .2 .8 1.4 2 difference in median MUAC Z-score with MUAC<0. drought adjusted. 2005/06-2008/09 Confidence interval (95 %) Estimated difference 16

FOSD Difference Intervention vs. Difference Control 25th percentile MUAC. Categorization by Investment 1 .8 % of sublocations .6 .4 .2 0 -1.5 difference in 25th percentile MUAC Z-score. drought adjusted. 2005/06-2008/09 -.8 -.1 .6 1.3 2 Control intervention 17

FOSD Difference Intervention vs. Difference Control 10th percentile MUAC. Categorization by Investment 1 .8 % of sublocations .6 .4 .2 0 -1.5 difference in 10th percentile MUAC Z-score. drought adjusted. 2005/06-2008/09 -.8 -.1 .6 1.3 2 Control intervention 18

Conclusions Existing program evaluation approaches average treatment effect This paper: new SD-based method to evaluate impact across entire distribution for non-experimental data Results show practical importance of looking beyond averages Standard DD regressions: no impact at the mean SD DD: intervention sublocations had fewer negative observations ALRMP II may have functioned as nutritional safety net (though only correlation, no way to get at causality) 19

Thank you. 20

Expanding SD to DD estimation controlling for covariates In regression DD: simply add (linear) controls In SD-DD need a two step method 1. Regress outcome variable on covariates 2. Use residuals (the unexplained variation) in SD DD In application below first stage controls for drought (NDVI) 21

SD, poverty & social welfare orderings (1) 1. SD and Poverty orderings Let SDs denote stochastic dominance of order s and P stand for poverty ordering ( has less poverty ) Let =s-1 Then A P B iff A SDs B SD and Poverty orderings are nested A SD1 B A SD2 B A SD3B A P1 B A P2 B A P3 B 22

SD, poverty & social welfare orderings (2) 2. Poverty and Welfare orderings (Foster and Shorrocks 1988) Let U(F) be the class of symmetric utilitarian welfare functions Then A P B iff A U B Examples: U1 represents the monotonic utilitarian welfare functions such that u >0. Less malnutrition is better, regardless for whom. U2 represents equality preference welfare functions such that u <0. A mean preserving progressive transfer increases U2. U3 represents transfer sensitive social welfare functions such that u >0. A transfer is valued more lower in the distribution Bottomline: For welfare levels tests up to third order make sense 23

The data (2) extent of malnutrition Table 3 10th percentile MUAC Z-score whole sample Year Garissa Kajiado Laikipia Mandera Marsabit Mwingi Narok Nyeri Tharaka Turkana 2005/06 -2.4 -2.14 -1.75 -2.65 -2.33 -2.36 -2.55 -1.67 -1.87 -2.26 2008/09 -1.88 -2.22 -2.1 -2.13 -2.29 -2.14 -2.35 -1.54 -1.74 -2.25 Table 4 25th percentile MUAC Z-score whole sample year Garissa Kajiado Laikipia Mandera Marsabit Mwingi Narok Nyeri Tharaka Turkana 2005/06 -1.97 -1.67 -1.16 -2.06 -1.79 -1.84 -1.96 -1.2 -1.45 -1.85 2008/09 -1.45 -1.76 -1.4 -1.69 -1.69 -1.68 -1.76 -1.15 -1.28 -1.86 24

DD Regression 2 Individual MUAC Z-score regression To test program impact with much larger data set Still no statistically significant average program impact 25

Results DD regression indiv data Dependent variable: Individual MUAC Z-score VARIABLES time dummy (=1 for 2008/09) 0.0785 (0.290) control - intervention by investment -0.0576 (0.425) Diff in diff 0.0245 (0.782) Normalized Difference Vegetation Index 1.029*** (6.25e-07) Constant -1.391*** (0) Observations 271061 R-squared 0.033 Robust p-values in parentheses *** p<0.01, ** p<0.05, * p<0.1 District dummy variables included. 26

Full table of SD results Median MUAC of obs < 0 Dominance Sublocation panel Individual data MUAC Z-Score Dominance % below -1 SD Which* * Almost Y Y Y Y Y Almost Y Y N Y Y Which* Signif. Dominance Signif. Which* Signif. I.1 Intervention 05/06-08/09 FOSD SOSD TOSD I.2 Control 05/06-08/09 FOSD SOSD TOSD II.1 Intervention vs. Control 05/06 FOSD SOSD TOSD II.2 Intervention vs. Control 08/09 FOSD SOSD TOSD III. Diff Intervention vs Diff. Control FOSD SOSD Y Y Y Y Y Y 08/09 08/09 08/09 08/09 08/09 08/09 Control Control Control - - - NS S S NS NS NS NS NS NS NS NS NS 08/09 08/09 08/09 08/09 08/09 08/09 # Control Control Control Control Control NS NS NS Y Y Y Y Y Y Y Y Y Y Y Y 08/09 08/09 08/09 08/09 08/09 08/09 Control Control Control Control Control Control S S S S S S S S S S S s NS NS NS NS NS NS NS NS NS Y (almost) Y Y N Unclear Unclear N Y? - - NS NS N Y - NS NS Interve ntion * Lower curves to the right are dominate for these indicators for which a greater number indicates better . ** For parts I. and II. higher curves to the left dominate for the proportion of observations below -1SD, as lower proportions are better . In contrast, for changes from 2005/06-2008/09 in part III. larger positive changes are better, so lower curves to the right dominate. # Control sites dominate up to MUAC Z-score of -0.1. Intervention sites dominate for MUAC Z-score > 0. 27

FOSD Difference Intervention vs. Difference Control Median MUAC of obs<0. Categorization by Investment 1 .8 % of sublocations .6 .4 .2 0 -1 -.4 .2 .8 1.4 2 difference in median MUAC Z-score of observations with MUAC<0. drought adjusted. 2005/06-2008/09 Control intervention 28