Experimental Insights in Social Sciences

Mechanism experiments Jens Ludwig University of Chicago & National Bureau of Economic Research

The social sciences, like the physical or biological sciences, are intellectual subjects, directed primarily toward understanding than action action Henry Reicken, 1969. President, Social Science Research Council understanding rather

Increasingly experimental action- oriented field

Experimental design problem: How to get as much policy-relevant info for given level of spending?

Y = outcome P = policy Usual approach: Test a P as close to real-world large- scale implementation as possible P Y Policy evaluation

$2 billion

Peter Rossis Iron Law of Evaluation The expected value of any net impact assessment of any social program is zero

PMY P = small schools Small school theory Y = student outcomes M = Greater school autonomy Peers with aligned interests Greater bonding with teachers & school staff

A tale of two possible experiments Policy evaluation P Y Mechanism experiment M Y Level of randomization could be either student or school Level of intervention is school Hence the $2 billion cost Get a sample of charter schools already have autonomy (one M) Randomize students to after school / weekend bonding time with school staff Group students by interests Cost < $2 billion

PY not always best way to get policy-relevant information We could look at P M We could look at M Y For testing P M we don t need same power as for P Y For testing M Y, don t need feasible, scalable real-world policies Mechanism experiments

What are mechanism experiments good for? Function Rule out policies Example Small schools

What are mechanism experiments good for? Function Rule out policies Expand set of policies for which we can forecast effects Example Small schools Moving to Opportunity (MTO)

The MTO Experiment MTO demonstration authorized by U.S. Congress -- Housing and Community Development Act of 1992 -- A randomized social experiment Open to families with children living in: -- public housing or in project-based assisted housing -- high-poverty neighborhoods (poverty rate >= 40%) 5 Sites: Baltimore, Boston, Chicago, Los Angeles, and New York -- 4600 families enrolled from 1994 to 1998 13

Random Assignment to 3 Groups Control No vouchers remain eligible for current project-based housing assistance Low poverty voucher (LPV) Restricted Section 8 voucher (<10% Poverty Census Tract) + Mobility Counseling Traditional voucher (TRV) Conventional Section 8 vouchers 14

Neighborhood Poverty Distribution (Weighted by time spent in each neighborhood during study period) Traditional Voucher Compliers vs Control Compliers 5 4 3 Density 2 1 0 0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 Neighborhood Poverty Rate Con Compliers (TRV) TRV Compliers 15

Neighborhood Poverty Distribution (Weighted by time spent in each neighborhood during study period) Low-Poverty Voucher Compliers vs Control Compliers 5 4 3 Density 2 1 0 0 .1 .2 .3 .4 .5 .6 .7 .8 .9 1 Neighborhood Poverty Rate Con Compliers (LPV) LPV Compliers 16

MTO effects on student test scores

What are mechanism experiments good for? Function Rule out policies Expand set of policies for which we can forecast effects Concentrate resources where there is greatest uncertainty Example Small schools Moving to Opportunity (MTO) Poverty / achievement link

P MY P = poverty M = stress Y = student achievement Policy evaluation (P Y) Mechanism experiment (M Y) Vs.

What are mechanism experiments good for? Function Rule out policies Expand set of policies for which we can forecast effects Concentrate resources where there is greatest uncertainty Strengthen causal inference Example Small schools Moving to Opportunity (MTO) Poverty / achievement link Schooling / crime link

Source: Swisher and Dennison LSE

Mechanism experiment Sample Vocational curriculum Executive functioning curriculum Control

Barriers to mechanism experiments

Lots of barriers to doing mechanism experiments are generic to all experiments Barrier to experimentation Randomization aversion Long-term outcomes (e.g. crime) Potential solution Incentives or clarify practical value Intermediate outcomes (school disciplinary actions) Etc.

Unique barrier for mechanism experiments

Basic research in medical sciences (good-to-great reputation) Basic research in social sciences (bad-to-terrible reputation)

Need to figure out how to get more from less