Understanding Graduate Returns and Variation

The economic returns to a degree: how great and how varied are they? (And what might they tell us?) Gianna Boero, Robin Naylor, and Jeremy Smith University of Warwick UEA 7th March 2016 1

Plan of Talk 1. Context 1: Context 2: Evidence and Policy Theory and Interpretation 2. Institutional Arrangements 3. Data and Methodology 4. Results 5. Conclusions and Further Work UEA 7th March 2016 2

Plan of Talk 1. Context 1: Importance of HE Evidence and Policy Human Capital, R&D, Economic Growth HE Participation and Labour Supply Socio-economic Mobility/Persistence Political Economy: fees and funding Returns to Education Years of Schooling Qualification Levels Grades Performance HE Policy relevance of estimated returns in UK Dearing Report and evidence from Blundell et al. (2000: PTO) Browne Report UEA 7th March 2016 3

Average Graduate Premium in UK Blundell et al. (2000): Estimates ln i w NCDS1958 birth cohort 1991 hourly wage data HE i = + + X i i where HE is Highest Educational Qualification Rich set of observable characteristics Assumes that: ( | i E HE X = , ) ( | ) E X i i i i Individuals with different HE do not differ on average in unobservables. Results Graduate Earnings Premium (Relative to control group with 2+ A-levels) 17% 37% Men Women UEA 7th March 2016 4

Variation around Average Subject Higher for Science, Social Science (Harkness and Machin, inter alia) Institution Higher for Elite HEIs (Chevalier et al., inter al.) Hence Differential Fees (Greenaway and Haynes) Prior Schooling (Naylor and Smith) Cohort Walker and Zhu, 2008 Expansion no effect on average Increased premium in highest quartile (Ability composition effect?) Degree Class Premia (Not available in NCDS) UEA 7th March 2016 5

Interpretation of Graduate Earnings Premium Human Capital Theory Signalling/Screening/Sorting Theories Statistical Discrimination Short-run only? Employer Learning/Statistical Discrimination UEA 7th March 2016 6

How might we interpret any variation in graduate returns by Grades/Performance? Again: HKT vs Signalling Labour market has various pools or non-continuities Employers of some types of occupation recruit only from Graduate pool Criteria for successful hiring? Cognitive skills Non-cognitive skills Measured by: Degree Class Transcript/HEAR Subject/HEI Interview Internship/experience Are these abilities: Revealed? Signalled? (US: Arcidiacono) (UK: opposite of US?) UEA 7th March 2016 7

Jo Johnson, Minister of State for Universities and Science, speaking about BIS s Green Paper Fulfilling our Potential: Teaching Excellence, Social Mobility and Student Choice , stated that, We want to encourage a (grade point average) system which provides greater information to employers about where attainment really lies. It needs to sit alongside, rather than replace, the honours degree classification But there is a very big band, the 2.1 band. It disguises very considerable differences in attainment. You can be at the top of the band and then be 50 percentage points below and still be getting a 2.1. And students who worked hard should be able to signal to employersthat s what they ve achieved. (Cited on BBC News website 06/11/2015.) UEA 7th March 2016 8

Degree Class Premia: Evidence from available data Background BCS70 LFS USR/HESA GCS Classification of Honours Degrees: Classification Rules: First Upper Second Lower Second Third Pass (Non-honours) (>=2.1 => Good ) (=<2.2 => Lower ) Based on: Overall average Papers in class Final exams/coursework Viva Anecdotally: Achieves vs Is ! = HKT vs Signalling! UEA 7th March 2016 9

Estimated log wage premia (BCS70) Good degree premium over Lower (1) (2) (3) (4) (5) Wages observed in year: 2000 2000 2000 2000 2000 Wages observed at age: 30 30 30 30 30 Good degree class premium relative to lower degree class 0.078 (0.007) 0.077 (0.008) 0.073 (0.012) 0.071 (0.014) 0.068 (0.019) Lower degree premium over A- levels Lower degree class premium relative to 2+ A-levels 0.119 (0.000) 0.105 (0.001) 0.107 (0.000) 0.103 (0.001) 0.109 (0.000) Family background No Yes Yes Yes Yes Ability at age 10 No No Yes Yes Yes Ability at age 5 No No No Yes Yes Non-Cognitive ability at ages 5, 10 No No No No Yes Other controls Yes Yes Yes Yes Yes No. of Obs 3046 3046 3046 3046 3046 R2 0.0814 0.0988 0.1029 0.1119 0.1188 Notes: p-values in parentheses. Ability controls include: BAS (verbal), BAS (numerical). Background controls include: parental income, parental social class, mother s interest in education, father s interest in education, mother s education, father s education. Other controls include: region (aged 10), gender, marital status and number of children, ethnicity. UEA 7th March 2016 10

Estimated log wage premia (LFS): selected birth cohorts in 1969-1971 Wages observed at: 2005-2012 Wages observed at age: 36-41 Good degree class premium (relative to lower degree class) 0.087 (0.001) Lower degree class 0.188 (relative to 2+ A-levels) (0.000) Other controls Yes No. of Obs 2930 R2 0.152 UEA 7th March 2016 11

Estimated log-earnings premia (USR91, graduate cohort), birth cohort 1969-1971 USR-FDS: First Destination Median Occupational earnings Earnings observed at: 1992 1992 Earnings observed at age: 21-23 21-23 Good degree class premium relative to lower degree class 0.046 (0.000) 0.043 (0.000) Ability and background controls No Yes Other controls Yes Yes No. of Obs. 22,459 22,459 R2 0.334 0.336 Note: p-values in parentheses. Ability controls include: pre-University qualifications. Background controls include: social class of parents, school-type. Other controls include: gender, marital status, University attended and type of degree course. UEA 7th March 2016 12

Estimated log-wage premia (GCS1990, graduate): birth cohort 1968-1970 (1) (2) (3) (4) Wages observed at: 1991 1991 1996 1996 Wages observed at age 21-23 21-23 26-28 26-28 Good degree class premium relative to lower degree class 0.051 (0.014) 0.049 (0.014) 0.084 (0.014) 0.079 (0.014) Ability and background controls No Yes No Yes Other controls Yes Yes Yes Yes No. of Obs 2839 2839 3652 3652 R2 0.127 0.131 0.115 0.119 Note: p-values in parentheses. Ability controls include pre-university qualifications, background controls include parental education, and other controls include age, gender, ethnicity, and marital status. UEA 7th March 2016 13

Estimated log-earnings premia (USR; HESA: selected cohorts) and by university type graduates aged 21-23 Cohort 1985 1986 1987 1988 1989 1990 1991 1992 1993 1998 All 0.025 (0.00) 0.026 (0.00) 0.023 (0.00) 0.030 (0.00) 0.021 (0.00) 0.034 (0.00) 0.043 (0.00) 0.061 (0.00) 0.064 (0.00) 0.064 (0.00) 103519 HE popn 80947 0.499 79057 0.468 80655 82738 83360 84656 83932 94863 99569 0.427 0.421 0.399 0.373 R2 0.336 0.275 0.273 0.214 UEA 7th March 2016 14

Figure 5a: Coefficients on degree class variables over time ( constant earnings) - Males 0.08 0.06 0.04 0.02 0.00 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 Coeff -0.02 -0.04 -0.06 -0.08 -0.10 -0.12 Year First 2:2 Third Other UEA 7th March 2016 15

Plan of Talk 1. Context 1: Context 2: Evidence and Policy Theory and Interpretation How might we interpret evidence of a premium by class of degree awarded? Why might any premium by degree class change across cohorts? UEA 7th March 2016 16

Contrast with Arcidiacono on US context of high schools vs colleges Hypothesis 1 Pay 2 8 0 Average mark reveals ability/productivity to Employer and this is rewarded in the labour market. 2 4 0 2 0 0 1 6 0 So when we have data on average mark, we estimate an average mark effect 1 2 0 8 0 0 2 0 4 0 6 0 8 0 1 0 0 Average Mark UEA 7th March 2016 17

Hypothesis 1 Pay Average mark reveals ability to Employer. 2 8 0 2 4 0 If Econometrician also observes this mark, then estimated coefficient reflects true effect of ability on pay. 2 0 0 1 6 0 1 2 0 8 0 0 2 0 4 0 6 0 8 0 1 0 0 Average Mark UEA 7th March 2016 18

Hypothesis 1 Pay 8 0 But if Econometrician observes only Degree Class, then there appears to be a Premium by Class: might wrongly interpret this as a discontinuity. 2 8 0 2 4 0 2 0 0 1 6 0 1 2 0 0 2 0 4 0 6 0 8 0 1 0 0 Average Mark UEA 7th March 2016 19

Hypothesis 2 Pay Employer regards Degree Class as a Signal of some dimension of ability, over and above the average mark, or does not see (or heed) the average mark. Average Mark UEA 7th March 2016 20

Hypothesis 2 Pay Employer regards Degree Class as a Signal of some dimension of ability. 32 0 28 0 But if Econometrician observes only degree class, then we cannot rule out alternative interpretation of (steep) gradient in mark (Hypothesis 1). 24 0 20 0 16 0 12 0 8 0 0 2 0 4 0 6 0 80 1 0 0 Average Mark UEA 7th March 2016 21

A Regression Discontinuity framework offers the prospect of being able to distinguish between the two hypotheses but requires us to observe both degree classification and underlying marks. Hypothesis 2 Pay Employer regards Degree Class as a Signal of some dimension of ability. I A discontinuity would be indicative of signalling or statistical discrimination in the sense of the EL-SD approach a Average Mark UEA 7th March 2016 22

A Regression Discontinuity framework offers the prospect of being able to distinguish between the two hypotheses but requires us to observe both degree classification and underlying marks. Hypothesis 2 Pay Employer regards Degree Class as a Signal of some dimension of ability. I A discontinuity would be indicative of signalling or statistical discrimination in the sense of the EL-SD approach a Average Mark UEA 7th March 2016 23

Signal-Noise Ratio There might be a difference across subjects in the Signal- Noise Ratio associated with the extent to which the overall average mark indicates ability/productivity. There is some evidence that marks in quantitative subjects have a higher S-NR in terms of cognitive ability. Nature of quantitative material Nature of quantitative tests Labour market value of quantitative skills/abilities UEA 7th March 2016 24

Plan of Talk 1. Context 1: Context 2: Evidence and Policy Theory and Interpretation How might we interpret evidence of a premium by class of degree awarded? Why might any premium by degree class change across cohorts? UEA 7th March 2016 25

f(a) 0.15 1 H L O A 30 16 40 14 a 0.15 0.93 0.50 0.78 0 1 Ability distribution across broad educational groups; 1995 characterisation. UEA 7th March 2016 26

f(a) 0.15 1 H L O A 30 16 40 14 a 0.15 0.93 0.50 0.78 0 1 Ability distribution across broad educational groups; 1995 characterisation. UEA 7th March 2016 27

f(a) 0.15 1 H L O A 30 16 40 14 a 0.15 0.93 0.50 0.78 0 1 Ability distribution across broad educational groups; 1995 characterisation. UEA 7th March 2016 28

Plan of Talk 1. Context 1: Context 2: Evidence and Policy Theory and Interpretation 2. Institutional Arrangements 3. Data and Methodology 4. Results 5. Conclusions and Further Work UEA 7th March 2016 29

Regression discontinuity and degree class effects Also see: di Pietro (2012); Feng and Graetz (2015) We use individual student data on an anonymous university located somewhere near the centre of England Anonymised DLHE returns for graduate cohorts of 2011/12, 2012/13, 2013/14. Matched by personal id to extensive individual student records. Data include: Age, gender, nationality, fees status, course, department, previous schooling, family background, degree class, marks per module per year. Labour market outcome, 5-digit SOC, SIC, salary, degree class, location UEA 7th March 2016 30

Degree classification Typically, based on performance in modules in years 2 and 3. Overall mark in each module comprises marks in end-of-year examinations and in coursework assessments. All marking is anonymous. Exam boards consider cases anonymously. Each overall module mark is classified as follows: 0 40 50 60 70 m 40 50 60 70 Fail Third (3rd) Lower Second (2.2) Upper Second (2.1) First (1st) m m m m The student s final degree classification is based on the marks achieved in the individual modules. UEA 7th March 2016 31

Degree classification Historically, the student s final degree classification is based on two aspects of the module marks: Overall average mark across modules Distribution of module marks by class of mark However, for cohorts entering from 2008, the harmonisation of classification criteria across all departments means that the overall average mark is the main driver of the final classification: 0 35 40 50 60 70 35 40 50 60 70 Fail PASS Third (3rd) Lower Second (2.2) Upper Second (2.1) First (1st) m In addition, there are rules for borderline cases within 2 percentage points of each critical minimum mark. These rules are based on the class distribution of marks, marks in the final year, and marks in core modules. m m m m m There is also a system of mitigation. UEA 7th March 2016 32

Plan of Talk 1. Context 1: Context 2: Evidence and Policy Theory and Interpretation 2. Institutional Arrangements 3. Data and Methodology 4. Results 5. Conclusions and Further Work UEA 7th March 2016 33

3. Data and Methodology To date, we have data only for individuals who have responded to the DLHE in each cohort we are waiting student records on all students in order to establish the extent to which DLHE respondents might differ in observable characteristics from non-respondents. The DLHE response rate is approximately 63%. Results to be presented today exploit data on graduates who are in full-time employment and have provided Research-Accessible personal salary information. The usable response rate to the salary question is approximately 41%. UEA 7th March 2016 34

3. Data and Methodology Total responses by post/online % of responses by post/online Graduates Population Respondents Response rate 2011/12 8323 5312 63.82% 2404 45.26% 2012/13 8839 5623 63.62% 2670 47.48% 2013/14 8788 5425 61.73% 1677 30.91% -------------- Total ------------- 25950 ------------- 16360 ------------- 63.04% ------------- 6751 ------------- 41.27% UEA 7th March 2016 35

Population of All Leavers n= 25950 DLHE Respondents n= 16360 (63%) Online/Post n= 6751 (41%) Telephone n= 9609 (59%) PG n= 3515(52%) UG n= 3265 (48%) FT-Employed n= 1691 (52%) Further Study n= 931 (29%) OLFU n= 6751 (19%) Salary Data n= 1404 (83%) UEA 7th March 2016 36

All 3265 UGs (responding to DLHE online/post) Degree Class Freq. Percent ------------------------------ 1st 1,147 35.13 2:1 1,709 52.34 2:2 358 10.96 3rd 51 1.56 ------------------------------ Total 3,265 100.00 UEA 7th March 2016 37

UG Students. Degree Class breakdown by: All DLHE Respondents, FT-Emp, Salary Info Degree DLHE In FT Salary Class Resp Emp Info -------------------------------------------------- 1st 35% 34% 35% 2:1 53% 54% 53% 2:2 11% 11% 11% 3rd 2% 2% 2% -------------------------------------------------- n 3265 1691 1404 UEA 7th March 2016 38

Degree Class breakdown by Faculty Class Arts Science Soc Sci Total 1st 268 596 283 1147 37% 41% 26% 35% 2:1 424 604 681 1709 59% 42% 62% 52% 2:2 28 208 122 358 4% 14% 11% 11% 3rd 0 46 5 51 0% 3% 0% 2% Total 720 1454 1091 3265 UEA 7th March 2016 39

UG n= 3265 UG HEU+3/4-yr+08-11start n= 2791 FT-Employed n= 1517 (54%) FT-Employed n= 1691 (52%) Salary Data n= 1404 (83%) Salary Data n= 1292 (85%) Analysis will be based on a sample which excludes OS students, any students starting later than 11/12 and any students on UG degrees other than 3 or 4 year duration. UEA 7th March 2016 40

Probability of receiving Treatment (= Good degree) Cut-off at 60 All UGs in responding to DLHE online/post: (except: OS, course duration ~=3|4) Scatter Plot: Bins=30 Based on n=2705 1 .8 .6 good .4 .2 0 50 60 70 80 90 overall_ave UEA 7th March 2016 41

Probability of receiving Treatment (= Good degree) Cut-off at 60 Science UGs in responding to DLHE online/post: (except: OS, course duration ~=3|4) Scatter Plot: Bins=30 n=1293 1 .8 .6 good .4 .2 0 40 50 60 70 80 90 overall_ave UEA 7th March 2016 42

Probability of receiving Treatment (= First) Cut-off at 70 All UGs in responding to DLHE online/post: (except: OS, course duration ~=3|4) Scatter Plot: Bins=30 n=2705 1 .8 .6 first .4 .2 0 50 60 70 80 90 overall_ave UEA 7th March 2016 43

Probability of receiving Treatment (= First) Cut-off at 70 Science UGs in responding to DLHE online/post: (except: OS, course duration ~=3|4) Scatter Plot: Bins=30 n=1293 1 .8 .6 first .4 .2 0 40 50 60 70 80 90 overall_ave UEA 7th March 2016 44

Compliers and Non-compliers All: 2.1 versus 2.2 n=1707: Compliers=92% All: 1st versus 2.1 n=2422: Compliers=89% <70 1422 272 >=70 <60 281 136 >=60 2.1 2 2.2 2.1 2 1288 1 726 Science: 1st versus 2.1 n=1091 : Compliers=90% Science: 2.1 versus 2.2 n=751 : Compliers=94% <70 548 107 >=70 <60 201 46 >=60 2.1 1 2.2 2.1 1 503 1 435 UEA 7th March 2016 45

12 11 lannualpay 10 9 8 40 60 80 100 overall_ave Scatter plot of log(annual pay) versus Overall Average Mark All n=1193(based on 1292, trimmed by payband) UEA 7th March 2016 46

10 8 Percent 6 4 2 0 8 9 10 11 12 lannualpay Density of log(annual pay)) UEA 7th March 2016 47

12 11 lannualpay 10 9 8 40 60 80 100 overall_ave Scatter plot of log(annual pay) versus Overall Average Mark Science n=646 UEA 7th March 2016 48

RD Plot: cutoff=60 10.5 10 9.5 40 60 80 100 Sample average within bin 4th order global polynomial Data-driven RD plots See Calonico, Catteneo and Titunik (2014) Bin scatter plot of log(annual pay) versus Overall Average Mark All Cut-off 60 n=1193 UEA 7th March 2016 49

All (again) Cut-off 60 RD Plot: cutoff=60 10.3 10.2 10.1 10 9.9 9.8 50 60 70 80 90 Sample average within bin 4th order global polynomial Data-driven RD plots: See Calonico, Catteneo and Titunik (2014) Bin scatter plot of log(annual pay) versus Overall Average Mark IMSE-optimal quantile-spaced method using polynomial regression. (qspr) All Cut-off 60 n=1193 UEA 7th March 2016 50