Causality@Exeter: Seminar Series - Individualised Treatment Effect in Educational Interventions

The use of large-scale Randomised Controlled Trials (RCTs) is fast becoming “the gold standard” of testing the causal effects of policy, social, and educational interventions. RCTs are typically evaluated — and ultimately judged — by the economic, educational, and statistical significance of the Average Treatment Effect (ATE) in the study sample. However, many interventions have heterogeneous treatment effects (HTE) across different individuals, not captured by the ATE. One way to identify HTE is to conduct subgroup analyses, such as focusing on low-income Free School Meal (FSM) pupils as required for projects funded by the Education Endowment Foundation (EEF) in England. These subgroup analyses are usually not standardised across studies and offer flexible degrees of freedom to researchers, potentially leading to mixed results. While the uncertainties in estimates of treatment effects associated with subgroup analyses can be ameliorated by standard or even individual pupil-level data (IPD) meta-analyses, the synthesis of evidence from such methods relies heavily upon the comparability of outcome measures and the consistency in design and implementation across trials. In this talk, I demonstrate how the pooled effects for literacy and maths, through conventional and IPD meta-analyses of 88 large-scale educational interventions, helped decision-makers answer “what worked” for sure, when, where, and for whom. But they also posed new problems that I believe Individualised Treatment Effect (ITE) using machine learning holds the promise to solve. The findings presented in this talk have implications for decision-makers in education, public health, and medical trials.