Causal machine learning at Skills Development Scotland

Skills Development Scotland have a careers advice website and a customer support system which it uses to help individuals plan their learning and careers. 90% of school students in Scotland have an account with SDS, although the coverage reduces for over-18s. SDS keeps a record on an individual’s progress through education, training and employment up to age 24, and offers targeted interventions such as career coaching. SDS wanted to understand common pathways through education and employment, and identify in which areas or demographics interventions were making a difference, and how they could be better targeted.

Fast Data Science investigated the correlations between interventions and outcomes, attempting to control for the confounding effect of deprivation. The central question was especially tricky as the deprivation level of a neighbourhood is itself strongly correlated with both the outcome of an individual, and the interventions given to that individual. We experimented with a number of statistical models, as well as some predictive and causal machine learning models. We also put the data into Microsoft’s causal AI package DoWhy, and tried Bayesian networks, before looking at techniques such as instrumental variables.

We found that the interventions appeared to be well-targeted, and identified some strategies to move forward in terms of further data collection so that Skills Development Scotland could begin to build a causal model using methods either from machine learning or from econometrics (instrumental variables estimation).