This week I presented presented a paper at #PESACONF2015 hosted at ACU Melbourne, which used computer simulation to argue that we should be very skeptical about infering much about school performance from student results. PESA was great! It was the best catered conference I’ve been to so if you like philosophy & education, get along to the 2016 edition.
It seems likely that most people believe schooling somehow affects student ability. After all, we as a society invest significant amounts of time and money in various endeavours trying to measure exactly this. Yet these endeavours face an epistemic challenge. Because we can’t measure the causal impact of schools directly, we can’t know this causal impact with certainty. Instead, we infer the causal impact of schools on student ability by way of proxy measures such as student results. If student results improve, then we can infer that some aspect of schooling caused this. Perhaps.
How warranted is this inference from student results to school performance? With the aid of computer simulation, I investigate the robustness of this inference mechanism in a variety of common scenarios. Simulation allows us to stipulate causal mechanisms that cannot be observed in the real word and measure how well our empiric inferences map actual causes. I show that when selectivity, either by student or school, is present, the inference mechanism from student results to school performance is very poor. And if our causal inferences fail when causes are known, they must also fail when causes are not known.
Written in Literate Coffeescript, this paper is an argument for epistemic scepticism about school performance measurement. Simultaneously a philosophical argument and a computer simulation that demonstrates the argument, it is best viewed in HTML as this offers the reader the chance to interact with simulation and visualisations. A static version suitable for PDF or print is currently under development.
You can view an interactive version of the paper & simulation at http://dave.kinkead.com.au/school-performance/.