In 2010 I published some thoughts on research, and on its promotion and assessment. The essay is entitled Research and how to promote it in a university.
The only revision required is that the UK’s Research Excellence Framework (REF) will now deliver its broad conclusion in late 2014, with details to follow in 2015.
Lessons can be learned following the enormous amounts of time and energy that have been diverted to REF. My view is that the whole process is seriously flawed.
Serious flaw 1: Institutions select their own samples of staff to submit as “Units of Assessment” (UoAs). Why introduce sample bias at the outset? Why not appraise all staff? Apart from the waste of time and effort, true comparison will be impossible, and the door has been left open for the unscrupulous to play the system.
Serious flaw 2: REF ratings are scaled down to individuals. This is absurd. The REF rating, whatever it means, is a property of an ensemble of individuals, not of any one – it is defined with words such as “all”, “majority”, “some” and “none”. None of us has a population density, nor a birth rate.
Serious flaw 3: UoA information sent to REF does not identify where the research was done, but where individuals happened to have been employed on 31 October 2013. Some institutions had hired and fired with the aim of skewing their results upwards. Some had not.
Serious flaw 4: The period of time during which “outputs” are eligible to be counted is too recent and too short (at 5 years) for any sensible comparison to be made. There has apparently been no attempt to calibrate. If applied retrospectively, REF would miss most major discoveries, and give weight to transient and lightweight findings – even to work based on assumptions that were later proved to be quite wrong. Hindsight reveals clearly that research output is not indicated by intensity of effort in searching for something that doesn’t exist.
Serious flaw 5: “Proxies”. These are acknowledged not to be the thing itself, but to be something that correlates with it. Even if a correlation is clear, it always requires qualification, and definition of boundaries of ranges. This is rarely done. When it is, it still applies only as a generalisation about populations, never to specific instances. See 4. And correlation is not causation.
Serious flaw 6: No baseline of comparison, leading to a spurious claims of objectivity. Metrics. If you ask someone the distance between London and Edinburgh, and the answer is given in hours, assumptions will have to be made before any conclusion can be drawn or comparison made with the distance to, say, Leeds. If the answer comes in litres of fuel, there still might be something to be learned – eventually. If, however, the answer is given in “metrics”, it would have been helpful to have received, instead, an honest “Sorry – I don’t know”.
There can be no doubt that REF has diverted resources away from the activity whose effectiveness it claims to measure.
I predict the outcome will contain few surprises, rather as if the whole exercise had been devised, and has been adjusted incrementally, by foregone conclusions about whether its components will yield a plausible answer.
Then there will be league tables. A few UoAs will move up and down. But not much. The sum of various UoAs will take their institutions with them. But not much.
And where changes in rank order are discernible, the question will remain: has anything actually changed? Or do the results reflect differing tactics, and choice of measures as targets?
So, why are we doing it?