Research Assessment and REF

In 2010 I published some thoughts on research, and on its promotion and assessment. The essay is entitled Research and how to promote it in a university.

The only revision required is that the UK’s Research Excellence Framework (REF) will now deliver its broad conclusion in late 2014, with details to follow in 2015.

Lessons can be learned following the enormous amounts of time and energy that have been diverted to REF. My view is that the whole process is seriously flawed.

Serious flaw 1: Institutions select their own samples of staff to submit as “Units of Assessment” (UoAs). Why introduce sample bias at the outset? Why not appraise all staff? Apart from the waste of time and effort, true comparison will be impossible, and the door has been left open for the unscrupulous to play the system.

Serious flaw 2: REF ratings are scaled down to individuals. This is absurd. The REF rating, whatever it means, is a property of an ensemble of individuals, not of any one – it is defined with words such as “all”, “majority”, “some” and “none”. None of us has a population density, nor a birth rate.

Serious flaw 3: UoA information sent to REF does not identify where the research was done, but where individuals happened to have been employed on 31 October 2013. Some institutions had hired and fired with the aim of skewing their results upwards. Some had not.

Serious flaw 4: The period of time during which “outputs” are eligible to be counted is too recent and too short (at 5 years) for any sensible comparison to be made. There has apparently been no attempt to calibrate. If applied retrospectively, REF would miss most major discoveries, and give weight to transient and lightweight findings – even to work based on assumptions that were later proved to be quite wrong. Hindsight reveals clearly that research output is not indicated by intensity of effort in searching for something that doesn’t exist.

Serious flaw 5: “Proxies”. These are acknowledged not to be the thing itself, but to be something that correlates with it. Even if a correlation is clear, it always requires qualification, and definition of boundaries of ranges. This is rarely done. When it is, it still applies only as a generalisation about populations, never to specific instances. See 4. And correlation is not causation.

Serious flaw 6: No baseline of comparison, leading to a spurious claims of objectivity. Metrics. If you ask someone the distance between London and Edinburgh, and the answer is given in hours, assumptions will have to be made before any conclusion can be drawn or comparison made with the distance to, say, Leeds. If the answer comes in litres of fuel, there still might be something to be learned – eventually. If, however, the answer is given in “metrics”, it would have been helpful to have received, instead, an honest “Sorry – I don’t know”.

There can be no doubt that REF has diverted resources away from the activity whose effectiveness it claims to measure.

I predict the outcome will contain few surprises, rather as if the whole exercise had been devised, and has been adjusted incrementally, by foregone conclusions about whether its components will yield a plausible answer.

Then there will be league tables. A few UoAs will move up and down. But not much. The sum of various UoAs will take their institutions with them. But not much.

And where changes in rank order are discernible, the question will remain: has anything actually changed? Or do the results reflect differing tactics, and choice of measures as targets?

So, why are we doing it?

12 Replies to “Research Assessment and REF”

  1. I suppose we are doing it because politicians do not have to take the Hippocratic Oath, and instead are happy to try anything even if it means first doing harm. Then devising new harm to replace the prior harm. Ad infinitum. Perhaps all in the name of pursuing value for money, which would be sensible were not so much money and time (thus, more money) spent on the exercise.

    1. Thank you, Michael. I tend to agree. However, “value for money”… What is the value of research, and to whom? Some managers explicitly consider money spent to be a research output. Knowledge advanced per unit of expenditure decreases accordingly.

    1. As evident from this work, any h-index would simply be proportional to the (square root of the) number of cites:
      http://www.ams.org/notices/201409/rnoti-p1040.pdf

      So if assessment is moving to aggregate / departmental h-indices, the departments will focus on producing highly citeable work.

      The problem at the core is that number of citations is a poor measure of scientific “success”, whatever that may be. To me, scientific success is finding interrelationships that may be employed by others to improve their own work. Success is seeing that others understand your finding and are interested in using it (paraphrasing Feynman there).

      Whether success is a scalar is another question altogether: is the Higgs Boson more of a success than the Nyquist-Shannon sampling theorem? How much more?

      1. I agree completely, Brian.

        I wrote in 2010:

        “Research output is new knowledge and understanding…Can it be measured? No. There is no simple, linear scale for any quantity corresponding to ‘research output’ in the sense used here. What would be its units? What arithmetical operations could be carried out on quantities of knowledge and understanding? Did Crick and Watson double or triple our understanding of the mechanism of heredity? What was the impact of Florey and Chain on medicinal chemistry, expressed as a percentage increase?”

        Opinion: Research and how to promote it in a university
        http://www.future-science.com/doi/full/10.4155/fmc.09.150

  2. Though “value for money” is the catch phrase, there has never been a measure of “value”. It is about cost to the Treasury. To make a hash of a once oft quote phrase: They want to know the cost of everything, but they know the value of nothing.

    1. Thank you, Dave. I agree. To the “bad apples” ignorant of science, “value” is reduced cost, or increased income, without attention to the consequences.

      Then income becomes outcome. Expenditure amplifies itself. The system will crash. This is common experience with, for example, uncontrolled positive feedback in sound amplification.

  3. “The Higher Education Funding Council for England’s funding allocations demonstrate that the entire system of research assessment is an expensive waste of time …. It makes virtually no difference to the allocation of funds to and between the research universities, and has not done so for 20 years.”

    – Sir Roderick Floud, former president of Universities UK, in Times Higher Education, 2 April.

    So, why are we doing it?

Leave a Reply

Your email address will not be published. Required fields are marked *