John F. Allen's Blog at jfallen.org

7th September 2014

Research Assessment and REF

Filed under: — John Allen @ 7:13 pm

In 2010 I published some thoughts on research, and on its promotion and assessment. The essay is entitled Research and how to promote it in a university.

The only revision required is that the UK’s Research Excellence Framework (REF) will now deliver its broad conclusion in late 2014, with details to follow in 2015.

Lessons can be learned following the enormous amounts of time and energy that have been diverted to REF. My view is that the whole process is seriously flawed.

Serious flaw 1: Institutions select their own samples of staff to submit as “Units of Assessment” (UoAs). Why introduce sample bias at the outset? Why not appraise all staff? Apart from the waste of time and effort, true comparison will be impossible, and the door has been left open for the unscrupulous to play the system.

Serious flaw 2: REF ratings are scaled down to individuals. This is absurd. The REF rating, whatever it means, is a property of an ensemble of individuals, not of any one – it is defined with words such as “all”, “majority”, “some” and “none”. None of us has a population density, nor a birth rate.

Serious flaw 3: UoA information sent to REF does not identify where the research was done, but where individuals happened to have been employed on 31 October 2013. Some institutions had hired and fired with the aim of skewing their results upwards. Some had not.

Serious flaw 4: The period of time during which “outputs” are eligible to be counted is too recent and too short (at 5 years) for any sensible comparison to be made. There has apparently been no attempt to calibrate. If applied retrospectively, REF would miss most major discoveries, and give weight to transient and lightweight findings – even to work based on assumptions that were later proved to be quite wrong. Hindsight reveals clearly that research output is not indicated by intensity of effort in searching for something that doesn’t exist.

Serious flaw 5: “Proxies”. These are acknowledged not to be the thing itself, but to be something that correlates with it. Even if a correlation is clear, it always requires qualification, and definition of boundaries of ranges. This is rarely done. When it is, it still applies only as a generalisation about populations, never to specific instances. See 4. And correlation is not causation.

Serious flaw 6: No baseline of comparison, leading to a spurious claims of objectivity. Metrics. If you ask someone the distance between London and Edinburgh, and the answer is given in hours, assumptions will have to be made before any conclusion can be drawn or comparison made with the distance to, say, Leeds. If the answer comes in litres of fuel, there still might be something to be learned – eventually. If, however, the answer is given in “metrics”, it would have been helpful to have received, instead, an honest “Sorry – I don’t know”.

There can be no doubt that REF has diverted resources away from the activity whose effectiveness it claims to measure.

I predict the outcome will contain few surprises, rather as if the whole exercise had been devised, and has been adjusted incrementally, by foregone conclusions about whether its components will yield a plausible answer.

Then there will be league tables. A few UoAs will move up and down. But not much. The sum of various UoAs will take their institutions with them. But not much.

And where changes in rank order are discernible, the question will remain: has anything actually changed? Or do the results reflect differing tactics, and choice of measures as targets?

So, why are we doing it?

13 Comments »

  1. I suppose we are doing it because politicians do not have to take the Hippocratic Oath, and instead are happy to try anything even if it means first doing harm. Then devising new harm to replace the prior harm. Ad infinitum. Perhaps all in the name of pursuing value for money, which would be sensible were not so much money and time (thus, more money) spent on the exercise.

    Comment by Michael Proulx — 18th November 2014 @ 2:32 pm

  2. Thank you, Michael. I tend to agree. However, “value for money”… What is the value of research, and to whom? Some managers explicitly consider money spent to be a research output. Knowledge advanced per unit of expenditure decreases accordingly.

    Comment by John Allen — 18th November 2014 @ 3:51 pm

  3. Concerning what to do with REF, there is an interesting discussion on BishopBlog: Metricophobia among academics. One proposal is to use an aggregate h-index.

    Comment by John Allen — 1st December 2014 @ 2:34 pm

  4. As evident from this work, any h-index would simply be proportional to the (square root of the) number of cites:
    http://www.ams.org/notices/201409/rnoti-p1040.pdf

    So if assessment is moving to aggregate / departmental h-indices, the departments will focus on producing highly citeable work.

    The problem at the core is that number of citations is a poor measure of scientific “success”, whatever that may be. To me, scientific success is finding interrelationships that may be employed by others to improve their own work. Success is seeing that others understand your finding and are interested in using it (paraphrasing Feynman there).

    Whether success is a scalar is another question altogether: is the Higgs Boson more of a success than the Nyquist-Shannon sampling theorem? How much more?

    Comment by Brian Pauw — 4th December 2014 @ 5:07 am

  5. I agree completely, Brian.

    I wrote in 2010:

    “Research output is new knowledge and understanding…Can it be measured? No. There is no simple, linear scale for any quantity corresponding to ‘research output’ in the sense used here. What would be its units? What arithmetical operations could be carried out on quantities of knowledge and understanding? Did Crick and Watson double or triple our understanding of the mechanism of heredity? What was the impact of Florey and Chain on medicinal chemistry, expressed as a percentage increase?”

    Opinion: Research and how to promote it in a university
    http://www.future-science.com/doi/full/10.4155/fmc.09.150

    Comment by John Allen — 4th December 2014 @ 5:40 am

  6. Though “value for money” is the catch phrase, there has never been a measure of “value”. It is about cost to the Treasury. To make a hash of a once oft quote phrase: They want to know the cost of everything, but they know the value of nothing.

    Comment by Dave Fernig — 3rd December 2014 @ 9:11 am

  7. Thank you, Dave. I agree. To the “bad apples” ignorant of science, “value” is reduced cost, or increased income, without attention to the consequences.

    Then income becomes outcome. Expenditure amplifies itself. The system will crash. This is common experience with, for example, uncontrolled positive feedback in sound amplification.

    Comment by John Allen — 4th December 2014 @ 5:54 am

  8. […] altogether more dangerous approach: we are on the path to redefine “good science” in terms more easily quantified and judged by bean counters. We are moving to a bastardized version where […]

    Pingback by The Dark Side of Science. | Looking At Nothing - A SA(X)S Weblog — 4th December 2014 @ 1:26 am

  9. “The Higher Education Funding Council for England’s funding allocations demonstrate that the entire system of research assessment is an expensive waste of time …. It makes virtually no difference to the allocation of funds to and between the research universities, and has not done so for 20 years.”

    – Sir Roderick Floud, former president of Universities UK, in Times Higher Education, 2 April.

    So, why are we doing it?

    Comment by John Allen — 4th April 2015 @ 3:57 pm

  10. […] John Allen on Research Assessment and REF […]

    Pingback by The Devil in the Detail | fanismissirlis | John F. Allen's Blog — 11th June 2015 @ 1:42 pm

  11. […] The Devil in the Detail | fanismissirlis | John F. Allen's Blog on 1.7 kg of knowledge […]

    Pingback by REF and Volkswagen’s cheating device | John F. Allen's Blog — 1st October 2015 @ 1:24 pm

  12. […] university A salute to whistleblowers – Peter Cameron’s blog Use of word in misuse of power Research assessment and REF Not extinct The bucket or the searchlight? The Devil in the detail – Fanis Missirlis’s […]

    Pingback by The Managerial University: A Failed Experiment? – Demos – John F. Allen's Blog — 24th April 2016 @ 12:25 pm

  13. […] F. Allen, ‘Research Assessment and REF’ […]

    Pingback by The RAE and REF: Resources and Critiques | Desiring Progress — 22nd September 2019 @ 3:12 pm

RSS feed for comments on this post. TrackBack URL

Leave a comment

Powered by WordPress