Azara Blog: Research Assessment Exercise and Research Excellence Framework

Blog home page | Blog archive

Google   Bookmark and Share

Date published: 2008/03/01

Mike Baker says on the BBC:

RAE: they are just three little letters. If you live outside the intense world of universities they probably mean nothing.

Yet they represent something vital to researchers who are seeking answers to problems such as hunger, disease, social exclusion, or space exploration.

They dominate the lives of many academics, causing them thousands of hours of work, worry and discussion.

For university managers they represent both a great expense and the key to balancing budgets.

And for students they could be the cause of their university course or department being closed.

The letters are the acronym for the Research Assessment Exercise.

For 20 years this process has determined how the government allocates some £1.4 billion in cash to universities for research via the funding councils.

Now it is being scrapped. The then Chancellor, Gordon Brown, dropped this bombshell last March in one of his final acts before becoming prime minister.

The reason: the government felt it was too complex, costly, and burdensome.

Now, almost a year on, the search for a quick and easy system to replace it has become a complex wrangle and the topic of anxious discussion in senior common rooms.

Indeed, three new letters are now on everyone's lips: REF. They stand for the Research Excellence Framework.
So what are the arguments about? The initial plan was to 'radically simplify' the RAE approach.

The RAE involved panels of experts sifting through piles of published academic research in order to grade university departments on a scale from 1 to 5*. It absorbed a huge amount of labour.

So crucial was its outcome, that universities even spent large sums on consultants to help them do trial runs. Slipping from a 5* to a 5, or worse still to a 4, could spell academic ruin.

So the simple idea was to move to a system based on so-called 'metrics'.

Metrics involves measuring things in order to compare their scale. The initial idea was to base metrics around the amount of other income researchers attracted from industrial or charitable sources.

This approach was attractive to the government because it wanted a measure that was more sensitive to the needs of industry and the economy.

However, this measure was soon found to be inadequate for many areas, particularly outside industrially relevant research. So another piece of metrics was proposed: counting the number of times a piece of research is cited by other researchers.

Baker usually writes sensible columns and this one is no exception. He goes no at length about the various issues.

The problem with any system is that it is going to be gamed. The RAE was dreadfully gamed. So for example, each time a new RAE was looming, academic departments started hiring like mad, because all the papers for a given researcher counted towards the institution where the researcher happened to be at the time the RAE was done, not where the work was done. Then in between RAEs, practically nobody was hired. What a completely crazy system.

But the REF is bound to just give us more of the same. The first problem is that it is being designed by bureaucrats, who haven't a clue about research. And the idea that there is some kind of perfect metric just waiting to be discovered could only come out of the consultant mind-set of someone like Gordon Brown. Just the name itself, "Research Excellence Framework", reeks of the New Labour spin doctors.

When it comes to individuals we already have a reasonable functioning valuation system. If an academic wants to publish a paper then it needs to be peer reviewed. This system is not perfect by any stretch of the imagination, but it does mostly work. And everybody knows which journals are worth more (e.g. Nature) and which are not (e.g. take your favourite pick, there are zillions).

Unfortunately the same system does not work for departments. The RAE has pretty much demonstrated that conclusively. There are so few departments that the corruption factor ("you scratch my back and I'll scratch yours") looms large.

All material not included from other sources is copyright For further information or questions email: info [at] cambridge2000 [dot] com (replace "[at]" with "@" and "[dot]" with ".").