Martin Paul Eve bio photo

Martin Paul Eve

Professor of Literature, Technology and Publishing at Birkbeck, University of London

Email Books Twitter Github Stackoverflow MLA CORE Institutional Repo Hypothes.is ORCID ID  ORCID iD Wikipedia Pictures for Re-Use

This post is part of an ongoing series where I intend to develop my full personal (not institutional) response to the HE Green Paper. Comments are welcome to refine this.

The Green Paper asks in Question 26:

What are the benefits of the REF to a) your institution and b) to the wider sector? How can we ensure they are preserved?

I am responding here in a personal capacity and so cannot write on behalf of an institution; I will instead address the benefits to the wider sector and the general public.

Beyond the QR that it awards (and the crucial “blue-skies research” that this enables) and despite the disliked administrative overheads, there are many benefits of REF to the sector. Before enumerating these, however, I want to briefly comment on critiques of the REF. The Green Paper notes, as have many of Jo Johnson’s recent speeches, that there have been many academic critiques of the REF. Some of these critiques are merited; as a researcher myself I am all too familiar with the stress and strain that is put upon busy individuals to comply with these systems. They can feel intensely pernicHE ious from below and they have led to severe stress and a degradation of working conditions for many in universities. Universities also play many games, often at the expense of their staff, and ways to counter this should be considered. However, I do want to add that most of these critiques do not propose an alternative assessment framework; they propose the abolition of audit frameworks in general. I sympathise with these views to some extent. It is not right, though, to simply return to a system where a small number of institutions are given cash handouts to do as they please. On a pragmatic level, I also assume that the government wishes to hold institutions accountable for the public funds that are presented and will not take the recommendations of these critics. I believe, therefore, that it is slightly disingenuous for the Paper and the Minister to be citing critiques of REF that essentially wish for its abolition as a rationale to merely tweak the proposed framework. To take the critique but not the recommendation feels like having one’s cake and eating it. I proceed here on the assumption that the government wishes to continue with a research excellence framework and now enumerate the benefits of such an exercise by comparison to either its absence or to other modes of assessment (such as a metrics-based approach).

Firstly, the REF can act as a driver of positive behavioural change. This has been seen in both the impact and open-access aspects of the existing formulation. While impact is not a universally liked element of REF – and while it has added costs – it is worthwhile for institutions and researchers to think about the change that will be engendered by their work. It also helps institutions to craft narratives about the benefits that they can bring. The challenge here, of course, is to make sure that basic research, which may not lead to direct impact, is still valued. But by incentivizing transitional work and public outreach, REF can act to push the public benefit aspect of university research into the spotlight. Likewise, the centralised mandate of green open access for the next proposed REF is extremely important. Most publishers (around 97%) already have policies that allow this. By using REF to engender behavioural change and to encourage researchers to deposit (even if it is both a carrot and a stick) will ensure that UK research is accessible to the UK public. Both impact and open access are extremely positive benefits of REF to the sector and to the general public. Preserving the latter (an open-access mandate) in any future form of REF will be easier than preserving impact, but I would urge extreme caution over withdrawing this.

Secondly, the REF has the extremely strong benefit of being grounded in peer review. As recent reports have conclusively demonstrated, rushing to a metricised approach for REF could be extremely damaging for the UK’s science and research base. The fact that REF can avoid total concentration of outputs and reward good research, wherever it is found, through expert peer-review panels – carrying the confidence of the sector – is something that should be protected. While there is a temptation to rush to supposed cost-saving alternatives, these are false economies in several ways. For one, as above, they could end up devastating the quality of UK research. For another, though, the fact that metrics can only be supplied by a limited number of entities (Elsevier or Thomson Reuters) means that the “market” for metrics packages is an effective duopoly. This is likely to lead to institutions feeling compelled to purchase metrics packages to benchmark their own performance at whatever price these entities desire. This will, therefore, not necessarily end up being more cost effective than peer review since institutions will be at the mercy of an extremely limited and therefore dysfunctional market.

Thirdly, the REF has the advantage of being relatively cost effective. While there are disputes about the total cost, the recent report on the costs of REF put its overheads as far lower than the research council approach:

The REF assessed the outputs and impact of HEI research supported by many types of funders. In the context of £27bn total research income from public sources in the UK over a six-year period, the £246M total cost for REF 2014 is less than 1%. In the context of dual support, the total cost amounts to roughly 2.4% of the £10.2 billion in research funds expected to be distributed by the UK’s funding bodies in the six years, 2015-16 to 2020-21. This compares with an estimate of the annual cost to the UK HE community for peer review of grant applications of around £196M or around 6% of the funds distributed by the Research Councils.

As above, there are problems with REF; discouraging gaming by universities and attempting to alleviate the pressure put on researchers (via their institutions conducting expensive simulation REF exercises) should be seriously considered. But by contrast to other approaches, given that there is no prospect of unaccountable funding, REF has many benefits.