Here are my draft responses to the parts of the Consultation on the Second Research Excellence Framework that attracted my interested. These are my individual thoughts, not those of any institution that I represent. They are also not my final submission.
1. Do you have any comments on the proposal to maintain an overall continuity of approach with REF 2014, as outlined in paragraphs 10 and 23?
With respect to the timings proposed in paragraphs 23 and 24: the period until submission is now very short. Since the decisions will only be published in the summer of 2017, but that these decisions may include changes to staff submissions, it is difficult for institutions to plan their submission. As below, if all “research-active” staff are to be submitted, this has given institutions just two years notice on this front. Although other changes such as variable output numbers will mitigate this, the short submission period may put undue pressure on some staff via their institutions.
3a. Do you agree that the submissions guidance and panel criteria should be developed simultaneously?
The simultaneous development of submission guidance and panel criteria is welcome and would ensure more cohesion in this area.
3b. Do you support the later appointment of sub-panel members, near to the start of the assessment year? The later appointment of sub-panel members poses a problem. It is important, if REF is to maintain the support of the sector, that criteria and guidance represent disciplinary norms, even where there are contradictory opinions on what this represents within a field. It seems crucial, then, that the criteria appear as fully co-owned by sub-panel support in their development.
5a. Based on the options described at paragraphs 36 to 38, what approach do you think should be taken to nominating panel members? I support a continued nomination process here with the organisations listed to be excluded. Care should be taken, though, in some fields such as Library Information Science to ensure that commercial entities with conflicts of interest are also excluded.
5b. Do you agree with the proposal to require nominating bodies to provide equality and diversity information? I support the increased burden here for equality and diversity purposes. By some measures, the UK fares very badly on equality and diversity among the senior professoriat. To ensure that we work to counter this, the additional burden here seems worthwhile.
7. Do you have any comments on the proposal to use HESA cost centres to map research-active staff to UOAs and are there any alternative approaches that should be considered?
Amended response: There are some problems with using HESA cost centres to map research-active staff to UoAs. First, this seems to discourage researchers from moving between different areas of assessment. If other aspects of the policy are designed to encourage inter- and trans- discplinary research, this mapping could work against them. Second, HESA cost codes could also be gamed by institutions, assigning staff to the cost codes that best suit their return, in much the same way as it is proposed that such a measure would counter. Thirdly, the mapping would have to be subject to some kind of community consensus process, since there will undoubtedly be disputed edge cases.
8. What comments do you have on the proposed definition of ‘research-active’ staff described in paragraph 43? The definition of research-active staff here seems sound, especially with the exemption for RAs. In terms of the danger of contractual changes, a mitigating strategy would be to penalise institutions that proportionally reduce research-active staff by a certain percentage over the census period (i.e. take REF2014-timing HESA figures and then compare with REF2021 figures, offset/baselined against total academic staff at an institution).
9. With regard to the issues raised in relation to decoupling staff and outputs, what comments do you have on:
9a. The proposal to require an average of two outputs per full-time equivalent staff returned? The potential challenge that I see here lies in the disadvantage to smaller UoAs as opposed to larger departments/research areas. Through the deregulation here that does away with oversight of individual circumstance exemptions, in a smaller unit submission there will be fewer other members of staff to draw upon to make up for outputs in, say, cases of maternity leave. In larger units, there will be more staff and it will be easier to up the level by spreading outputs among other members.
I wonder, too, whether the claimed burden reduction for selectivity is here overegged. That is, institutions will still internally rank and evaluate outputs for submission, they will just not have the additional constraint of ranking within each submitting staff member.
9b. The maximum number of outputs for each staff member? I agree that a maximum cap is a good idea and the proposed limit of six here seems to work well.
9c. Setting a minimum requirement of one for each staff member? A minimum of one output is a good idea and better than zero. It does not seem right to ask for the return of all research-active staff members but then not to assess at least something from each of those individuals. However, this has a knock-on effect for the elimination of “individual circumstances” proposed elsewhere.
10a. Is acceptance for publication a suitable marker to identify outputs that an institution can submit and how would this apply across different output types? Acceptance for publication, marked by deposit within an institutional repository within three months, is a sensible timeframe for some types of output, but hardly for all. Monographs in the humanities, as just one example, can take a very long time to produce and so should be exempted if this measure is used.
However, a more sensible move may be to stipulate that new staff members at an institution may not submit outputs accepted within the first six months of their arrival. There could be a comment added that additional scrutiny on the time of deposit in relation to time of acceptance with respect to outputs submitted in the first year of a staff member’s hiring. (But this will require more monitoring.)
10c. Would non-portability have a negative impact on certain groups and how might this be mitigated? I believe that the arguments that non-portability would adversely affect certain groups, such as ECRs, is over-stated. Such arguments, when I have heard them, have circled around the problems of being hired. However, if no institution can expect to hire in order to use outputs, then the playing field is level. Providing exemptions to specific staff disadvantages others and will restrict the mobility of non-exempted staff. The benefits of non-portability incentivizing institutions to create a research environment that is conducive to producing good research and retaining staff seems more important here.
Note, however, that there is a problem if non-portability is not implemented but if the idea of taking an average number of staff across the census period is used as the quantity baseline for a UoA. This would make the situation of last-minute poaching much worse for many institutions who would suffer from having outputs withdrawn while still having to meet an average number of outputs for submission based on staff levels taken from across the census period.
10d. What comments do you have on sharing outputs proportionally across institutions? Sharing outputs would be a good solution to the problem of apportioning credit but I would note that it comes with additional monitoring requirements. The burden here is very high.
11. Do you support the introduction of a mandatory requirement for the Open Researcher and Contributor ID to be used as the staff identifier, in the event that information about individual staff members continues to be collected in REF 2021? This is a good idea but I will note, again, that the timescale for its implementation is (too) short.
18. Do you agree with the proposal for using quantitative data to inform the assessment of outputs, where considered appropriate for the discipline? If you agree, have you any suggestions for data that could be provided to the panels at output and aggregate level?
I continue to be concerned about the data quality and the use of citation measures to asses research quality. See the recent “Analysis of the international journal publishing activities in Switzerland with special emphasis on gold open access publishing” Data Paper by Max-Planck Digital Library, Big Data Analytics Group: Alexander Machado, Laura Hoppmann, Johannes Knaus & Margit Palzenberge at https://zenodo.org/record/167381#.WG-OqvGLT0o for an implicit critique of the SCOPUS database, for instance.
Furthermore, without reliable semantic information about the way citations are used, this method is not sound. For just one further example, see citation of Andrew Wakefield’s now-retracted and fraudulent paper “Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children”: http://www.thelancet.com/journals/lancet/article/PIIS0140673697110960/fulltext.
29. What comments do you have in relation to the inclusion of examples of impact arising from research activity and bodies of work, as well as from specific research outputs? I support this approach. I would also add that allowing impact activities conducted at an institution, based on a new staff member’s work at a previous institution, should be permissible.
36. Do you agree with the proposals for providing additional credit to units for open access? Amended response: I note that the proposal for extra credit to be awarded for superseding the basic requirements for open access in the environment is already part of REF policy but that, according to the FAQs as of Thursday 12th January 2017, HEFCE has “not yet developed the criteria for awarding credit”.
I welcome the incentivization of open-access publication and am pleased to support the idea of extra credit in environment where the default policies have been exceeded. However, I would like to raise the following as concerns/conditions for that support:
At least some guidelines on what this “extra credit” will actually comprise – and how it will be assessed – should be made explicit as soon as possible. Otherwise it is very hard for institutions to know what will and will not be rewarded. If the extra credit is supposed to change behaviour of institutions towards open access, then this needs to be put into place quickly. For instance, an example could be proportional budget re-allocation vs subscription offsetting, relative to an institution’s size.
I would caution against using crude metrics to assess for the award of such extra credit. For instance, extremely wealthy institutions may be able to evidence a larger spend on open access publications for their researchers. Taken in absolute terms, however, such measures merely measure existing wealth and the ability to spend it. Furthermore, institutions that do not have large (or any) humanities departments may be able to evidence a higher percentage of total outputs under open-access provisions, since monographs – which are acknowledged to be more difficult, and often more expensive, to make open access – do not feature heavily in their output profiles.
I would also like HEFCE to consider how various pieces of evidence for TEF around library provision intersect here with the policy. In such a competitive framework, institutions may wish to use the size of their exclusively held subscription library assets as part of their narrative evidence of provision to students. Such an incentive, however, runs directly counter to the goal of encouraging open-access provision.
43. What comments do you have on the proposed timetable for REF 2021? As above, I still feel that the timetable is too short given the lateness of this consultation.
44. Are there proposals not referred to above, or captured in your response so far, that you feel should be considered? If so, what are they and what is the rationale for their inclusion? I note that a previous draft of this consultation document, prior to the Stern review, considered splitting the 4* category in two and raising the number of stars that could be awarded. I thought that this was a sensible approach and I am sad to have seen it dropped from the final version.
Indeed, the problem for the panels if the output volume is dropped to two per staff member is ensuring that there is sufficient ability for the panel to discriminate. Restricting the levels to a four-star plan does not seem to address this.
I would also note that I am highly supportive of the proposal in Annex C to move towards open-access monographs for the REF after next. However, despite the convening of the UUK OA Monographs Working Group to address this, it will require substantial and ongoing intervention to begin to get this mechanism rolling. Additional help in coordinating pilot schemes and other initiatives would be greatly appreciated.