Thinking aloud. The San Francisco Declaration on Research Assessment (DORA) is undoubtedly a good idea, in my view. The thrust of the declaration states that research should only be assessed at the unit level, rather than at the level of the journal/venue. Certainly, I wish DORA was better worded to include more strongly disciplines where the specific measure of Impact Factor isn’t used (though it disavows “journal-based metrics” more widely). But nonetheless the spirit, if not quite the letter, of the declaration is sound. Good work can appear anywhere. Bad work can appear in top venues.
There is a real challenge in implementing DORA on the ground, though. The labour involved in hiring panels is heavy and this is why proxy measures of journal brand have come in to substitute for the quality of the work itself.
One suggestion for how we might fix this is to move to a mode of assessment where candidates for hiring present a research narrative in which they outline the impact, outcomes, and overall arch of their research, referring to a couple of key outputs, to which a hiring panel might turn and read in detail (the kind of “ImpactStory” approach). This sounds good in principle, even with the entirely valid concerns about the Impact agenda in the UK. It re-enforces the importance of understanding why we do research and what the work told us, while also moving away from relying solely on the prestige of the venue in which the work appeared.
The problem with this is the onus it puts on candidates. Applying for jobs is arduous, unpaid work, with a slim chance of a payoff in academia. I remember when I was applying viciously resenting every department to which I applied where they all required a unique, tailored statement, with their own individual online application forms. I applied for well over 90 jobs before I got my first post, almost all before I had my PhD (I treated it as a chance to refine the application 90 times), but the work involved was immense.
So my dilemma is this: in implementing DORA through displacing the burden onto researchers/applicants to narrativize their work, we do some good things. It is good that researchers should think more broadly about their work and that they can articulate this to a wide audience. This also gives those with a more quirky, non-prestige-based, track record a better chance of employment in academia (at least in theory).
On the other hand, this approach asks candidates to take on more work, in order to spare the work of hiring panels (who are employed members of staff). If candidates have disabilities, (child)care responsibilities, or a host of other life circumstances, though, this method once more privileges those who can afford to put the most time into a gamble on an academic job.
In short: I think we need to be careful in the implementation of DORA around equality and diversity. This is not to say that there aren’t massive E&D problems in the hiring system for academia as it stands at the moment. I would just like to raise this, though, in our consideration of who does the actual work of implementing DORA. We must make sure that it is those of us with posts, and not those hopefuls from whom we might exclude great candidates because we have displaced the labour.