There has been a major furore in recent days on social media about the rise in AI citations that have been hallucinated. These citations appear in manuscripts sent out for peer review, and even in the published literature.
This is a dire situation, of course. But it prompts in me some reflections on citation and its purpose. In particular: the problematic way in which we use referencing as a signaling mechanism rather than purely as an epistemological phenomenon.
The absolutely core fundamental problem here is actually not AI. Yes, it’s not great that LLMs produce workable references that are plausible in form. I wrote about this back in 2017!
The core problem instead is actually about whether we read or do not read the material in footnotes. It seems clear to me that we should be reading this material. I am somewhat bewildered and horrified at the number of people who were talking about this matter and saying that, of course, you don’t need to read the things you are citing. It is a fundamental part of scholarly activity to understand the context and existing work in one’s field! It is also an educational activity in its own right with intrinsic benefits of people knowing things, which is something to which I have dedicated my entire professional life.
Yet, there is a substantial literature on the phenomenon of “not reading”. In Close Reading with Computers, I wrote about this:
reading avoidance is nothing new: “not reading,” writes Lisa Marie Rhody, “is the dirty open secret of all literary critics.” (Lisa Marie Rhody, “Beyond Darwinian Distance: Situating Distant Reading in a Feminist Ut Pictura Poesis Tradition,” PMLA 132, no. 3 (2017): 659.) from Eve, Martin Paul, Close Reading With Computers: Textual Scholarship, Computational Formalism, and David Mitchell’s Cloud Atlas (Stanford University Press, 2019), p. 4.
There is more to read in the contemporary world than can be read within a single lifetime. Therefore, all reading is subject to a type of economic decision making that rests on time as the unit of currency that is to be spent by an individual. You are allotted a certain quantity of time in your life; an unknown quantity for sure. How you spend it with reading is up to you, but you have to accept that no matter how much you read, you will not read everything.
The contemporary academy, though, pushes researchers towards ever more output of publication as a proxy for assessing performance. In the cutthroat job market of higher education, it becomes imperative to have a solid publication record behind one’s name. This, of course, coupled with the above observations, leads people to take shortcuts, one of which is to seemingly use AI either to generate citations or to summarise a work using AI rather than reading it in full.
This is terrible scholarly practice but it’s not wholly driven by AI. It’s driven by the fact that footnotes have become symbols of scholarly work. They are not merely an epistemic lookup but are instead the foundational building block that shows whether this is “good scholarship” or otherwise. I have even been guilty of using this myself. One of the first things we do when checking articles at Orbit is to ensure that a manuscript has an adequate and comprehensive reference system in place. Unsurprisingly, this does often act as a good initial gatekeeping sweep.
If the system worked well, I can even envisage a world where the LLM could be actively useful for scholarship in this realm. I yearn and have done for quite some time for a discovery system that will let me say “I want to learn more about this field where should I start reading?”” and then get a list of books, journal articles and other materials that would form the basis of my investigation. It is very hard when you’re working in an interdisciplinary space to find where the start of the trail is in your literature review. My usual approach is to do a title search and dive in on one of those works and then look at the citations in its bibliography in turn and follow up there. There is no reason a computer could not help with this discovery aspect. The distributionally highlighting features of LLMs (i.e. they look for the most prominent and recurring features) should be really helpful when this is what you want to find: “who is at the centre of the network in Field X?”
But note the use I envisage is one where I would then go and get these books and articles and actually read them, because the whole point of this exercise is human knowledge. It is not about simply churning out papers one after the other to appease the academy’s endless desire in a publish or perish culture. Instead, it is about people learning stuff and people knowing things. Using AI to do your thinking will not achieve this.
I am not as fanatically anti-AI as many people are, although I am very scared for the future of labour and what these technologies might do to that. Because I think that they actually do perform better than most critics credit. I can imagine worlds where such technologies could be helpful, but I also think that at the moment we have a perfect storm of new technologies and cultural and economic conditions that create environments that are hostile to learning. In their typical way, the Silicon Valley folks have decided that this technology will be pushed everywhere in the world, without any due consideration as to what is appropriate in a certain space. I mean, it’s great if AIs can detect cancer cells. But do I think academic and student writing should be produced by an LLM? I certainly do not. Basically we need a great deal more care and consideration over where we use these technologies and what we use them for. Such has been the hype around AGI that this has been forgotten, because the “G” stands for General.
In the meantime I will continue to insist that the problem is not solely the LLMs, but scholars not checking footnotes and authors not reading the material they are citing. These behaviours have less to do with the technologies of the Academy but are more about the current crisis in universities worldwide, spurred on by governments who do not wish for an educated population. And we must not forget, as this post has emphasised, that the goal of what we’re doing is for real human beings to learn things, not for machines to abstract away that knowledge from us.