Each year, the South African government, through the Department of Higher Education and Training, disburses significant funds to universities for the publications and post-graduate students (Master’s and Doctoral) that they produce. Each publication unit produced by academics – an academic paper, publishing conference proceeding or book, for example – is tallied and the total is then divided by the available budget. In 2024, the government will disburse R2.9 billion for publication outputs this way, allocated to universities based on the number of outputs. The graph below shows the percentage each university will earn from these outputs.
The DHET system is relatively unique because it predominantly rewards research quantity rather than quality. For example, a paper in the Quarterly Journal of Economics, first on one of the global ranking systems, would count the same as a paper in the South African Journal of Economics, number 348 on the list. This is despite the much greater effort and ability required to publish in the QJE, and the much bigger impact it is likely to have on the field.
This preference for quantity over quality has had the intended outcome of increasing the number of papers produced by researchers at South African universities. As the figure below demonstrates, almost all universities in South Africa have seen an increase in output in the last seven years, from less than 30 000 units in 2016 to almost 43 000 in 2023.
But has this increase been a net positive? Does it make sense for the government to spend money on more papers?
One reason to doubt the efficacy of this system of incentives is to think carefully about what type of problem science solves. As experimental psychologist Adam Mastroianni has recently argued, science is a strong-link and not a weak-link problem. ‘Weak-link problems are problems where the overall quality depends on how good the worst stuff is. You fix weak-link problems by making the weakest links stronger, or by eliminating them entirely.’ Food safety, he explains, is a weak-link problem. We want rules to ensure that the worst food does not kill us; we don’t need rules to ensure that we have at least one 3-star Michelin restaurant. A car engine is a weak-link problem. It helps little to have an expensive cylinder block if the pistons don’t work. And as anyone who has played volleyball would know, team sport is usually a weak-link problem: the other team will soon identify the weakest link, and concentrate their efforts on them.
But not all problems are weak-link problems. As Mastroianni explains, strong-link problems are those where the overall quality depends on how good the best stuff is, and the bad stuff barely matters. He mentions music. It would be bad to have a Bureau of Music that vets every single song recorded. That will produce unnecessary costs and delays and, ultimately, will only ensure that some ‘bad music’ stops being produced. But bad music, unlike bad food, imposes no cost on society. Instead, what we care about in music is creativity: we want artists to push the boundaries, perhaps even to the extent that a Bureau of Music might not approve. Sure, most of these boundary-pushing songs are rubbish, but those that succeed can be breathtaking, allowing us to experience an emotion that no previous music could.
Strong-link problems are surprisingly common, even if we don’t think of them as such. Mastroianni mentions a few: winning the Olympics (all that matters is how good the best athletes are), friendships (your closest friends are valued far more than acquaintances), and venture capital (the one that thrives pays for all those that go bust). Strong-link problems have high variance (because you really want more outliers at the top), no gatekeeping (because you might keep the best out), and risk (because there is little or no downside).
Science is a strong-link problem. Mastroianni explains clearly: ‘In the long run, the best stuff is basically all that matters, and the bad stuff doesn’t matter at all.’ One way to know whether something matters or not is to see whether other researchers ‘cite’ that work. Consider the following statistics: 12% of published medicine papers are never cited. In the natural sciences, that increases to 27%, and for the social sciences, further to 32%. It gets worse: 82% of published papers in the Humanities are never cited! Consider it the other way around: 0.5% of papers accounted for 20% of citations, 2.6% for 50% of citations and 7.2% of papers for 80% of citations received. (These numbers are for 2005, but the trend was worsening.) It is difficult to make an argument that those uncited articles deserve the same support as papers that actually do have an impact.
The already dire situation in which the South African government views science as a weak-link problem will be exacerbated by the emergence of ChatGPT and other cutting-edge artificial intelligence platforms, which threaten to widen the gap even further.
Wharton School professor Ethan Mollick recently asked ChatGPT4 to, well, write an academic paper. He uploaded a 60MB US Census dataset and asked the platform to ‘explore the data, generate its own hypotheses based on the data, conduct hypotheses tests, and write a paper based on its results’.
What happened? Well, it did exactly that, testing three hypotheses with regression analysis, finding one that was supported and continuing to check it through various statistical tests. And then, it wrote a paper about its findings.
Mollick summarises: ‘It is not a stunning paper (though the dataset I gave it did not have many interesting possible sources of variation, and I gave it no guidance), but it took just a few seconds, and it was completely solid. And that, again, is kind of amazing. I think we are going to see massive changes coming to academic publishing soon, as journals struggle under the weight of these sorts of real, but automatically generated, papers.’ And, let me remind the reader, these systems will only get better.
One likely outcome is the proliferation of scholarly output in South Africa over the next two years, especially in those fields that do not generate new data in laboratories. Whereas the most productive scholars could, depending on the field, do one paper every two months, ChatGPT, with the right prompts, will allow a paper a day. These papers won’t be pitched at the top journals, and they will very likely generate no future citations with no impact on our understanding of the world, but because they count the same as a paper in a top-ranked journal, all the incentives align to make an enterprising scholar fabulously productive – and rich. (Yes, authors do receive some part of the amount that’s paid to universities, although the exact amount varies considerably across universities.)
The abuse of the incentive scheme will also make life much more unpredictable for universities. Whereas the ranking of most universities has stayed the same the past few years – except for UJ, which has steamed up the list from eight to third – what will happen if a low-ranked university simply hires a few incredibly productive scholars who use ChatGPT to produce research output? Because the total budget is fixed, this could easily mean that a high performer like UKZN could earn substantially less next year, making any kind of long-term planning impossible.
The reasonable outcome is that, unless major adjustments are made, the system is likely to implode. This might seem concerning, but it could also pave the way for a transformation in how research funding is allocated, moving from a weak-link to a strong-link system. It would be more beneficial to steer funding towards researchers who make significant contributions. Let’s not reward things that don’t matter; rather, let’s set up our incentives to encourage the excellent and exceptional.
An edited version of this article was first published on News24. Image created with Midjourney v5.1. I thank Johann Mouton for providing data and feedback on an earlier version.