Research quality and ways to improve it
The bar will soon be much, much higher
My HoD asked me to present a session to my department on ‘Research quality and ways to improve it’. Here’s a summary of my main points.
I should preface this talk by saying that I have never published in a top-5 journal, or a top-10, or even a top-50 economics journal. So take whatever I say today with that in mind.
1. The landscape
We are about to see the quality of research increase dramatically. If publishing was tough, it just got tougher.
But fast, sensible adoption of AI can help us compete, even from the southern tip of Africa. The productivity numbers are already non-trivial: in one trial, developers using GitHub Copilot completed coding tasks about 56 per cent faster; in an MIT randomised trial, professionals using ChatGPT finished writing tasks about 37 per cent faster while also producing higher-quality work. Broadly, people report time savings of up to 60 per cent on coding, testing and documentation. And those were older versions of LLMs; Claude Code has increased my productivity by at least 10x, if not 100x. (More on that in a future post.)
At the same time, the competition is fiercer. Submissions to top-5 journals nearly doubled between 1990 and 2012. Acceptance rates have fallen from roughly 15 per cent (around 1980) to about 6 per cent today. At the extreme, the QJE is closer to 3 per cent, with the JPE and RES around 5 per cent. Papers are also substantially longer than they were in the 1970s – about three times longer, by one estimate. So yes: the bar is rising, and it is rising fast.
My take: data and context will become more important, and that may be both a disadvantage (data are expensive) and an advantage (context requires a local footprint). (More in point 4.)
2. Tools and workflow
I use both ChatGPT and Claude Code. In the last month, my workflow has been completely overhauled. I expect it to be overhauled again in the next few months as these tools become more powerful and new tools appear. The one constant is change.
You cannot afford not to be on X, where many of these changes are discussed by leading economists, and you cannot afford not to play around with these tools. I will not be presenting how I use these tools (Matthew Olckers will do that next week), but I do want to emphasise that I use them constantly. This includes coding (unbelievably powerful), writing of my papers and blogs (this requires more selective use), and even app creation (I have created PubZub (try it out!), and a new performance appraisal app for our department). Scott Cunningham, who visited us in Stellenbosch last year, has a fantastic series on Claude Code. There are many, many other resources out there.
One practical lesson that is worth stating explicitly: think first, then type. You get dramatically better results than typing first and hoping the AI figures it out.
3. Tools are aids, but not sufficient
Tools are aids but they are not sufficient to produce quality research. One line of Scott Cunningham’s post is worth repeating: “Claude Code is extraordinary at executing ideas, but the ideas still have to come from somewhere.”
To do good research, you need good questions. And for a good question, you need to know what is being debated in your field. In my case: what are the leading economic historians thinking about? Then ask, humbly: how can I contribute to this question? In my field, I ask: Is there something interesting in South African or African economic history that would allow me to answer it?
This is really the only way to produce high-quality work. What it therefore requires is reading. Conclusion: know where the top authors in your field publish, and read what they write.
4. The dataset temptation
It is tempting to start with a dataset. (It is very tempting in economic history, because you might be searching in the archive for one thing only to discover another.) I will not say you should never start with a dataset, but you should never lose sight of point 3. Your dataset must still be in service of a question, and not the other way round.
That said, unique data will become more important as a way to differentiate yourself. Those of us based in countries where (a) public datasets are rarely digitised or available online, but (b) are available in archives or government departments, may have an advantage going forward. We should use it.
5. Methods, software, and staying current
Asking the right questions and having unique data are necessary but not sufficient for publishing well. I will not say too much about methods, as fields differ widely, but as someone with probably the poorest econometric knowledge in the department, I can confidently say that it is essential to make sure you try to answer your fascinating research questions with methods similar to those of the papers you read in the top journals. Do not just use methods you studied a decade ago. They may now be obsolete, or at least no longer persuasive.
Either learn these new methods (or abuse the AI tools and have them teach you these methods; they have infinite patience) or collaborate with someone who is skilled in these methods.
On software: my sense is that R and Python usage are rising and Stata is declining, partly because AI tools have deeper libraries in open-source languages. Yes, Stata still accounts for about 70 per cent of AEA replication packages. So the shift is real, but it is not complete, and you should be strategic about where path dependence still matters.
One thing about the workflow: I explicitly ask the AI tools to generate R scripts. This way, I can go through it myself, and I have a record of my analysis. Do not just use these tools to generate the output themselves, even if they can now do that.
6. Collaboration
I mentioned collaboration in the previous point, and this is something I think is underappreciated in economics, and perhaps in our department, too. There are many advantages to collaborating on a project, from using complementary skills (see point on methods above), to external validation for what you are doing, to someone checking up on you when the going gets tough, to verbally explaining your results, which is often the best way of coming up with new ideas and tests.
The data on co-authorship is also worth stating plainly. In the 1970s, roughly 75 per cent of top journal articles were single-authored. By 2014, single-authored papers were down to about 25 per cent, and the mean number of authors per paper rose from about 1.3 to 2.1 or more. Collaboration is not a niche strategy. It is increasingly the norm.
How do you find potential co-authors? It is really a mix, but here I would say two things help: (a) publishing is a very strong signal to likely co-authors that you are someone they would be keen on working with; (b) presenting your work at conferences within your field (in my case, AEHN, EHA and EHS, and, every three years, the WEHC) is where you actually meet these co-authors. It also requires you, surprisingly, to talk to people. That may not come naturally to many academics, but it is a vital part of the job. It is often called ‘networking’, which I think is a silly name. Think of it as making friends. You do not have to actually talk about economics; some of my co-authors I met sitting on a bus to a conference event and asking about their family background.
7. Funding
Finding co-authors is also a vital part of finding funding. South African funding schemes are tiny compared to European and US-based ones. Given my point earlier about the importance of data, funding becomes even more important, because data are expensive. There are many places to apply to (and our research office does an excellent job of circulating all these calls), but ultimately you almost always apply with someone for a project that you are already working on. (Put differently: successful funding is bottom-up rather than top-down.)
To move to associate professor and professor levels, finding funding is key. Once you have reached these levels, other funding options open up, like donor funding. I have been incredibly fortunate to have a handful of donors who have supported the work I do through LEAP, but all of that came after I was appointed as an associate professor. Causality, for me at least, runs clearly from producing quality output first to attracting this type of funding.
8. Student collaboration
One type of partnership that you can work on early is student collaboration. Working with good students can be an incredible source of inspiration, satisfaction and output. But it also creates challenges: students require a lot of investment, mostly in terms of time but also sometimes resources. For example, you really should send your PhD students to international conferences.
A great student can be a wonderful ally; a weak student can drain your energy, cause frustration, and lead to massive productivity losses. What differentiates a good student from a weak one? It is not really smarts (high grades, as a proxy). It is also not their background, or their personality type, or their desire to succeed. It is their ability to, despite challenges, get things done. That is it. How I proxy for that is my secret, but finding good students remains one of the cornerstones of building an academic career.
(I would add postdocs here as well, although they have typically already shown that they have the capacity to produce. They present different challenges: they are expensive, and they can suffer from point 9 below.)
9. The core of my message: getting things done
Which brings me to the core of my message: research quality is also about getting things done. The main challenge for most of the best people I know is not getting something on paper; it is sending something off for publication. Perfection really is the enemy of quality.
I am not saying you should just send every half-baked idea to a top journal. That will not help your reputation one bit. What I am saying is that you cannot sit on a paper for more than a few months without any progress. That is a signal that even you, the author, do not find the idea compelling. Why should a journal?
No, get yourself a good publication flow: have many ideas, start some of them, discard many quickly when they do not work (that’s why you need PubZub! Did I mention that it’s free?). Try to present your draft papers within a seminar series or at conferences. There is evidence that doing so is associated with a 10–15 percentage-point increase in publication probability. The feedback might be useful, but it is even better if people see what you are working on, as it invites collaborators and students (see previous points). (Also, as the tweet below makes clear, future seminars might be the place where other economists actually test whether you’ve understood the work you present as your own.)
Turn the paper into a working paper (the Stellenbosch series is great), and note another empirical result: open-access working papers seem to increase citations by something like 8–15 per cent. If you have a strong network, send the working paper to potential referees (and whose work you cite in the paper). Ask for advice on how to improve it.
Then, once you have incorporated key feedback, send the paper off and immediately begin on your next project. If the acceptance rate at top journals is around 6 per cent, you need a pipeline. At 10 per cent acceptance, you would need something like ten papers in motion to expect one acceptance.
10. Time
You might wonder where you would find the time to write all of these papers. Even with AI tools, time is the most critical resource. There are two simple strategies I use to open time. They are not for everyone, but they work for me.
First, make sure you do not spend too much time in meetings. This includes formal meetings (though I struggle with this one, as I sit on many committees), but it especially includes meetings with students and random office chats. You do not need to be a monk, but it sometimes helps to close your office door if you do not want any visitors. With students, keep meetings short and to the point. If your course requires lots of tutoring, get TAs. All of this also applies to emails. Here’s a tough one I struggle with but find super useful: Do not respond to emails for which there is an obvious answer elsewhere. Not responding raises the cost of sending the email to the sender (because it is basically free to send off that first email), and through others learning from your behaviour, you will get fewer and fewer unimportant emails.
There’s some evidence to back it up: in a large survey of principal investigators, scientists reported spending about 42 per cent of research time on administration and meetings. Another set of estimates suggests faculty spend only about 3 per cent of the work-week on primary research and about 2 per cent on writing, with around 17 per cent in meetings.
Second, and most important (and hardest), find time outside your working hours to write. Almost all my papers get written outside working hours. Many will know that I am an early riser, meaning that I typically write between 3 am and 7 am. I rarely do any work after 9 pm. Before kids, I used to work a lot of weekends. Importantly, I say ‘work’, but because I enjoy what I do, I have never really seen it as something I am compelled to do. More data to back my claim: faculty who write in brief daily sessions (30–90 minutes) produce three to four times more published output than binge writers.
Here is the hardest advice: if you do not feel like doing research when you have the time and you are at your sharpest, then perhaps a life in academe is not for you. Research really is something you want to do, not something you ‘have’ to do. (Of course, it might be that you are just not interested in the question, or that you have not yet found the right subfield. But if you have tried different things and still there is no energy, then find a job that does not require you to think about the thing you hate day after day.)
11. Luck
Finally, as I say on my new website, serendipity is underrated in science. There is no formula for producing high-quality research, which somewhat undermines what I wrote above. But the more time you have to explore – academic journals for questions, methods (and AI packages) as tools, datasets for answers, and classrooms and conferences for partnerships – the luckier you will get. And, by the way, get yourself a clean (and perhaps even beautiful) website. It helps a lot to be discoverable and therefore open to serendipitous encounters. Claude Code now makes such a website close to cost-free. And get a GitHub repository while you’re at it.
If you feel I’ve missed anything – or you disagree with something I’ve said – feel free to post a comment below. Happy to hear how others do it!]
Skimming down that list, Johan, I see you missed an important one: "Learn to write well."
And AI won't do it for you. Already I'm seeing text that looks good at a glance, but when I try to edit it I find it's mechanical and dead. The author used an AI tool to 'polish' it. Read an article in a top journal and you will be in touch with an intelligent and well-educated human being whose writing is sophisticated and distinctive and could only be the work of years of practice, aided by wide-ranging reading. And perhaps the help of a top-class human editor.
No one needs to be on Twitter. X is like getting married in community of property to someone you just met.