Are we losing our cutting edge?

A new paper indicates the UK's research base is becoming irrelevant, and that’s very frightening.

tl;dr

• Nightingale and Phillips cast doubt on the UK’s claims to be a science superpower because UK-authored publications are largely absent from the very cutting edge of key fields.

• If we are becoming globally irrelevant then that’s a big problem. Debating how to concentrate our resources is a useful route to finding solutions, such as improving our connectivity with the global research system and experimenting with different funding approaches.

• The citation-counting approach that N&P used is inherently flawed, but it’s the best we can currently do. Rather than deny the importance of cutting-edge work as a core component of a globally relevant research system, we should improve our understanding of research performance with increased support for the field of metascience.

Last week Paul Nightingale from the University of Sussex and James Phillips from UCL published a paper with some interesting new insights into the UK’s status as a leading science superpower. There has been some good discussion around this, and if you haven't read the paper, then it is well worth doing so. James Phillips includes a helpful bullet point summary on his Substack page (linked above), but the full paper is not a challenging read.

The paper is intended to be provocative, and this blogpost is therefore an attempt to formulate a response.

First, a summary of what I took from the paper. The headline is that Nightingale and Phillips (N&P) cast doubt on the UK's continued claim to pre-eminence as a scientific nation. They are concerned that the UK is falling behind in key fields, and the paper sets out reasons why this may be the case (discussed below). They recall COVID-19, noting that while the UK was outstanding in vaccine development and deployment using existing technologies, we had long missed the boat on the new technology of mRNA – a technology which promises much greater transformative impact through, for example, cancer vaccines.

Having dismissed other claims of the UK’s research prowess as unevidenced – e.g. because the REF outcomes are founded on marking our own homework – their paper zeroes in on one very overused data point: the UK’s impressive 13% share of the top 1% most highly cited research publications. The UK government (and other institutions) use this statistic liberally as evidence of the UK punching above its weight. On the surface of it, this is an impressive stat with abundant comms value for any science ministry!

N&P take issue with this. They argue that, in the global megaindustry that is modern research, 1% of the world’s output represents a huge stack of around 18,000 papers published annually. Such a large corpus cannot realistically represent the edge of the art – the truly blockbuster papers. Their analysis instead focusses in detail on the top 100 most cited papers in a small number of prominent fields, arguing that these 100 papers will more accurately reflect the most important work in a field than the many thousands of papers in the top 1%.

As I said, the paper itself is not a challenging read, and the data are laid out clearly, so I won’t repeat the results in full here. N&P’s key finding is that, at the narrow end of the bookshelf that is the top 100 most highly cited papers in the fields of AI and engineering biology, the UK sadly doesn’t much feature at all. Only by setting very generous parameters do N&P find a modestly better performance in the field of quantum. And where we do feature in these fields, we are clearly dependent on a tiny handful of high-performing institutions with peculiar characteristics, such as the privately owned DeepMind or the sacrosanct Laboratory of Molecular Biology.

Is it a problem if the UK does not feature heavily in the top 100 most highly cited papers? On this point, N&P point to a 2022 paper by Michael Nielsen and Kanjun Qiu that sets out how ‘outlier’ results may be driving scientific progress to a far greater degree than is captured by current comparative evaluation systems (in a research funding context). The Nielsen/Kanjun paper is worth reading in full, if only because it’s one of the most beautifully written metascientific analyses we’ve seen for many, many years! But I also commend that paper for offering new and interesting ways to think about research funding choices, identifying and working through the pros and cons of various scientific funding modes. This is particularly valuable because it helps to point to some ways forward from the situation identified by N&P.

The Nielsen/Kanjun paper also includes some neatly argued takedowns of quantitative comparative analyses of research per se:

Perhaps most fundamentally, it's not possible to neatly quantify the value of scientific discoveries in a commodity-like way. Indeed, arguably it's a mistake to attempt to quantify their value at all. Such discoveries are not fungible, nor do they add up. You can't measure the value of new scientific discoveries in units of milli-CRISPRs (say), and pile up three dozen 30 milli-CRISPR discoveries to get something more important than CRISPR. How many milli-CRISPRs was General Relativity? More or less than 27,000? It's all a little ridiculous.

This recalls some of the limitations with citation analyses that James Wilsdon et al. found when writing The Metric Tide in 2015. It was striking to me, when working on that project, to realise just how poor citations are as a measure of research quality:

• Papers themselves may not reflect the true extent of a research project’s findings and impact, particularly due to terrible tendencies in academic culture that value volume over quality and prestige over originality; 

• Citations themselves may not be reflective of a paper’s long-term scientific importance, with many important papers remaining unnoticed for years, or bad work becoming highly cited in refutation;

• The raw counting metrics such as Journal Impact Factor and h-index are so hopelessly skewed by a small number of trendy papers to be largely meaningless as an aggregate measure;

• The citation indices themselves have big holes in them where large numbers of citations, publications, or even the output of entire disciplines are missed by the scraping software;

• The citation indices do not always agree with each other on the number of citations any given paper has – with sometimes very wide variations in citation counts between different providers;  

• Some amazing research is never published, sometimes for dumb reasons, and sometimes for very good reasons.

It is of course entirely possible to apply such a critique to the work that N&P have done in their own paper. Does a handful of the most popular PDFs really represent the most important advances in a field? How confident are we in the publishing processes and organisations that have quality-assured and disseminated these PDFs? And how about the social infrastructure of journal editors, peer reviewers, issue article limits, article processing charge budgets, exhausted grad students doing literature reviews, disciplinary or departmental citation expectations, citation ‘Matthew Effects’, local citation clubs/rings, and so on – are we expected to believe that this complex and flawed mix of academic-cultural phenomena is reliably and robustly sending every piece of transformative work straight into the Billboard 100?

Clearly not, but the flaws inherent in a Billboard 100 approach should also be recognised as applying to that overused 13% of 1% stat. At least N&P are careful to caveat their analysis, and to corroborate it where possible. They are also clear that trying to address this situation by chasing an increase in highly cited papers would likely backfire, and I agree.

It would nonetheless be easy to dismiss all this sort of analysis as being based only on the flimsy evidence base of citation counts – a garbage-in, garbage-out interaction that therefore signifies only what we choose it to, and which further reinforces that famous streetlight effect where what gets measured is what counts, even if it doesn’t reflect the truth of the matter. (I’m not being particularly clever or original by highlighting these problems – I’d wager this discussion takes place regularly in every research and knowledge exchange committee in every university in the world.)

I can’t find the original source of this cartoon but you’ll likely have seen it before. It is a favourite of metascience powerpoint presenters!

It isn’t my intention to dismiss N&P’s paper on methodological grounds, because, despite the limitations and caveats, their paper indicates something very important about the UK’s claims to be a scientific superpower. To appreciate this, we need to accept that any definition of research excellence must include the great leaps in knowledge and understanding that can have profound and far-reaching impacts.

Consequently, today’s scientific superpowers are those which can lead with a very sharp cutting edge of research, a cutting edge of excellence that can repeatedly and reliably create a bow wave of transformative change throughout the global academic community. This should never be the sole definition of research excellence, but it is a vital component that we ignore at our peril.

Moreover, this is about the UK’s relevance as a global research player. The false precision of citation counting may rub us up the wrong way, but let’s not pretend that there is no link at all between citations and relevance. The N&P paper indicates we’re becoming irrelevant, and that’s very frightening. 

Nightingale and Phillips state that, if the UK is to regain our position as a leader, greater attention (and funding) is needed at the earlier stages of discovery research. This seems a sensible conclusion to draw – but how we approach this task is not straightforward. I couldn’t help being drawn back to some of the reflections on breakthrough discoveries that Nielsen/Kanjun put forward in their paper. In this, they note:

There's a romantic version of the history of science in which it's all about the great discoveries – Galileo and his telescopes, Darwin on the Beagle, and so on. It makes for fun and perhaps informative stories. However, some people retort that focusing on the great discoveries is actually misleading. Such "outlier" discoveries usually come out of a rich surrounding context of "minor" discoveries – small increments or even errors or false trails – which are later forgotten by all but historians, but which were crucial to the evolution of humanity's understanding.

Conversely, Richard Jones asked on Twitter if we should “support fewer researchers, but fully fund their work, rather than relying on cross-subsidies, and provide them with proper technical and engineering support”. This builds on a similar line of thinking in the recommendations N&P make. Jones states this is an uncomfortable discussion, but one worth having – and I agree on both counts.

Fans of the genre will note with a wry smile that we have landed on an old, familiar dilemma in research policy. Do we help the UK system to become more competitive by concentrating resources in fewer people or institutions, or does doing so put our research base at risk? Taking concentration to an extreme, do we risk hermetically sealing our top talent in a UK system that supports them in a more sustained way, probably for years past their prime, and running our research base into the same sort of dependencies that made us a world-leader in non-replicating viral vector vaccines but relatively nowhere on mRNA?

Well, that depends to an extent on how hermetically sealed the UK system becomes. To be a bit grandiose for a moment, we cannot escape our place in the international Republic of Letters – and nor should we shirk our responsibilities to it. I’m tempted to make a facile argument about the importance of UK association to Horizon Europe – but let me instead say that international connectivity is a permanent and inescapable feature of our shared global research culture, of which intra-European collaboration is just one component part.

The above argument is obviously not a satisfactory answer to Richard Jones’ question, nor does it really address the quoted point about ‘minor’ discoveries. But I hope it shows how thinking about how to concentrate our resources can be a useful exercise in helping us focus on what’s important. If the UK’s relevance to global research is, in part, a function of our connection to it, then perhaps we need to prioritise increasing the international connectivity of our research base.

Similarly, we might also alight on arguments about the importance of taking risks on fresh thinking, rather than overfishing the same pond of overfed researchers with a high h-index. Or we might conclude from the Nielsen/Kanjun work that we need much more experimentation with diverse funding models, or be willing to test some highly disruptive new approaches – as the UK is doing with ARIA. We might also decide to pioneer new approaches in AI-driven research, given the rapid advances we’re seeing in that area.

N&P propose a number of recommendations here, many of which merit further examination on their own terms. Some of these don’t necessarily follow from their own citation analysis, as N&P take a more wide-ranging view of what’s needed to support excellent people and institutions. I have no problem with this, and would add that there is no shortage of other viewpoints on what it will take to turn the UK into a genuine scientific superpower, all of which also need examining on their own terms. I have my own views on what we need to do, but I didn’t want to turn this blogpost into yet another indexed list of headings and priorities for UK science policy.

Nightingale and Phillips have held up a mirror. It is admittedly a distorted carnival mirror because it exaggerates certain features. But it’s only a more distorted version of the carnival mirror that we’ve been staring into for years – that hackneyed 13% of 1% stat, the REF outcomes, and so on.

If we do lack a proper understanding of how the UK system is performing, then there is at least a simple solution: we should prioritise a radical increase in our support for the field of metascience. The Nightingale and Phillips paper also states the importance of improving how we assess research performance, and yet I couldn’t help thinking that their paper is the most interesting thing to emerge in a while from the UK’s research-on-research scene.

That’s interesting in itself, because what Nielsen and Kanjun are doing via their Science++ initiative is on another level. Other US players like Open Philanthropy and Stripe Ventures have been getting increasingly active in this space. I have many further thoughts on this which I’d like to expand on in future, so I’m just collecting a few salient points here to put a few markers in the ground:

• I’ve noted calls from some areas to radically rethink our notions of ‘excellence’, and I’ve often heard people argue that research excellence itself is a flawed concept – or at least that it is contested. I worry that a possible endpoint of these debates may be to move us even further away from the cutting edge, or that the debate is used as a way to avoid making hard choices about where we need to improve.

• Excellence in applied research may not be sufficiently captured by our current metascientific approaches. Perhaps as a consequence, applied research is insufficiently supported by the current funding environment, and over the years I have seen that (with some exceptions) how the current research culture doesn’t welcome this sort of work readily.

• I’d also assert that our current evaluation practices (and administrative processes) may not be set up to cope with truly ambitious programmes that bring together multiple disciplines and partners. Yet these kinds of programmes are probably exactly what we need to be pursuing if we are to solve problems in a focussed way. I have a lot of sympathy with calls for new institutional structures to address these shortcomings, but we also need to fix the issues within the existing evaluation system.

• It is possible that the next wave of great advances in machine learning, cryptography, surveillance, bioengineering, robotics, or materials science will now take place in the private sector. (Deepmind publishes its work, but many businesses rightly will not.) A nation that can harness its knowledge-creating power to support a cutting-edge private sector will be able to gain significant strategic advantage – yet we currently can’t effectively track this, so in some ways it doesn’t count.

To conclude, the N&P paper forces us to reflect on what it means to be a great research nation. Maintaining academic relevance by publishing impactful work at the very cutting edge is a key part of this – and it would be highly perilous to ignore this dimension. But it’s clear that we need to explore the whole problem space much more deeply, including so that we can integrate other important dimensions of excellence into our metascientific worldview.

Understanding what greatness looks like in research is in itself a deeply interesting intellectual problem, and one that we should now prioritise. Those that crack this puzzle will be at a significant advantage.