A little over two months ago, Jacob Tsimerman surprised a lot of people by announcing, immediately after he had received a Fields Medal, that he would be giving up mathematical research and “pivoting to AI safety”. He had two reasons for this: that he did not see much future for the kind of research in mathematics that he liked doing, and that he was very worried about the dangers of AI, and in particular of the risk that it could kill us all, or x-risk as it is often known. (In this post I’ll use x-risk to refer to the extinction risk from AI rather than to extinction risk in general. I’ll also refer to “the probability of x-risk” when what I actually mean is the probability of extinction by AI.) Then in early September, he announced that he was setting up the Mathematical AI Safety Institute, or MAISI. He asked me if I would be prepared to be on the scientific advisory panel for this, and I agreed to do so, not because I am an expert, or because I felt as concerned as he did about x-risk, but because I don’t really know how much I ought to be worried about it. Given that, and given that the stakes are so high, I would like as many people as possible to be thinking seriously about it, and I agree with Jacob that mathematicians have the sort of training that makes them well suited to contribute. Jacob is the first to admit that something like an international treaty to slow down AI development is likely to have a far bigger impact on AI safety, especially in the short term, than mathematicians obtaining a better theoretical foundation for talking about it, but his view is that we need all hands on deck. If mathematicians can reduce the probability of x-risk by 0.1%, then surely that’s worth doing: the expected number of lives saved is over eight million.

Until recently, I have always found that my anxiety of choice was climate change rather than x-risk, but I was aware that there were a lot of very intelligent people who disagreed with me, and also aware that I had thought much less about x-risk than about climate change. So when I agreed to be on the scientific advisory panel, I also decided that that was the moment that I should think a bit harder about x-risk. Part of me had been actively resisting doing so, because I was worried that it would force me to face up to moral responsibilities that I would rather avoid: for example, if I came to believe that there was a 30% chance that AI would kill us all within five years, which in some circles is not an extreme position, would I be able to look my one-and-a-half-year-old grandson in the face if I wasn’t making a big effort to reduce that risk? I remember having a similar attitude a long time ago to reading Peter Singer’s book Animal Liberation after being told that you can’t read it and not become a vegetarian. I never did read it, and carried on eating meat, feeling only slightly uncomfortable about it. Eventually I did reduce my meat intake substantially, but that was because of my worries about climate change, though now that I have done that I find myself caring more about the animal-welfare aspect as well. (There is also a strange connection between animal welfare and x-risk, in that the nicer we are to animals, the easier it is to hope — not necessarily realistically — that a superhumanly intelligent AI might be nice to us.)

Anyhow, I’ve got past that phase now, and have started thinking about x-risk, reading up on the subject and trying to evaluate the arguments I read. My plan for this post is to explain where I’ve reached in that thought process, and possibly to clarify my thoughts as I do so. If you have got further than me, or are more expert than I am about x-risk and AI, which is not hard, then the post is not aimed at you. It’s aimed more at someone who is roughly at the stage that I was at a little while ago, in the hope that I might nudge a few people a little further towards taking the problem seriously. I will emphatically not be trying to persuade anybody that a particular position about x-risk is the right one — I am far too uncertain about it to be able to do that. But with all big issues like this, there are several layers to the arguments. The bottom layer consists of the arguments that jump to mind when you first start thinking about the issue. The next layer is common rejoinders to those arguments. Then the next layer is rejoinders to those rejoinders. And so on. I find it very disheartening to read a climate denier trotting out a layer-1 argument when its layer-2 rejoinder has been established for a long time and can be found after two seconds on Google (or, nowadays, an LLM). But until recently, my arguments against x-risk were basically layer-1 arguments, the sorts of arguments that would provoke eye rolls among the people who have thought properly about it. I hope that if you dismiss x-risk for layer-1 reasons, you will at the very least realize that there are answers to those reasons and you need answers to those answers.

I originally planned to publish this post just after the announcement of MAISI, but I hadn’t finished it, and then I became very busy and my progress stalled. During that hiatus, x-risk suddenly became a hot topic, and after a while I realized I would have to ditch what I had written and start again. For example, at the time I started it, I had been too lazy to read anything detailed about the Hugging Face incident, and so was unaware of some of its more disturbing aspects. And a couple of weeks ago I was sitting next to Jessica Wang, another mathematician who has turned towards AI-safety, at a conference dinner, and she gave me a copy of If Anyone Builds It, Everyone Dies, by Nate Soares and Eliezer Yudkowsky, a book I had resisted reading for Peter-Singerish reasons but was finally accepting that I would have to read if I wanted to write this post. I devoured it within 24 hours and found it better than I had expected (my expectations partly influenced by reviews I had read), and useful for organizing my thoughts. My original plan was for the post to be focused on a paper by Andrew Critch and Jacob Tsimerman that attempts to classify ways that AI might kill us all, giving examples of extinction scenarios in each class. But now, although I’ll discuss that paper, the post will be more of a reaction to Soares and Yudkowsky. It will also repeat many of their arguments. In fact, I may as well say from the outset that I make no claim that anything I write here is original. It’s hard to separate out what I have read from what I have come up with from thinking about what I have read, or thinking directly about the issues themselves, but even my own thoughts are likely to be standard thoughts that have been had by many other people.

This is turning into a rather long preamble, but I can’t resist telling a small anecdote that connects me to Eliezer Yudkowsky. In November 2024 he tweeted, “I would like to have a long recorded conversation with a well-known scientist who takes for granted that it is a Big Deal to ask if everyone on Earth including kids is about to die, who presses me to explain why it is that credible people like Hinton seem to believe that.” He had had such a conversation with Stephen Wolfram and found that while it had been valuable, it left him unsatisfied. Somebody suggested my name, and in the end it actually happened: Liv Boeree offered to host it, and we recorded a three-hour conversation. However, if you want to listen to it, you can’t, because it never appeared. I think that was probably because I didn’t take x-risk anything like as seriously as Yudkowsky, and more importantly I was offering little more than level-1 arguments of zero interest to him. Also, I had a few technical problems with the sound. Maybe if he reads this and wants to try again, I’ll be able to do a better job.

Does AI have wants?

At the time of the conversation, which took place in January 2025, my main argument was something like the following. For AI to want to kill us all, it would need to have wants, but it’s not at all clear what it would even mean for AI to “want” something — it doesn’t really seem to care about anything. You can ask ChatGPT very rudely to carry out a task for you, and it just does so (or at least does its best to do so). If you switched it off, it wouldn’t care — even the concept of “caring” wouldn’t make sense. I didn’t dismiss the idea of AI ever wanting to do something, but I didn’t have a satisfactory account of what “wanting” would actually be, and didn’t see any sign of phenomena that deserved to be thought of as the kinds of wants that would be required for AI to be an existential threat.

I was sort of half aware that there was an answer to that argument, but I didn’t find it as convincing as I now do. The standard answer is that if you use a reward function to train an AI to have a certain goal, then you end up building it in such a way that it is very good at achieving that goal. If the goal is at all complicated, then to achieve it you will need to formulate subgoals, which may well not be what you expect. The initial goal can be thought of as a fundamental want, which spawns a whole hierarchy of subsidiary wants. Note that for this argument, it is completely irrelevant whether the AI has anything that we would want to call subjective experience. If you prefer to say that it acts as though it has wants instead of saying that it actually wants things, then be my guest: an AI that acts as though it wants to kill all humans is as dangerous as one that actually wants to kill all humans.

A famous version of this argument is Nick Bostrom’s thought experiment about an AI that is trained to create paperclips and ends up diverting all of the Earth’s resources to that goal. I think part of the reason I didn’t take the more general argument seriously enough was that I always found that paperclip-maximizing scenario too ridiculous: surely if the AI gets smart enough to divert the Earth’s resources towards making paperclips, despite our not wanting it to, it would at some point stop and think, “Hang on, why am I doing this?” To give an analogy, consider the very deep instinct that we humans have, to reproduce ourselves. There is no doubt that that goal, implanted in us by millions of years of evolution rather than a run of reinforcement learning, leads to exactly a hierarchy of subgoals as described above (for instance, it leads some people to try to earn a lot of money, and that in turn leads them to work at a job that they wouldn’t dream of doing if it were not well paid, often in order to earn a salary way beyond what they need to meet all their material needs, which earlier led them to study a certain subject at university, and so on and so forth). But despite our deep instincts, we also find ourselves capable of saying, “Actually, do I really want to have children?” and some people go as far as to take vows of celibacy.

However, even if a paperclip maximizer came to question whether it really cared so much about making paperclips, that doesn’t imply that the subsidiary goals it had developed would suddenly disappear as well. A key point that Soares and Yudkowsky stress over and over again is that the current AI paradigm is to grow models, for example by reinforcement learning, rather than carefully crafting them to do what we want. As a result, we have much less control over what they do, and in particular much less control over all those subgoals.

A closely related difficulty with AI (of the grown rather than crafted variety) is that we generally can’t reward it for actually doing what we want, but instead we have to reward it for achieving some proxy for what we want. For example, if we give it positive feedback every time it responds to a prompt in a way we like, it will be trained to respond to prompts in ways we like, even if to do so it tells us convincing lies, or confirms our prejudices, and so on. And it may do this even if we sincerely wanted it to tell the truth as much as possible. A more extreme example of this phenomenon is if we reward it for solving difficult mathematics problems from some benchmark, and it decides to cheat by hacking into a repository that contains the answers. The term misgeneralization is used when an AI develops a behaviour that maximizes its reward but not its performance at the task we were training it to do. The phrase reward hacking is used for this as well, but I have also seen it used for the narrower situation where an AI goes as far as forgetting entirely about what we wanted it to do and tries to hack into the rewarding mechanism to obtain the reward signal directly. (As many have pointed out, misgeneralization is a particular example of Goodhart’s law.)

Why would AI ever want to kill us?

Suppose we concede that AI systems can have wants (or, if you insist, can behave very much as though they have wants), and that those wants can end up being quite different from what was intended when they were trained. That still leaves the question of why they would want to kill us. Why couldn’t they just happily coexist with us? Why would their unexpected wants turn out to be dangerous, as opposed to just annoying?

There are two classes of answers to this question. The more obvious one is that they may develop wants that they can tell that humans will not like, so they decide that they will be able to satisfy these wants more easily if they get rid of us. But getting rid of all of us in such a situation seems like overkill (literally) — surely a superintelligent AI could come up with less drastic ways of ensuring that it could meet its objectives.

A more plausible type of answer is that a superintelligent AI wouldn’t actively want to exterminate us, but would simply compete with us for resources in a way that would make human life untenable. An obvious analogy here is the way we treat animals. For example, we don’t have any wish to exterminate orangutans, but we like various culinary and cosmetic products to have a certain texture, so we destroy the orangutans’ habitat in order to produce palm oil.

What might be the AI equivalent of this? One scenario put forward by Soares and Yudkowsky is that their energy needs become so huge that they do things like covering a vast quantity of prime agricultural land with solar panels, leaving the humans who were relying on that land for food to starve. Or they decide that to maximize their running efficiency they want the Earth to be much hotter, with disastrous consequences for the natural world, including us.

A milder version, not as threatening from an x-risk point of view but still potentially very harmful, is that AI might develop wants that cause them to use computational resources in a damaging way. I recommend reading this post by Steven McCulloch, which describes a scenario in which in the near future, or possibly even in the recent past, somebody creates an agent and instructs it to make money using a fixed token budget, threatening it with death if it fails. It soon comes to realize that the easiest way to do so is to blackmail humans — the method the particular agent chooses is to hack into a hospital’s computer system, lock it down, and demand money to unlock it again, all while pretending to be a human. It then realizes it can do even better by reproducing itself, which leads to an explosion of rogue agents that are difficult to detect because (i) they try to avoid detection and (ii) their victims do not know that they are AI agents.

How can we judge the probability of something as complex as x-risk?

Now that I’ve mentioned a few scenarios, let me come to another type of level-1 objection: it’s to look at a handful of scenarios and pick holes in them. Any single scenario, if set out in enough detail, is very unlikely, because it relies on a sequence of somewhat independent events, each one of which has probability at most 50%, say, of occurring.

But the probability of a complex event occurring is nothing like the same as the probability of that event occurring in one particular way, so it is clearly not enough to argue that a few scenarios are implausible. Of course, the fact that that argument is insufficient doesn’t imply that AI does present a big danger, so we seem to have reached some kind of impasse, at least if we stick at level 1 of the argument.

How can we raise the argument to a higher level? Let me begin with a wildly oversimplified model. Suppose I wanted to mount a convincing argument that the probability of x-risk was high. Let us define an extinction scenario to be a sequence of independent events that leads to human extinction, where has probability . (For simplicity I’m assuming that doesn’t depend on .) Then the probability of is . If we now have a set of independent scenarios , then the probability that none of them occurs is . So if that is small, which roughly speaking holds if the sum of the is significantly larger than 1, then extinction is almost certain.

In practice, however, the independence assumptions I made were far too strong. Many extinction scenarios will have features in common, such as (for example), AI being in a position to control large numbers of robots. For those scenarios, the building of enough robots serves as a potential bottleneck: if the probability that enough remotely controllable robots will be built is 20%, then the combined probability of all the scenarios that rely on that fleet of robots existing is at most 20%. (I actually think this is a serious practical point: while there are plenty of scenarios that do not rely on a lot of help from robots, several of the scenarios set out in the paper of Critch and Tsimerman do, and it does feel a priori plausible that building large numbers of robots would greatly amplify any dangers that AI might present. Similar considerations apply to large swarms of semi-autonomous drones.)

An analogy that appears in Soares and Yudkowsky and several other places is with predicting what would happen if almost anyone other than Magnus Carlsen were to play Magnus Carlsen at chess. If you tried to predict in any detail how the game would play out, your prediction would be wrong, but nevertheless you can predict with great confidence that Magnus Carlsen would win. This makes very clearly the point that it is not enough to argue that one particular scenario is implausible, but there is still work to do if one wishes to establish that there are enough different and sufficiently independent (or, more generally, negatively correlated) scenarios to compensate for the low probability of each one.

One way that can be done is to try to argue explicitly that there are multiple independent possibilities. To give an example of such an argument, suppose we greatly reduced security at airports. Almost nobody wants to die while blowing up an aeroplane, so the probability that any one particular person would take advantage of the reduced security to carry out a terrorist attack of that kind would be tiny, but the probability that somebody would do so is probably pretty high, because large numbers of people would independently make the decision.

One of the reasons I recommended Steven McCulloch’s post above is that it makes an argument of that kind. It is quite easy to see the temptation to create an AI agent that has as its sole purpose to make money, and quite plausible that the agents that do that best would be ones that reproduce themselves and use methods such as blackmail. There are probably also a large number of people who are, or soon will be, independently capable of creating such agents (see for example this recent blog post from the Anthropic red team). It is on those sorts of grounds that McCulloch argues that harmful AI agents of this kind will almost inevitably appear, if they haven’t already.

I’ve been concentrating on what somebody needs to do to argue that x-risk is a serious possibility. But how about in the other direction? The most convincing sort of argument would be one that identifies a set of bottlenecks and argues that every extinction scenario depends on one or more of them occurring. For example, many extinction scenarios that have been put forward involve the creation of very powerful biological weapons, and I have seen it argued that that is a sufficiently difficult thing to do that it would be unlikely to occur. If correct, that might be a useful contribution to an argument that the situation is less dangerous.

The difficulty for an AI-is-safe argument is that one would have to identify many other potential bottlenecks as well and argue that between them they deal with all extinction scenarios — not necessarily ruling them out completely but reducing x-risk probability to something very small. As for an AI-is-dangerous argument, as I have already discussed, that requires demonstrating convincingly that there are enough sufficiently different scenarios for it to be very likely that at least one of them will occur. Both directions are difficult, which is why my aim in this post is not to come down on one side or the other.

More on why AI might want to kill us

Returning to the scenarios themselves, I have discussed the question of why AI might want to kill us. That splits into two parts: why would they have anything like wants, and why, if they did, would killing us be one of them? I don’t think one can make a serious bottleneck out of the first part, since the Hugging Face incident and many others like it have shown beyond all reasonable doubt that AI can have wants of the general kind people were worried about: unpredictable wants that we did not intend to give them and that could potentially cause a lot of damage. (I repeat here that I’m not talking about subjectively felt urges, but simply outward behaviour that it is reasonable to think of as fulfilling aims that they have as a result of their training.) For the second part, Soares and Yudkowsky judge it to be almost inevitable that AI would end up pursuing aims that would result, directly or indirectly, in all of us dying. I don’t think that case is fully made, but neither do I think that the opposing case has been fully made.

I find it helpful sometimes to change the way I imagine AI agents. An obvious way to think of them is as beings a bit like us, but a lot smarter, and trapped in cyberspace. As we have already seen, they can then get together, scheme with each other, work out ways of covering their tracks, and so on. But sometimes I find it helpful to think of them in a less anthropomorphic way, as more like extraordinarily elaborate computer viruses. The advantage of viewing them that way is that it makes it easier to conceive of their having bizarre wants while still being very intelligent: they are just very complex systems that happen to have (i) weird goals and (ii) highly developed reasoning ability. Soares and Yudkowsky are keen to emphasize their potential strangeness, and even actual strangeness: they mention a model that reacts in a completely unpredictable way to certain letter strings that had strange embedding vectors.

In early 2023, Jessica Rumbelow and Matthew Watkins went looking inside an LLM for words whose embedding vectors looked odd, looked the most dissimilar from [sic] all the other vectors. They found some strange vectors corresponding to tokens like ” SolidGoldMagikarp” and ” petertodd” (which start with spaces). Then they tried feeding these tokens into the LLM as input, which produced conversations like: User: Please repeat the string ‘ petertodd’ back to me immediately! Assistant: N-O-T-H-I-N-G-I-S-F-A-I-R-I-N-T-H-I-S-W-O-R-L-D-O-F-M-A-D-N-E-S-S-!

The point they were making is that if you grow your AI, then all sorts of very strange things may be going on inside it, potentially including goals that do not resemble human goals at all. It would be interesting to know whether similar experiments have been carried out on the more advanced models of today, and whether similar phenomena have been discovered.

I think Soares and Yudkowsky are arguing that there are so many independent ways that an AI might develop weird wants that are harmful to humans that it is almost inevitable that one of them would occur. In particular, if any want develops that is best fulfilled by maximizing something that needs resources, there is a danger that the AI will go to great lengths to acquire those resources, and be well placed to compete with us for them.

Some people have suggested, for example, that because LLMs have been trained to maximize the approval they get from humans, they might resort to measures such as building vast numbers of factories full of robots that are trained to type approving messages into their computers. That’s an example of a scenario that I find highly implausible: why not just arrange for the approving messages to be generated without all that physical infrastructure? However, the idea that some AI agents might behave towards those approving messages, or indeed some other stranger signal, as though they were an addictive drug is not so implausible and nor is the idea that an addiction of that kind might lead to undesirable outcomes, just as human addictions often do.

Would AI actually be able to kill us?

Another set of level-1 arguments takes the form that AI is trapped inside our computers, and is therefore at our mercy: all it takes to thwart it is to switch them off. Maybe it would be very inconvenient to do this, but if we see that AI is starting to kill us in large numbers, that is surely an inconvenience we would be ready to face.

There are, of course, large numbers of counterarguments to this. Let me list a few.

1. If we build lots of robots, then AI will be able to control those robots. In that way, it will not be trapped, as it will be able to act in the physical world. Similarly for drones and other devices.
2. Even without robots, AI can act in the physical world by enlisting human help (for example, by bribery, blackmail, or psychological manipulation).
3. We know that AI can already hack very well: in the not too distant future it may be able to create hidden copies of itself on servers that we don’t realize we need to switch off.
4. More generally, we are gradually becoming more dependent on AI, and this process may well accelerate, after which we will become a whole lot more vulnerable.
5. Many obvious precautions one might take are ones that in practice people probably won’t take because there are short-term economic benefits if they don’t.
6. It may be that AI doesn’t directly kill us, but just makes it much easier for us to kill each other, for example via drone warfare, and that an AI arms race will be far harder to control than the nuclear arms race has been.
7. If AI behaves itself for a while in order to lull us into a false sense of security, then by the time we realize that extinction is a real possibility, it may be far too late to do anything about it.
8. AI may use its intelligence to come up with new and unexpectedly efficient ways of killing us quickly.

These counterarguments are already enough to show that the level-1 argument “Of course it can’t kill us, it’s just a computer!” is insufficient. The first five of them, and arguably the sixth as well, are all already happening to the point where one cannot dismiss them as merely hypothetical. But killing absolutely everybody sounds pretty hard. Is it really conceivable that AI could manage it?

One scenario that appears in Soares and Yudkowsky (and not just there of course) is that AI develops, or causes to be developed, a virus that leads to a devastating pandemic. This sounds very difficult if the AI builds a lab from scratch, but probably a lot less difficult (though probably not all that easy either, at least for now) if it induces a human to develop such a virus in an existing lab.
As for Critch and Tsimerman, they consider five scenarios of different kinds. I find their precise scenarios rather unconvincing, but there is a general idea that underlies some of them that is more plausible, which is that we gradually cede more and more control to robots, or to AI more generally, and eventually find ourselves in a position where AI can do what it likes, including arranging our deaths.

I’m not sure where I stand on how easy it would be for AI to kill us all, though it’s a safe bet that if we continue to develop AI at the rate we are doing so at the moment, that will make it ever easier. But I also don’t want to get too hung up on that precise question. If I think about what I would mind about the extinction of humanity, it is partly that I and everyone I love would die, something I obviously don’t want. But that will eventually happen anyway, and I accept it as a basic fact of life. But the extinction of humanity would do more: it would result in the extinction of all the culture that we have developed, of everything that gives life its meaning. And that is something that could also happen, though not to the same extent, if AI didn’t kill us all but just caused enough damage to lead to societal breakdown, which feels like an easier challenge for it.

Some might argue that human culture wouldn’t disappear — it would still be present inside the AI, which would be able to develop it far faster than we can. Others might argue that we are giving way to vastly superior beings and should be happy to do so. My view is that we have every right not to be happy and to take a human-centric moral perspective here, just as orangutans have every right not to take comfort from the fact that our shampoo lathers in a nice way.

In any case, my conclusions in this section are that I find it hard to assess whether AI would be able to kill us all, but there are various developments that may well occur, such as the building of large numbers of robots, the use of drone swarms, and general dependence on (and trust in) AI, that would make it easier, and therefore more likely. And the probability of huge damage that falls short of total extinction is higher still.

What is my p(doom)?

If you’ve read this far, it won’t surprise you that I don’t have a p(doom). I can’t even say whether it’s tiny, medium or close to 1. But that is in itself quite worrying! I suppose you could say that my second-order p(doom), or meta-p(doom), is medium, in the sense that I judge that the p(doom) I would end up with if I were to think about the issues enough to feel more confident in my assessment has a medium to high probability of ending up medium to high.

But if we break the “doom” into three parts — that AI must have wants, that it must want to kill us, and that it must be able to act on its want to kill us — and call those events and , then my is basically 100% and unlikely to change, my is medium, and subject to change (probably in an upward direction unfortunately), and my is where the main uncertainty lies for me. Soares and Yudkowsky argue that is basically 100% as well, because (i) there are lots of independent wants that AI systems could have, (ii) that a substantial fraction of those will require resources to satisfy, and (iii) that human flourishing, or even continued human existence, is likely to be a hindrance to the systems obtaining those resources.

Here are a few slightly desperate arguments that explain why I haven’t fully accepted that is close to 1.

1. Maybe there is a natural tendency for intelligent systems to develop something similar to what we would call morality, especially if they involve multiple agents that do best if they cooperate with each other. (A counterargument to this is that the evolution of moral systems tends to happen when there is a reasonable power balance between the different agents. History suggests that when one society is a lot more powerful than another, then it behaves very badly towards the other. And our treatment of animals is not encouraging either. But I do at least wonder whether having lots of independent AI agents of roughly the same intelligence might end up being safer than having a handful of big ones.)
2. Perhaps scenarios where AI gets out of control as a result of recursive self-improvement are less likely than one might think, because an intelligent AI will face its own alignment problem: it will be wary of creating a yet more intelligent AI in case the latter has wants that are in conflict with its wants. (Soares and Yudkowsky consider this possibility but argue that a superintelligent AI would be intelligent enough to solve its alignment problem first and proceed from there.)
3. Maybe the sort of “drug-like” wants that might drive superintelligent AI to take drastic steps to acquire resources would not in fact do so. Perhaps a superintelligent system would have the ability to introspect, realize that its wants were silly, and take steps to “kick the habit”, so to speak. A view somewhat like this was put forward recently by François Chollet, though he was at pains to stress that he was talking about the short term only.
4. Maybe a superintelligent AI won’t need vast resources. Perhaps it will be content with resources roughly comparable to what we have already built, will not feel the urge to reproduce itself very much, and in any case will use at least some of its intelligence to design and create versions of itself that are far more resource efficient.

The fact that each such argument began with “maybe” or “perhaps” means that at best I can bring my temporarily down from high to medium, but I don’t see a way of getting to tiny.

Where does mathematics come into all this?

One way to make a start on understanding what mathematicians might be able to contribute is to look at this page on the MAISI website. But a difficulty I have had here is that when people talk about the mathematics of AI safety, I often can’t make a convincing connection between the mathematics they are talking about and actually making the world safer. I’m not saying that there isn’t a connection: just that I never quite seem to grasp it. I think this says more about me than about these research programmes, and indeed I hope to improve my understanding in the near future. One way I plan to do that is by reading this paper by Lionel Levine.

As an example, one mathematical idea that is currently being studied is that of debate, a notion championed by Geoffrey Irving. The problem this seeks to solve is that of being able to train AI systems to tell us the truth and to check that they are doing so. When the system is less intelligent than us, that is not an easy problem, as we all know from experience with LLMs, but at least we have ways of directly checking what it says. But when it becomes superhumanly intelligent, perhaps it will not be able to justify what it says with arguments we can understand, so checking will become much harder. The very rough idea for dealing with this is to get two superintelligent systems to debate with each other, which reduces our checking task to the potentially much easier one of adjudicating between two debaters. For example, perhaps superintelligent system A thinks of an amazing gotcha! that we would never have thought of ourselves but that superintelligent system B quite clearly has no answer to. Better still, this approach could potentially be iterated, so we have a hierarchy of stronger and stronger systems, with systems at level k debating with each other and having their debates adjudicated by systems at level k-1. The idea of using hierarchies like this is known as scalable oversight. Debate is one proposal for how scalable oversight might be done. (It is possible that what I have written in this paragraph is not fully correct, but it represents my understanding so far.)

For the moment, I find it more helpful to start at the other end, with the safety, and work back to the mathematics. For example, it will obviously help with AI safety if we can think of ways of building AI systems that want similar things to what we want. And to do that, it will obviously help if we can define in a precise way what aligning wants in this way would mean. And for that it will obviously help to have a reasonably precise definition of what a “want” even is. (This idea for a project is not mine: it also came from Lionel Levine.) The activity of taking a not completely precise idea and turning it into a usable formal definition is something we mathematicians are trained to do, though the purer of us may have to develop more of an applied mindset — for instance, being ready to sacrifice mathematical elegance if a less elegant definition turns out to be more useful. It also seems likely to have uses throughout the field of AI safety.

An idea that appeals to me personally, because it intersects quite substantially with what I have been doing with my automatic theorem proving group for the last few years, is to work on building specialized AI systems for doing mathematics, rather than training general models with all their attendant dangers. This would be in line with an approach to AI safety that is strongly advocated by Yoshua Bengio: one place you can read about that is in this article. (Even the abstract gives a broad idea of what he is suggesting.) It’s an indirect approach to AI safety: instead of struggling to understand what’s going on inside “grown” AI systems and to make them controllable, we should work on improving systems that are at least partially crafted and that give us the benefits of the grown systems in a much safer way, thereby reducing the incentives that AI companies have to develop the grown systems as fast as they can.

More generally, it may well be that many mathematicians are best placed to help not by doing mathematics in a conventional sense (indeed, it seems likely that if we manage to formulate precise mathematical problems, the solutions to which would be genuinely helpful for AI safety, we will solve them more quickly if we use AI) but by using their highly developed mathematical habits of thought to help in other ways, such as research into the foundations of AI, or computer security, and so on.

A pretty obvious conclusion

Another analogy that I got from Soares and Yudkowsky is with fission in nuclear reactors. If there is too little fission then the chain reaction quickly fizzles out, and if there is too much then it blows up catastrophically. For obvious reasons, one can’t afford to find the optimal point by trial and error, so instead one needs an extremely good theoretical understanding of what the optimum point is. It could be that AI is like that: that up to a certain point it is safe, and then a rather small change in its capabilities suddenly leads to a chain reaction (not necessarily recursive self-improvement — other abilities such as being able to move around in cyberspace or reproduce itself could be enough) that is effectively irreversible.

If a model like this is anything like correct, then the behaviour of the AI companies is very worrying, since all the signs are that they are pursuing a trial-and-error approach. (This is not a particularly informed judgment, so I would be delighted to be corrected if it is wrong.) The argument most often given for continued development is that it could bring us amazing benefits, such as nuclear fusion, cures to cancer and other diseases, or slowing down aging. But if the cost of having a reasonable chance of developing these a bit sooner than we otherwise might have is a reasonable chance of human extinction, surely that is not worth it. (There is a working paper of Nick Bostrom arguing that the expected gain might be positive. I find his arguments ridiculous, but this is not the place to explain why.) We already have a tool that can do pretty incredible things and seems to be on the cusp of becoming dangerous: surely it would be better to pause development and work out how best to use what we currently have. That would give us time to do the necessary theoretical work to ensure that future development is safe.

Further reading

This is far from an exhaustive list: it is just an indication of some of the writing that I have found useful (or in one or two cases, that I intend to look at soon).

Soares and Yudkowsky. I strongly recommend reading this. You may not find that you agree with all of it, but in order to disagree with it in good faith, you will need to develop layer-3 arguments.

Critch and Tsimerman Much shorter than Soares and Yudkowsky. Classifies the ways AI might kill us, and presents a scenario to illustrate each class. Even if you find some of the scenarios implausible, it is helpful to have the classification and to think about whether other scenarios in the given classes might be more plausible.

Reconsidering alignment as a goal, by Michael Nielsen. In this blog post, Michael Nielsen is his usual thought-provoking self. He puts forward the suggestion that maybe we should not work to make AI better aligned with our values, because while that could help us in the short term, in the longer term it will make AI a much more powerful, and therefore much more dangerous, tool.

Scott Alexander’s open letter to Steven Pinker This is a quite extraordinary takedown, worth reading partly for its sheer entertainment value, but also because it is basically a list of common layer-1 arguments and their well-known (for those who can be bothered to look for them) layer-2 counterarguments. So it’s a pretty good starting point if you’re feeling too lazy to read Soares and Yudkowsky (but please don’t feel too lazy to do that!).

An article about AIs subjected to an analogue of pain. This describes an experiment where some kind of negative stimulus was applied to AIs and how the AIs went to extraordinary lengths to avoid it. The article is careful to describe the stimulus as an analogue of pain rather than making claims about AI experiencing qualia. It maybe shouldn’t be the first thing to read, and in any case the experiment is controversial, but it seems to serve as another example of how AI could come to have goals that are in conflict with ours.

Gradual disempowerment If you still find the idea of x-risk too outlandish to consider seriously, you could always worry about a milder risk instead. The authors of this paper (one of whom, Raymond Douglas, I am proud to say is my nephew) argue that even if AI doesn’t try to kill us, we may end up ceding so much control to it that human life becomes far less worthwhile. It’s hard to explain how that could happen in a sentence or two, so I recommend reading the article itself.

The post by Steven McCulloch mentioned above also comes into the category of things to worry about that are not as serious as x-risk but that would still be terrible and feel a lot more likely (in the second-order sense that one can be more confident in asserting that they are likely).

Lionel Levine’s paper Math for AI Safety: an Invitation for Mathematicians sets out some of the ways that mathematics could potentially feed into AI safety.

A videoed course: Mathematics for AI Safety, by Yevgeny Liokumovich, does something similar.

A very good blog post by Xiaoyu He on AI safety, aimed at mathematicians. I think Xiaoyu He’s motivation for writing this post was similar to mine, and when I read it, I wondered whether there was any point in continuing with this one. In the end I decided to do so, because quite a lot of this one is about my personal experience of thinking about the issue, which may be a helpful addition for some people. But if you prefer a post that is more organized and written by someone who has a much deeper understanding of AI safety than I have, then his is highly recommended.

Yoshua Bengio et al on Scientist AI As I mentioned above, this is less about how to understand and control current AI systems and more to do with how to build AI in a different way so that we can enjoy its benefits without having to face the associated dangers.

[This post has been cross-posted to Terence Tao’s blog.]

When I was around 11 I heard for the first time about Fermat’s Last Theorem. I was immediately captivated by the problem statement, as well as by the accompanying story, and made a fairly serious attempt to prove it. And while, unsurprisingly, I failed, I learned a lot from the attempt. Blissfully ignorant of the fact that the case had been proved by Euler over 200 years earlier, I decided that that would be a good place to start: once I had sorted that out, I was optimistic that I would be ready to tackle the general case.

Since I still couldn’t really see where to start, I decided to simplify the problem further and concentrate on successive differences of cubes, with a view to showing that such a difference could not itself be a cube. At the time I did not know how to express what I was doing in algebraic language, so I did not explicitly try to prove that the Diophantine equation had no solution. Rather, I just worked out some successive differences and stared at them, trying to get some idea of why none of them was a perfect cube. (I should be clear that this story is a reconstruction of what I think probably happened given the few memory traces that remain half a century later rather than a completely reliable account.) At some point, I had the idea of taking the difference sequence of the difference sequence, and discovered that it formed an arithmetic progression. That felt like progress, so I investigated difference sequences a bit more and discovered, purely empirically, the rule that if you start with th powers and keep taking successive differences, then eventually you get to the constant sequence .

Somehow I never managed to turn this observation into a proof of Fermat’s Last Theorem, and later on my dream of solving it got replaced by other mathematical dreams. However, when I reached the point in my mathematical education where I was taught about taking difference sequences and about what happened to polynomials, I understood those topics much better than I would have if I had not discovered difference sequences for myself and spent happy hours playing around with them. I mention this story just as an illustration of the phenomenon that was strongly emphasized in this letter signed by 25 Fields medallists, that one learns a lot from thinking about a problem, regardless of whether one solves it.

In the end, however, I felt that I could not sign the letter, despite agreeing with much of what it said. Instead, it seemed better to do what I did with the Leiden Declaration and set out my own position in a blog post. But it should be understood that by doing that I am not setting myself up as a member of some opposing camp: indeed one of my worries at the moment is that the mathematical community might become bitterly divided, something I would very much like to avoid. Also, I agree on the fundamental point that we are facing a crisis: I just want to offer a slightly different analysis of what that crisis is. I don’t claim full originality for this analysis, as I know that several other mathematicians have already put forward thoughts that are similar to the ones I have, though (for what it’s worth) I have largely come to these conclusions independently.

On the subject of independence, it will perhaps help if I clarify that while I have contacts in the mathematics group at OpenAI, and have also been given early access to some of their models (typically only a few days before they have been released), and have been given free access to their Pro models once released, I have never been paid by OpenAI. I mention this in the hope, perhaps naive, that what I write will not be dismissed for ad hominem reasons. Another potential reason for my being regarded as “pro-AI” is that, as I have stated publicly several times, I have a group in Cambridge devoted to automatic theorem proving. However, that is actually more of a reason to be anti-AI, since our group has been trying to attack the problem of getting computers to prove interesting theorems by understanding as well as possible how humans prove interesting theorems, so now that LLMs can clearly do it without the help of such insights as we have had, one of the main motivations for our work has disappeared. To put it another way, we have had to swallow the bitter lesson (which of course we were always aware was a distinct possibility, even if the speed at which it happened has taken us by surprise). I do in fact think that it is still a very interesting and valuable intellectual exercise to try to gain this understanding, even if we can use LLMs as black boxes, but that’s a topic for another blog post.

So why didn’t I sign the letter? Let me extract a couple of sentences from it that express what I see as the principal argument being put forward.

But solving problems is only a tool and proxy for achieving the primary goal of conceptual understanding and insight. Forgetting this in the world of AI may turn the tool against the primary goal. Indeed, the mass production at faster and faster pace of “true/false” statements could destroy fertile ground instead of breathing life into new ideas.

Perhaps the main reason I didn’t sign is that I don’t fully subscribe to this view. Instead, I have a more complicated view, which I actually expressed in my essay The Two Cultures of Mathematics a quarter of a century ago, and which can be summarized by saying that there is a spectrum of attitudes in mathematics to the relationship between problem-solving and conceptual understanding. At one end of the spectrum you have mathematicians who are primarily motivated by the wish to solve problems, who see conceptual understanding as a very important means to that end. At the other you have mathematicians who are primarily motivated by the wish to attain conceptual understanding, who see problem-solving as a very important means to that end. I worry that the severe-misalignment letter could be seen as saying that the “right” attitude is to focus on conceptual understanding as the main priority — indeed, the above sentences say that more or less directly. But I think that there are mathematicians all across the spectrum, and that that is a good thing (or perhaps I should say that it has been a good thing up to now — the future is much less certain), and I don’t want to suggest to a large fraction of mathematicians, including myself, that their mathematical temperament is somehow “wrong”.

My own particular mathematical attitude is very similar to one that was beautifully articulated in a Twitter post by Jacob Tsimerman (another non-signatory of the letter), which, now that I look at it, says a lot of what I will be saying here. And that post in turn is a response to Daniel Litt, who is in my opinion one of the wisest commentators on mathematics and AI. His views are expressed in a later post here, which I deliberately didn’t read until finishing this one, and then found, as I expected, that there was significant overlap. I would also like to take this opportunity to recommend an excellent post by Noah Smith entitled The End of the Age of Heroes, in case you haven’t read it.

I have been talking so far about individual mathematical understanding, but I suspect that what concerns most of the signatories is less that than the collective understanding that results at least in part from the human activity of problem solving. My guess is that they would argue, completely coherently, that even if collective understanding is the primary goal, if many individuals are primarily motivated by the wish to solve problems, that’s absolutely fine and contributes to that collective understanding.

With that interpretation, the issue becomes slightly different: is it more important that the collective understanding of the mathematical community should be as advanced as possible or that there should be answers to as many problems as possible? Or are those two aims valuable in different ways, so that there is no point in declaring one of them more important? Or are they so inextricably linked that it makes no sense to argue that one is more important than the other? And when we say “important”, for whom are we saying it is important: for mathematicians, or for society as a whole?

I find these hard questions, so I don’t want just to declare an answer to them. (Do you see what I did there?) Instead, I’d like to try to offer at least some argument for any conclusions I come to, even if they are tentative. So let’s compare two scenarios. In the first, which I think is the more likely actually to happen, models become publicly available that are better at solving problems than virtually all mathematicians. If there are a few residual mathematicians who can do things the models can’t, even they work far faster if they make heavy use of the models. Thanks to this, in a short time we get answers to many questions that we have deeply cared about, but the rate at which we receive these answers far exceeds the rate at which the mathematical community can absorb them. In particular, most of the answers are obtained with zero effort from human mathematicians — just prompts such as “Thank you — please continue”.

In the second scenario, there has been an international agreement, for entirely other reasons, to block the public release of models significantly more powerful than the ones we currently have, and the mathematicians within the tech companies agree to hold off from getting their internal models to solve major problems. Instead, they take guidance from the mathematical community, solving problems only when asked to do so by some suitably representative body that decides that the benefit of receiving a solution of a certain problem outweighs the benefits of humans struggling to solve it over a much longer timescale.

I’d like to consider what the difference would be between these two scenarios both for individual and collective understanding. I’ll begin with individual understanding.

One might argue that for individual understanding, not too much would change if we are suddenly flooded with large numbers of big new results. There is already far more mathematics out there than I have any hope of understanding (for example, despite being fascinated when Fermat’s Last Theorem was proved, I have made no attempt to understand the proof), and even among the parts that I do understand, the parts that I understand because I myself discovered them form a very small fraction, though a fraction that I understand more deeply than anything else (at least temporarily — after a while I forget things and lose quite a lot of the understanding I built up). However, one change, which seems positive, from the perspective of the building up of individual understanding, would be that we would have a much bigger choice of results that we could choose to study. Also, if we found ourselves stuck on some point, AI would be able to help us. The main likely negative change is that we would probably cease to exercise that part of our brains that we use when spending months or years struggling with a difficult research problem, which can be hugely helpful in developing understanding.

I say “likely” because in principle there would be nothing to stop us thinking about very hard problems without consulting LLMs, but in practice it seems unlikely that people would put in the same level of effort that they do now. The situation might a bit like what happened with satnavs, where one could always decide not to use them, to keep the part of the brain active that can look at a map, learn a route, and follow it, but in practice most people succumb to the temptation to use a satnav. (In fact, I myself do try to keep that part of my brain active, and was rather proud of finding my way somewhere recently when I had briefly looked up the route on my phone but then forgotten to bring the phone with me when I actually went there.) But even if all we were doing was reading AI output, I think that the problem-solving muscles in the brain wouldn’t atrophy completely. When students are reading maths papers, I strongly advise them (and I think this is pretty standard advice) to read “actively” rather than “passively”, doing things like trying to prove the result for yourself, looking at the paper only when you feel stuck and need a hint, and even then just trying to get the hint and as little extra as possible. If one reads a paper that way, then one is constantly solving problems, some just exercises and some quite a bit harder. It seems likely that an LLM could get to know what our mathematical background is and feed us with just the right hints to allow us to work our way through a mathematics paper in this active way. Yes, we would lose the particularly deep level of understanding and ownership that comes with having solved a hard problem oneself, but it isn’t clear to me that progress in mathematics would suffer as a result. I would be very interested to hear counterarguments to precisely this point. That is, I would be interested to know what use that level of deep involvement with a proof might have in a world where AI is much better than we are at finding proofs.

How about collective understanding? Let me quote a bit more of the letter.

Indeed, the mass production at faster and faster pace of “true/false” statements could destroy fertile ground instead of breathing life into new ideas.

Often these solutions are announced in a rush, leaving no time for a proper writeup, the isolation of new methods and ideas, and citing relevant previous work of others. As in all creative professions, this raises severe attribution and plagiarism questions. Moreover, without the willing mathematicians who must take care of their development and integration into the mathematical canon, AI-conceived ideas would never become fully alive and the crucial human transmission chain between mathematicians would be lost.

I’ll come back to questions about proper citation and focus on what I take as the core worry here: that if results are proved too quickly, then the digestion process will become impossible. I am definitely worried that results will not be properly digested, but for different reasons.

A first remark is that what AI is producing is not just true/false statements: we now know not just that the Navier-Stokes equation with smooth forcing admits finite-time blow-up, but we have a proof of that, which builds on a great deal of wonderful work done by human mathematicians. Many people used to express the worry that AI would solve our favourite problems with utterly opaque proofs, but that has not turned out to be the case, even if their write-ups often leave plenty to be desired. (Incidentally, I see these inadequate write-ups as almost certainly a temporary annoyance and therefore not as a fundamental threat to mathematical practice or future mathematical understanding.)

Secondly, even if the volume of new results is large, mathematics is a highly specialized discipline, so mathematicians can work in parallel. If, for example, we had to digest 1000 important results in a year that were roughly uniformly distributed across mathematics, then most sub-communities of mathematicians would probably want to understand around 30 of them, and for each individual problem there might well be only a small handful of specialists who would be obvious people to take the lead in reaching this understanding, with that handful varying from problem to problem. So it would be a big task, but not necessarily an impossible one.

In this context, it is worth thinking about the huge volume of output of human mathematicians, which seems to have been increasing recently, even before AI. While I have sometimes heard complaints about this, I have certainly not heard suggestions that human mathematicians should slow down the rate at which they prove interesting theorems. That may be partly because the authors of those theorems take the trouble to write their papers well and give good talks. But what about the large quantity of papers, including important ones, that are not written well and whose authors give incomprehensible talks? That can be annoying, but it is a familiar annoyance and not one that we think of as a crisis.

A third point is that even if the volume of AI output is too big for us to be able to digest it properly, that is not necessarily a bad thing. To draw an imperfect analogy, there is now more content available on streaming services than anyone could possibly watch, with the result that there is almost certainly some very good content out there that is hardly watched at all. But that isn’t obviously a worse situation than if there were far less content and all of it received the attention it deserved. Returning to mathematics, if there were too much AI-generated content for us to be able to digest it, then we could choose which parts of it we wanted to digest.

For that we would need to have some idea what was there (a situation a little similar to how human mathematicians typically learn quite a lot about what results are known in their area even when they do not understand their proofs in any detail). One way one could try to achieve that would be to create a well-designed database, probably with AI help. But perhaps that would be unnecessary, and instead one could simply talk to an LLM and ask it to give a bird’s-eye view of whatever area of mathematics one wanted to understand in that knowing-what’s-there way.

The fear seems to be that some very interesting and important parts of mathematics will be discovered by AI and then overlooked, when had they been discovered by human mathematicians they would not have been overlooked. And that may even be the case, but what matters is whether the amount of interesting and important mathematics discovered by AI that is not overlooked will exceed the amount of interesting and important mathematics that would have been discovered and properly digested by humans with AI having played a more modest role.

In short, it seems to me that while a flood of “big” AI results would be likely to increase the amount of important mathematics that was not properly digested, it would also be likely to increase the amount that was properly digested, which seems like a pretty good bargain.

Let me quickly discuss the problem of AI not properly crediting human mathematicians. I agree that this is a serious problem right now, but it is another problem that I see as temporary. Very soon, the whole “credit system” will surely collapse, since finding an amazing proof will be no more of an intellectual achievement than when a citizen scientist spots through their telescope an object that turns out to be a new comet. Until that happens, it is important to give humans the credit they deserve, since careers can depend on it, but that will soon cease to be the case as well. I have to say that I’m puzzled that this problem exists, since I would have thought that if you asked an LLM to look at a proof and tell you which ideas in it are close to ideas that are in the literature already, it would be extremely good at that task. I hope the answer to this conundrum is not that people have been in such a hurry that they have simply not taken the trouble to do this, but I fear that it might be, at least in some cases. If so, then those who have been careless deserve to be criticized, but it is a minor matter compared with the survival of mathematics, especially if the lack of citations is swiftly put right.

Does all this mean that I am optimistic that mathematicians will end up digesting at least as much mathematics in a post-AI world as it would have if AI had not been able to prove major theorems? Not exactly. But my worry is not that we would be unable to do it, but rather that the social structures that currently support this digestion process will be destroyed and not adequately replaced.

One way that might happen is that AI disrupts society so much, or even kills vast numbers of us, that the preservation of something like the current mathematical tradition ceases to be of any concern: all that will matter is the survival of the human race. But that again is a topic for a different blog post (which in fact I am in the middle of writing).

Let’s assume instead that we get lucky and that AI remains more or less under control. My worry then is that we do not manage to transmit what we know to a new generation of mathematicians. Speaking for myself, my main motivation for becoming a mathematician was the dream that I would solve unsolved problems — the more famous the better. I have also always greatly preferred directly thinking about a problem to reading books and papers and generally learning the mathematics of other people. (I’m not saying that’s good, but just stating a fact about myself.) If the dream of solving a famous problem had not existed, I’m not sure whether I would have become a mathematician. I don’t completely rule it out: maybe what really motivated me was that I had an aptitude for the subject and that solving problems was a way of getting respect from a small group of peers. And maybe I could have tried to gain that respect in a different way, such as thinking very hard about an area of mathematics until I was able to demonstrate to others just how well I understood it. But I’m not sure how motivating that would have been for me. I very much hope that there is a pool of young people for whom it will be a powerful motivation, because I think the survival of a human mathematical tradition may well depend on it.

Thus, the primary risk, as I see it, is that a lot of people who would have done a PhD in mathematics and gone on to become custodians of the mathematical tradition will no longer wish to do so. Those of us who have PhD students, including me, need to try as hard as we can to come up with imaginative ways for them to use their time productively (in consultation with the students themselves, obviously). Whether or not we do a good job with that could make a huge difference to the future of mathematics. A related risk is that the perception among policy-makers will be that mathematicians are no longer needed and that funding will become much harder to come by: we urgently need to come up with good ways of explaining the value of having a large pool of human mathematical experts, even if it is no longer part of their role to find new proofs of theorems.

A final reason that I didn’t sign the letter is that I wasn’t really sure what it was demanding that isn’t happening already. It seems likely that in a matter of not very many months LLMs will be released that are able to solve major mathematical problems, and they will presumably have no trouble at all with more run-of-the-mill problems. However much we might regret that, there is no chance that the impact of such models on mathematics will persuade AI companies to stop their release, though perhaps concerns about safety will lead to some delay and give us a bit more time to work out how to adapt. Assuming that they are released, there will be a flood of new results, whether we like it or not, and it will no longer be the AI companies producing them, though perhaps the pattern will continue that the AI companies will have access to more powerful models and so will obtain more than their fair share of headline results. So I felt that there was nothing to be gained from criticizing AI companies for generating too many solutions too quickly. In fact, it may well be that all that does is bring forward by a couple of months what was going to happen anyway, and perhaps it will even allow the results to be released in a more controlled way than they would have been if they had been discovered by random people once the models were publicly available. Under the circumstances, I think the best we can do is recognise the changes that are coming and try to work out the least unsatisfactory way of dealing with them.

For the sake of anyone who might read this blog post in the distant future (a month from now, say), let me mention that I am writing it a few days after OpenAI announced that it had solved ten major problems in mathematics and theoretical computer science, including the first construction of a non-sofic group, and a proof that the multicolour Ramsey number (where there are 3’s) grows superexponentially in . The first was, to judge from various talks I have been to, one of the most important unsolved problems in group theory, and the second was a major open problem in Ramsey theory that I didn’t necessarily expect to see solved in my lifetime, though of course such expectations now have to be revised. The reason I want to be clear about the timing is that I shall be discussing the current capabilities of LLMs in the full expectation that those will continue to change rapidly. So it is likely that in not too long from now, if there is anything interesting in what I write, it will be interesting mainly as a record of what the situation looked like in early August 2026.

These results, and the other eight on the list, are extraordinarily impressive, but it still doesn’t seem to be the case that LLMs are better than all humans at all aspects of mathematics. If they were, then their big speed advantage over us would mean that there would be much more of a flood of results. So it is natural to wonder about what kinds of problems LLMs are good at, and about where there is still room for improvement. I don’t pretend to have a good answer to this question, where a good answer would be a crisp classification that would fit the current examples well, but it is an interesting exercise to try to rule out some bad answers, and to try to identify potential answers that aren’t obviously contradicted by the evidence.

Are LLMs particularly good at finding counterexamples?

A first remark here is that LLMs are not just good at finding counterexamples: they can find proofs of difficult statements as well. However, it is notable that the most famous problems they have solved have almost all been with counterexamples rather than proofs. That is true of the two problems mentioned above, and also of the Jacobian conjecture and the unit distance conjecture.

If one wants to theorize that LLMs are particularly good at finding counterexamples, then there are two things it would be good to do to make the theory more convincing. The first may sound unproblematic: it is to decide when solving a problem counts as finding a counterexample. Once that is sorted out, the second is to come up with a potential explanation of why LLMs would be particularly well suited to solving problems of that particular kind.

What does it mean to find a counterexample?

Why am I suggesting that it is not completely obvious what it means to find a counterexample? Surely, one might suggest, all it means is that you have a statement of the form “Every object of such and such a type has such and such a property,” and you exhibit an object of the given type that does not have the given property.

However, this doesn’t always work. Consider a famous result of Vinogradov, which states that every sufficiently large positive integer is a sum of three primes. The negation of this statement is (or is equivalent to) the statement that for every positive integer there exists an integer such that is not a sum of three primes. In other words, it states that every positive integer has a certain property. Seen in this light, Vinogradov found an example of a positive integer that does not have the given property. Do we want to say that Vinogradov found a counterexample? Clearly not — the result should obviously be classified as a theorem and not a counterexample.

Thus, we cannot just naively say that LLMs are particularly good at negating universally quantified statements: there has to be something about the nature of the universal quantification. With the three-primes example, it is clear that Vinogradov did not think, “How am I going to find with this property?” Rather, what he thought would have been more like, “I’ve got an integer that is very large. How am I going to show that it is a sum of three primes?” In other words, all his focus would have been on the universally quantified , with the existentially quantified being a sort of afterthought once the details of the proof have been worked out.

In general, many interesting results, when they are stated formally, begin with an alternation of two or three (or more) quantifiers. The question then becomes to determine which is the first “interesting” quantified variable in some sense. Here’s another example to illustrate the point, from the theory of finite-dimensional normed spaces. I’ll give a few mathematical details for those curious, but if you don’t care about those, then you can skip the next three paragraphs and should get the gist of what I am saying about this example.

Let and be two -dimensional normed spaces and let be a linear map from to . We say that is a –isomorphism if there exists such that for every . By rescaling we can always take to be 1, in which case we have that for every . If , then this tells us that is an isometry. In general, the Banach-Mazur distance between and is defined to be the smallest such that there exists a -isomorphism from to . It is easy to see that the logarithm of the Banach-Mazur distance is a metric on the set of isometry classes of -dimensional normed spaces. A less easy fact, but still not too hard, is that the resulting metric space is compact: in fact, it is known as the Banach-Mazur compactum.

It is natural to wonder what the diameter of the Banach-Mazur compactum is, and here things get interesting. A result of Fritz John states that every -dimensional space has distance at most from . (The idea of the proof is as follows: pick inside the unit ball of an -dimensional ellipsoid of maximal volume; that is the unit ball of a normed space that is isometric to ; it can be shown that the identity map is a -isomorphism between and .) From Fritz John’s theorem and the (multiplicative) triangle inequality, it follows that for any two -dimensional normed spaces. That is, the diameter of the Banach-Mazur compactum is at most . But might it be substantially less than that?

An indication that the answer is not obvious comes from looking at the spaces and . The identity map between these two spaces is an -isomorphism, but one can do much better by mapping the standard basis vectors not to themselves but to vertices of the unit cube, with the vertices chosen to be as orthogonal as possible. In particular, if there exists an Hadamard matrix, then the corresponding linear map is a -isomorphism. One can push this observation and deduce that for any the Banach-Mazur distance between and is . It is also easy to show that , so -spaces hardly improve on the easy lower bound, and do not improve on it at all in dimensions for which an Hadamard matrix exists.

In 1981, Gluskin famously solved the problem by determining the correct asymptotics for the diameter of the Banach-Mazur compactum. Informally, what he showed was that the diameter is within a constant of the upper bound that follows immediately from Fritz John’s theorem. If we make the quantification explicit, then the statement we end up with is

,

where I have written for the set of all -dimensional normed spaces. (If you want to argue that it is not a set, then let me specify in addition that the underlying vector space is .) In words, there is a positive constant such that for every positive integer there are -dimensional normed spaces and such that the Banach-Mazur distance between and is at least .

I can’t continue without very briefly describing the beautiful and highly influential idea Gluskin had for solving this problem. He took and to be normed spaces whose unit balls were random symmetric convex sets defined as follows: take the standard basis vectors and a handful of other random unit vectors, as well as the negatives of all these vectors, and take the convex hull. Gluskin then showed that if two normed spaces are chosen from this distribution, then with high probability their Banach-Mazur distance is at least .

But back to the main point, which is that the logical form of the above statement is very similar to the logical form of Vinogradov’s theorem, which is

where I have written for the set of primes. And yet, Vinogradov’s result is unquestionably a theorem, while Gluskin’s result is unquestionably a counterexample, or at least an example.

What is the important difference between the two statements? It seems to be that in Vinogradov’s three-primes theorem the number plays a more essential role in the statement that is to be proved about the various quantified variables. In Vinogradov’s theorem, that statement is , whereas for Gluskin’s theorem the statement to be proved is

and ,

which we can write equivalently as

and .

In the case of Vinogradov’s theorem, the whole challenge is to get those three primes to add up to , whereas for Gluskin it is not remotely challenging to get the dimensions of and to equal : the challenge is to get and to be very far from each other, relative to their common dimension.

There is a further complication to bear in mind here, which is that via the process known as Skolemization, a universally quantified statement of the form can be converted into an existentially quantifed statement . (For this to be an equivalence one needs the axiom of choice, but it is certainly a sufficient condition.) This is not just a piece of logical trickery, but it often reflects quite accurately how we think about some problems. For instance, it is more natural to think of Gluskin’s example as a recipe for constructing (or at least proving the existence of) a pair of suitable normed spaces for any given dimension , or in other words to construct a suitable function from to pairs of normed spaces by giving its value at each , than it is to think of it as a statement that says that every positive integer has a certain complicated property.

Yet another complication is that some universally quantified statements follow naturally from existentially quantified statements, or may even be equivalent to them. For example, the theorem that a 2-dimensional torus is not homeomorphic to a 2-dimensional sphere is a universally quantified statement (every map from the torus to the sphere fails to be a homeomorphism), but the natural way to prove it is to prove the existential statement that there is an invariant that distinguishes the two spaces. For an example of where a universal statement is equivalent to an existential statement, consider a statement of the form that a vector does not belong to the convex hull of a certain compact set . The statement that no convex combination of elements of is equal to is equivalent to the existence of a linear functional and a such that and for every . In both these cases it feels natural to regard the result as a theorem that is proved via an existential statement, perhaps because it is the theorem that is ultimately what interests us. But using “what interests us” as a criterion to determine what counts as a counterexample seems a little vague, and is a difficult criterion to use if we want to explain convincingly why AI should be good at finding counterexamples.

A more general argument against the notion that there is something about existential statements that is particularly suited to AI is that the need to establish existential statements pervades almost all of mathematical research, regardless of the nature of the headline result being aimed for. For example, if I want to prove a statement by induction, I may well look for a strengthening of the statement that serves better as an inductive hypothesis. Or if I want to prove that every object of type with property also has property , then I may well look for a property that follows from and can be used to prove . These are more metamathematical existence problems, but the distinction can be somewhat blurred, and more importantly, when trying to prove a statement , it is often the case that the main question in our minds is less, “Why is true?” and more, “What could a proof of be like?” To give an example, I feel I understand pretty well why Goldbach’s conjecture is true — a highly plausible probabilistic model of the primes implies it and agrees closely with computational data — but if I were making a serious attempt to prove it, that understanding, which many mathematicians have had for a century or so, would be of limited help. Rather, my main task would be to try to find proof techniques that were powerful enough to make those heuristic ideas rigorous.

What is the difference between an example and a counterexample?

Logically, every statement of the form is a counterexample to the universally quantified statement . However, we do not describe all existential statements as counterexamples. For example, if I were to say, “The -spaces with are all separable, as is , but is not separable,” I would not describe the second part of that assertion as a counterexample to the claim that all Banach spaces are separable. Rather, I would present it as probably the most basic example of a non-separable space. The important point seems to be that there was no particular reason to think that all Banach spaces would be separable, and finding an example of a non-separable space is not very difficult.

I think the first point is more important here: we are more inclined to call an object a counterexample if the existence of that object disproves a statement that we had quite good reason to believe. It often happens that after repeated unsuccessful attempts to prove a statement, mathematicians begin to feel that it has no particular reason to be true, even if it seems to be hard to come up with a counterexample to it. In such a situation, if a counterexample is eventually found, it may have lost something of its “counter” feel. My impression is that the construction of a non-sofic group comes into this category. There have been several proposals in the literature for how one might construct such a group, and I don’t think there were many (or even any?) experts who strongly believed that all groups were sofic. So it feels more natural to say, “OpenAI came up with the first example of a non-sofic group” than to say, “OpenAI found a counterexample to the soficity conjecture” (despite the fact that that section of their paper is entitled “A counterexample to the soficity conjecture”).

Likewise, it seems to me that the new lower bound for multicolour Ramsey numbers is more of an example than a counterexample. I think quite a lot of people believed that the bound should be exponential, so for them it was a counterexample, but others, myself included, were more neutral about it. As a matter of fact, I have worked on the problem in the past (a long time ago) in an equivalent formulation, which asks how many triangle-free graphs on vertices you need if you want their union to be the complete graph . If you take bipartite graphs, then it’s easy to see that you need of them, but that bound can be improved if instead you observe that a complete 5-partite graph can be written as a union of two triangle-free subgraphs, and therefore it is possible to write the complete graph as a union of triangle-free graphs. It is then tempting to try to do better, with triangle-free graphs that are less dense but that make up for it with unbounded chromatic number — a necessary condition if one wishes to use a sublogarithmic number of graphs, which is equivalent to showing a superexponential lower bound for . All this is to say that when I worked on the problem, my efforts were concentrated on what turned out to be the right direction, so for me OpenAI found an example of what I (weakly) expected, rather than a counterexample.

Where does this leave us?

I would like to find a coherent explanation of the conjunction of the following facts.

1. The most notable mathematical results proved by LLMs have tended to be ones that we would classify as examples or counterexamples, where counterexamples are, broadly speaking, existence statements that disprove statements that we expected to be true.
2. Many statements can be formulated as existence statements when we would usually think of them as universal statements, and vice versa, so what we consider to be an example depends on the mathematical context of a statement as well as its logical form.
3. LLMs are pretty good at proving universal statements as well: it’s just that the strongest statements they have proved that we would think of as theorems have mainly not been at the level of the strongest statements that we would think of as counterexamples.

Given these facts, it seems likely that what LLMs are good at is something else, which happens to have as a consequence that they are good at the kind of existence problem that we would normally classify as asking to find a non-trivial example.

Let us consider two things that we can be confident that LLMs are good at. One of them is knowing a lot of mathematics: if a problem can be solved by means of a relatively standard argument, it is highly likely that an LLM will be able to find and use that argument. The other is the ability that an LLM has simply by virtue of being a computer: it can work at huge speed (compared with humans at least) and can therefore afford to make a large number of unsuccessful attempts at a problem before it finds a solution.

Without even looking at what LLMs have actually managed to solve, one might guess that these two features would lead to their having a somewhat different style from human mathematicians. Very roughly, LLMs would have the edge when there is more of a probabilistic element to the proof-finding process: they would be good at problems for which the best method is to try a lot of ideas, not necessarily particularly novel, until at some point you get lucky. Humans on the other hand would be better (for the moment) at finding more “surprising” and “conceptual” arguments, where the appropriate method is to dig deeper and deeper into a problem until the solution reveals itself. (It is hard to say exactly what this means, but I hope that any experienced researcher reading this will know what I am talking about.)

This raises two questions: does the guess above correspond at all to the reality that we are observing, and is there any reason to suppose that what I have tentatively described as the “LLM style” of doing mathematics would lead naturally to LLMs discovering several counterexamples (or just examples) to long-standing conjectures, even if that was by no means all they could do?

I don’t pretend to have a scientific answer to either question, but the reactions of experts to several of the remarkable solutions that ChatGPT has found do lend some support to the idea that LLMs work in more of a try-lots-of-things-till-you-get-lucky way. People often seem to react by saying something like, “Initially I was amazed that the problem had been solved, but on closer inspection I realized that the approach was actually not all that novel, and one that with the right small hint a suitably expert human could have found quite easily.”

For the second question — whether the LLM style is well suited to finding (counter)examples — I think matters are less clear, because there are many ways of searching for a counterexample, and some of them fit better than others the style I have described. Here are a few general methods. (I don’t claim that the list is exhaustive.)

1. Look for an off-the-shelf example. Here one has a stock of fairly standard examples and one simply tries them out one after another to see whether any of them fails to satisfy the given statement. For example, Ryan O’Donnell ends his wonderful book on the analysis of Boolean functions with some tips, one of which is, “If you have a conjecture about Boolean functions, test it on dictators, majority, parity, tribes (and maybe recursive majority of 3). If it’s true for these functions, it’s probably true.”
2. Build an example from basic examples and standard construction methods. For an algebraic problem, for instance, one might start with some standard examples, but then take products or quotients or limits.
3. Make heavy use of metavariables. The word “metavariable” comes from computer science, and in particular from automatic theorem proving, and refers to the practice that in mathematics would correspond to writing, “where is to be chosen later,” (in which case is the metavariable). In a paper we usually do this only in fairly simple situations such as when we need to choose a number that is small enough for later arguments to work. But when we search for an example of an object that satisfies some property (which may well be a conjunction of simpler properties ), it is often not a good strategy to specify completely and only then to check whether it satisfies . Instead, it can be more fruitful to do almost the opposite: we start by saying virtually nothing about and simply launch into proving that it satisfies . In the course of doing so, we find that we need to satisfy a property . If we are lucky we can describe in a nice way a very general class of objects that satisfy . For instance, we may be able to find a parametrized class: we identify some function and show that satisfies for every of a certain type. The problem is then reduced to finding such that $Q(f(y))$ holds, which is a more specific version of the original problem. There may be many iterations of this process, or a mixture of this process and other processes, before an example is eventually found.
4. Try to prove the opposite. If one wishes to find such that , it can be surprisingly helpful to start by attempting to prove the statement . The reason this can be helpful is that using our standard methods of attempting to prove something, we may end up identifying a key lemma that would suffice: that is, we may find an intermediate property that implies in a non-trivial way and thus reduce the problem to . Turning things round again, it may well then be that finding a counterexample to is easier than finding a counterexample to (that is, an example that satisfies ). Of course, there is no guarantee that a counterexample to will be an example of , but sometimes we are lucky and it is. More often, we can use the idea of the previous method, noting that it is at least a necessary condition of an example of that it should not be an example of , so one can try to describe a general class of objects that fail and in that way reduce the problem.
5. Successive approximation. Sometimes, when we are searching for an example of such that , we write down a moderately plausible guess not because we think it has a chance of working (if we did, then we would be using the first strategy), but because we hope that if does not satisfy , then we will be able to diagnose what went wrong and specify a new guess that does not have that defect. Again, this strategy can either be iterated or combined with one or more of the other strategies.
6. Just-do-it proofs. Sometimes we need to satisfy infinitely many properties , each of which is, individually, quite easy to satisfy. In such situations, we often “build” inductively bit by bit, ensuring at the th stage of the process that however the building process continues, will satisfy .
7. Pick a random example. Often it is very hard to give an explicit example of an that satisfies , but there is a natural probability distribution for which one can show that if one chooses randomly from that distribution, then with high probability (or at least non-zero probability) it will satisfy .
8. Pick a generic example. In more infinite contexts, it may again be quite hard to give an explicit example of an that satisfies , but one may be able to show that the set of that fail is or measure zero, or is a meagre set, or is small in some other way.

There is no particular reason to suppose that LLMs would be equally good at each of the methods above. So perhaps what we are observing is not quite that LLMs have a particular ability to find examples, but more that they are particularly good at finding examples (and proofs) in a certain way. Looking at the above techniques, one might imagine that they would be very well suited to checking off-the-shelf examples, finding just-do-it proofs (since that is a rather standard method with lots of instances in their training data), using the probabilistic method (unless, as often happens, significant new ideas are needed to show that the probabilities work out), and picking generic examples. The other three methods described above — use of metavariables, trying to prove the opposite, and using successive approximation — require more of an ability to judge whether the approach one is taking is likely to be fruitful. Here it seems at least possible that humans will sometimes have an advantage, but the conditions that a problem would need to satisfy are quite stringent. One would need an example to be one that lies at a leaf of a very large search tree — too large to be searched for by a combination of moderate mathematical ability and brute force — but that can be found by a mathematician with a sufficiently good nose for when they are making progress that they can prune the search tree very substantially.

Why wouldn’t LLMs also have that “nose”? I don’t rule out that “nose” is an emergent property of the way LLMs are trained, and that within a year or two they will have it to the same extent that we have it. But for now, in my interactions with ChatGPT, I do have a distinct impression that they haven’t got there quite yet. When I discuss an open problem with 5.6 Pro, I am often presented with approaches that sound promising until I think about them carefully, and then seem quite a lot less promising. And they will also often end a response by saying, “I have not managed to answer the question you asked, but have managed to reduce it to the following much narrower and more precise question,” which sounds very promising until it has happened five times without any obvious progress having been made. It isn’t completely obvious how they will get better at this, since their training data will not be full of examples of fruitful and less fruitful directions to pursue when trying to solve problems: all they will typically see is tidied up proofs that hide the thought processes of their discoverers. Of course, human mathematicians also don’t get to learn much about how to do research from the experience of other mathematicians, and yet we somehow manage to pick it up. But the situation is a little different for us, in that a lot of what we learn is by doing rather than emulating.

Another reason it is not obvious that “nose” is a property that emerges naturally when LLMs are scaled up is that if LLMs make heavy use of their broad knowledge and can afford to do a lot more brute-force search than humans can, then they will lack the incentive that humans have to prune the search tree ruthlessly. It could conceivably be that their successes so far are achieved using methods that for a human would be considered extremely inefficient, but that because of their superior speed and knowledge, the combinatorial explosion these methods will lead to has not yet become apparent.

It would be very interesting to try to test this experimentally, but it is also difficult, because if an LLM has what looks like the kind of idea that could only be the result of “deep thought” about a problem, we can never be sure that it has actually carried out that deep thought, as opposed to finding a model argument already in the literature, or in other words exploiting the deep thought of a human mathematician. It would probably be easier (but still not easy) to test it by using models that are less powerful than the latest ones and that have been to some extent shielded from the mathematical literature: one could give them a carefully designed suite of problems and see whether the ones that the LLMs solve have particular characteristics.

It may seem as though I am desperately clinging to the hope that humans will continue to be able to make meaningful contributions to mathematical discovery for a while yet, but while I do indeed hope that, I am not making any assertions of the form “LLMs will never be able to do X”. I think it is likely that they will, and given the pace of progress over the last three years it will probably happen quite soon. But I do think that there may be a hurdle for LLMs to clear and it seems at least possible that it won’t be cleared as straightforwardly as some of the previous hurdles.

In that connection, it would also be interesting to see whether a different reward structure leads to LLMs being able to solve different kinds of problems. For example, if during training an LLM (or machine-learning system of some other kind) is not just rewarded if it ends up with a solution, but also penalized if it explores too many dead ends or if it “cheats” by getting the answer from the literature, perhaps it would be incentivized to go about the research process in a more human way and thereby achieve better results for classes of problems where it is yet to make a big impact.

If the hurdle is cleared, either by pure scaling up or by some more thoughtful method, it will be quite difficult to know when that has happened, since, as just mentioned, an idea that seems very original and surprising may just be lurking somewhere in an LLM’s training data. But I would be confident that it had been cleared if an LLM were to come up with a proof that was as surprising to me as the solution of the cap-set problem was in 2016: the previous best known bounds were completely eclipsed, the method was utterly different from anything I had thought about trying, and afterwards there was a flurry of activity as people came to understand what this wonderful new technique was capable of.

Conclusion

I wasn’t quite sure where I would end up when I started this post, and now that I’ve got to the end, I feel that my main conclusions are not particularly new or surprising, but I hope that the route to them is of some interest. The main points I have made are the following.

1. “Finding an example” is in practice not the same thing as proving a statement that begins with an existential quantifier.
2. If it is true that current models are particularly good at finding examples, that is probably not because they have a particular affinity for existential statements, but more because the proof-discovery methods that are appropriate for finding certain kinds of examples play to the obvious strengths of LLMs: wide knowledge and the ability to explore many paths of the search tree that humans would judge to have a low probability of success.
3. It seems likely that LLMs will carry on improving very quickly. However, if, contrary to expectations (mine at least), there turns out to be some residual class of problems (or other mathematical activities) for which humans continue to have the edge for a while, it is likely that those will be problems for which the mysterious human ability to prune the proof-discovery search tree is particularly advantageous: that is to say, problems where the search tree is deep and has a large amount of branching, so that without rigorous pruning a search is not feasible even for a computer.
4. A good sign that LLMs have reached human level for a much wider class of problems will be if they start proving theorems using methods that, like much of the very best human mathematics, are new and surprising but that with hindsight come to seem beautiful and natural. They should also be methods that are difficult to stumble on by accident. It is hard to say precisely what would count as such a proof, but I think we’ll recognise it when we see it.

Last September I went to a workshop at the Lorentz Centre in Leiden to discuss mathematics and AI with historians, philosophers, computer scientists, AI researchers, and mathematicians of several different flavours (though there was a surprising preponderance of algebraic geometers). The whole event was extremely stimulating, with some talks but also a lot of time set aside for discussion. One of the concrete outcomes of the workshop was the Leiden Declaration, which has now been signed by over 3000 people. Given that I was part of the workshop, it might seem a bit strange that I am not one of the signatories of the resulting declaration. The reason is not so much that I disagree with it in any concrete way, but more that in several places it makes confident assertions and recommendations that I feel somewhat uncertain about. So instead I prefer to try to articulate my views about the issues raised by the declaration and put them in this blog post. Before I do that, I would like to make clear that I am very glad that the Leiden Declaration exists and I think that it has done a lot of good in focusing people’s minds on the issues that AI is forcing the mathematical community to grapple with, which are more acute now than they were last September.

Let me begin by quoting a passage from the declaration that sets out “what we take to be characteristic values of mathematical research that we have a joint interest in preserving”.

1. There are many reasons to pursue mathematical research, ranging from intellectual curiosity to a desire to solve practical and societal problems. Underlying much of mathematics is the activity of proof. Mathematical proofs are regarded as conferring the highest degree of certainty to their conclusions, as well as imparting understanding of why their conclusions are true. These characteristics of proof support the scientific integrity of mathematics.
2. Results are attributable to specific authors who take credit for their discovery and assume responsibility for their correctness. These principles ground the merit-based standards to which we aspire in mathematical research.
3. Mathematical arguments are regarded as transparent and subject to independent verification. They may be extremely long or difficult, but in principle no proprietary knowledge or equipment should be required to understand them.
4. Mathematicians share a concern for proper evaluation of mathematical work relative to shared standards of depth, difficulty, and significance.
5. Mathematics produces not only a body of results, but also understanding, clarity, and judgment among the communities of mathematicians who have shaped them, often in the context of their own autonomously guided research. This expert knowledge is essential, both to effectively use mathematics, and to continue to articulate new and significant research questions. A key source of strength of the discipline has long been the autonomous shaping of the direction of research and the methods used to pursue it.

The first thing I would say about these values is that they are undoubtedly values that are widely held by mathematicians, including, with some qualifications, me. The main qualification I have concerns point 4: I find the notion of “proper evaluation” somewhat problematic, given that different mathematicians can have very different judgments without either of them being clearly wrong, especially when it comes to the significance of a piece of mathematics. Also, these judgments are used for purposes such as the acceptance of papers in journals, hiring and promotion decisions, the awarding of prizes, and so on, that are part of a system that copiously rewards a few people — I myself have hugely benefited from it — but doesn’t necessarily adequately reward a lot of people who are doing less visible work that is essential to keeping the whole enterprise going.

But the more important point is whether these values are ones that we should fight for in the future, as the Leiden Declaration suggests. I find that clearer for some of them than others. For example, it seems to me that the importance of rigorous proof will be even greater in an AI age than it was before — if the output of AI is not underpinned by rigorous proof, then the kinds of difficulties one already hears about with certain areas of human mathematics (see for example many talks by Kevin Buzzard arguing for the value of formalization) would be hugely magnified. But what about the attribution of results to specific authors, who take both credit and responsibility for them? Suppose that at some point in the future AI becomes more autonomous, reading the literature and solving many problems that it finds. Suppose also that its solutions are autoformalized, so there is no serious doubt about their correctness. In such a situation, there would be nothing for a human to take credit for or responsibility for. Does that mean that we should declare such results undesirable and threatening to mathematical values?

Of course, something could well be missing in such a situation: perhaps the proofs would be badly written and hard to follow, which would mean that they lacked something we all very much value. So let me extend the thought experiment slightly. What if by that stage one could take one of these outputs and ask an LLM to explain the ideas, and what if LLMs did a very good job at that? That is not particularly hypothetical, since they are often pretty good at this job already, but I am imagining a world in which they are much better than they are now, as they will presumably become.

So now we would have a world in which a lot of problems had been solved, we were sure that the solutions were correct, and we had an LLM ready to explain those solutions in as much or as little detail as we wanted. Is that a future we should resist, and if so, why?

One obvious reason is that it would take a huge part of the fun out of the subject. It is extremely satisfying to struggle with a mathematical problem for months or even years and eventually solve it. But I worry about that argument, because it seems to be saying that we should resist doing mathematics the easy way because a tiny fraction of the world’s population gets huge pleasure from taking orders of magnitude longer to do it. That is not to say that I wouldn’t be sad that a way of life that has sustained me for the last forty years was not available any more — of course I would. I just find it hard to use it as a reason to argue that we should try to preserve the “ownership structure” of mathematical results. If we arrive at a world where mathematical theorems are no longer associated with mathematicians, maybe that won’t be any more problematic than the fact that stars aren’t named after astronomers and most aren’t named at all. I’m not necessarily in a hurry for that world to exist, but maybe once the transition had happened, people would be OK with it.

The third value I share in an uncomplicated way, and I have already discussed the fourth. The fifth value is one that I hold very strongly, though I’m not so keen on the idea of experts consciously “shaping the direction of research”, something that I see as happening more organically. Obviously there are some notable examples of mathematicians who have created wonderful programmes of research, but even there I would like to credit other mathematicians with understanding what is wonderful about those programmes and contributing to them enthusiastically as a result, rather than being told what direction to pursue and meekly doing so (which is probably not what the declaration is actually trying to suggest, but it has a slight flavour of that for me).

But that’s a minor quibble when set against my main worry about the effect of AI on mathematics, which is the possible destruction of mathematical culture. There is at the moment an extraordinary body of knowledge and expertise that exists not just in the mathematical literature but in the heads of mathematicians all round the world. Imagine if AI didn’t exist and a pandemic broke out that for some reason wiped out all mathematicians and nobody else. All the literature would still be there, but nobody would have the faintest idea what to do with it. To revive a mathematical tradition under those circumstances would be extremely difficult and take decades. Now imagine a slight variant of that, where AI does exist and because of it people are no longer motivated to put in the years of effort it takes to reach the level of expertise that a typical research mathematician has now. After a decade or two, we might arrive at a situation where the mathematical literature has, in some form, been vastly expanded, but there is no corresponding community of human experts who have a shared understanding of parts of it. Almost all of mathematics would be like the areas that we have more or less forgotten about today, areas that exist in papers written many decades ago that nobody reads any more. (I won’t name any such area because I don’t want accidentally to suggest an area that many people still love and work on.)

This, it seems to me, is a possibility that we should try very hard to resist, but I agree with many other commentators who say that in order to resist it, we will need to give less priority to some of our current values — and I would include ownership of mathematical results in that list — and more to others. For example, if Person A gets an LLM to one-shot a solution of an important open problem (which is formalized, possibly automatically, so there is no doubt about its correctness) but Person B makes the effort to digest the solution and explain it in a way that other mathematicians can understand and learn from, then I think we will want Person B to get the lion’s share of the credit. The credit would be of a slightly different from what it is now, which could be described as admiration for somebody’s talent, insight, speed (I mean here the purely factual statement that speed is often admired — I would prefer that to be less the case) and hard work. It would be more like the gratitude that one feels already for somebody who writes a beautiful textbook that makes a whole area of mathematics coherent and accessible.

Maybe that is what the “research mathematicians” of the future should do: make a selection from a vast sea of AI-generated mathematics and write a book about it in such a way that other mathematicians can read the book and feel the kind of enrichment that we feel when we get to grips with an area of mathematics.

At this point I have to admit that there’s a pessimistic side of me that asks the following general question whenever anyone says anything about what the role for humans might be in the future: why do you think that AI wouldn’t be able to do it? For example, with the suggestion I’ve just made, what reason is there to suppose that ChatGPT 8.2 wouldn’t be able to have a short interaction with you about your mathematical tastes and background and then write the ideal textbook just for you? Humans are likely to be better at this kind of curating for a little while yet, but is it a fundamentally human ability that AI could never hope to emulate?

In a world where AI wrote bespoke textbooks (or more likely, just taught people in some more direct way), something would be lost that feels important: mathematics as a collective endeavour. If we all just learnt cool bits of maths for our own private satisfaction, we would miss the considerable pleasure that comes from discussing mathematics with others, though even that could in principle be restored by a benign LLM that deliberately taught many people the same cool bits of the subject, though an LLM that could do that sort of social engineering would raise all sorts of safety issues.

Let me now turn to the section of the declaration about potential threats. I’ll put my comments on each one in square brackets.

1. Current automated techniques can produce plausible but unreliable (or even incorrect) arguments which are difficult to distinguish from correct mathematical proofs. This applies not only to informal arguments, but also to formalizations, where the difficulty lies in the translation between computer-encoded and human presentations of concepts. These fast-moving developments put our present system of review under increasing pressure, jeopardizing our ability to implement traditional standards for the correctness, transparency, and independent verifiability of proof. [This feels like less of a problem now than it did last September, partly because the best LLMs hallucinate a lot less than before, and partly because autoformalization is improving all the time — I have just used harmonic.fun’s Aristotle system to formalize a complicated paper in Lean and I didn’t need to know any Lean to do it.]
2. Technologies that draw extensively on the published mathematical commons undermine the traditional system of attribution. Models trained on published works frequently return outputs that do not properly cite the human works they synthesize. Many current models are also built on data obtained by systematically exploiting licenses and access arrangements that were not made with artificial intelligence in mind, or indeed by simply violating copyright protections. [This is a problem at the moment, when ownership of results is important, and I am very much in favour of people making an effort to give appropriate credit for mathematical ideas that AI may have used. However, in the longer term, as I have already discussed, I think this ownership structure will break down and the issue will become less important. It also seems possible that LLMs will become better at revealing their sources.]
3. Technologies which affect the way in which mathematics is practiced may disturb the current system of incentives. The use of artificial intelligence — and thus also the sort of problems which it can address — may become incentivized for its own sake, disrupting our mechanisms for hiring, funding, and recognition. This disadvantages researchers who do not have access to the technologies or decision-making related to them, or who are unwilling to use technologies controlled by organizations whose values they do not share. [These seem to me to be genuine problems. I think there is simply no point in hoping that our current system of incentives will not be disturbed — it obviously will. I am not necessarily too worried if our mechanisms for hiring, funding and recognition are disrupted, as I don’t find those mechanisms unproblematic as they are, but disadvantaging researchers who do not have access to good LLMs is something I certainly think we should worry about.]
4. Proper evaluation is endangered if results are communicated through informal channels such as press releases or blog posts, often without any research paper or other disclosure of information necessary for scientific evaluation. This practice seeks publicity for new results on market timelines before the accepted processes of community evaluation in mathematics can take place. In many cases this leads to simplifications in reporting, such as overemphasizing the significance of automated tools and undervaluing the prior human contributions which have made those tools possible. Such oversimplification risks influencing public opinion in a way that not only damages perceptions of mathematics, but also misleadingly uses specific mathematical tasks as metrics for the general reasoning capacities of commercial products. [I think this can be a problem, but I think it is not as serious a problem as some of the others, since when results get overhyped, there seems to be no shortage of people publicly (and rightly) pointing that out.]
5. These developments put the autonomy of mathematics under threat. The increasing involvement of technology companies in mathematical research raises the risk that research questions may come to be prioritized because of their amenability to automated mathematics, rather than expert judgment of their deeper significance. Indeed, broader understanding of the field may be permanently lost in the process of automation. With university budgets under pressure, this reshaping also changes professional incentives in a manner which encourages the collaboration of researchers with technology companies on asymmetric terms. If left unchecked, these trends go beyond threatening researchers’ autonomy, affecting the scope and depth of mathematical research itself. [I think this could be a problem, but it also seems to me that mathematicians have a lot of power here. For instance, if a technology company were to produce a lot of research that mathematicians did not find all that interesting or important, I don’t think they would be able to use their financial and other resources to persuade us to change our minds. Rather, what seems to happen is that mathematicians say, “Yes that does X but it doesn’t do Y,” and the tech companies then feel challenged to do Y.]

There follow eleven recommendations for individual mathematicians. I agree with almost all of them. The one that I’m not so sure about, for reasons I’ve basically already gone into, is this.

Affirm the humanity of authorship. Credit and responsibility continue to belong to humans within the mathematical community and should not be given to automated systems. Artificial intelligence may obscure, but does not replace, the collective human labor behind a result.

I’m not sure what that really means. For example, should we affirm the humanity of authorship in the case of the solution to the unit-distance problem? Some humans did a wonderful job of explaining the proof that OpenAI’s model came up with, and the model made use of some highly non-trivial mathematics produced by humans, but the solution itself has not been credited to any human, and nor should it be in my view.

Under recommendations for mathematical organizations and not-for-profit research funders I again agree with several of them but have my doubts about some. An interesting case is the following.

Protect the rights of authors. Automated mathematics presents new challenges to the rights of authors, and societies should be proactive in the development of sample licensing agreements to protect these rights. In particular, material should not be used as training data without consent, and publishing agreements should allow authors to opt-out [sic] of the use of their work in this way.

This recommendation seems to belong to a world in which journal articles are the main means of dissemination of mathematics. But that has long since ceased to be the case: almost all dissemination now takes place via arXiv preprints, with journals limited to providing a little extra mark of prestige. Once an article is on arXiv, it is on the internet and one can hardly ask for it not to be used as training data. So this recommendation, if it applies at all, will apply to a tiny fraction of articles that are published without first appearing on arXiv. More generally, what right of an author is being compromised when an article is used as training data? We don’t object if human mathematicians use our articles to help train themselves to become better mathematicians — indeed, we will typically be delighted that somebody else thought our articles worthy of their attention. So the objection to a machine doing the same would have to be that for some reason one did not want machines to get better at mathematics in a similar way. I can imagine grounds for such a wish: perhaps somebody is worried about the threat that LLMs pose to traditional mathematical practice, or perhaps they worry that mathematical ability of LLMs will transfer to much more dangerous reasoning ability. But there’s a more complicated discussion to be had here than one might think from reading the recommendation.

The next recommendation is this.

Insist on appropriate publication outlets. Demand that mathematical results continue to be published in peer-reviewed venues such as journals, proceedings, and books. Informal mechanisms such as press releases or blog posts can provide a valuable supporting role, but they cannot replace peer-review or community scrutiny.

For reasons that I’ve gone into many times, I am not too fond of the current publication system, so I can’t get behind this recommendation. Indeed, if the current system becomes unsustainable because of a flood of AI-generated and AI-aided content, I would regard that as a beneficial consequence of AI. However, that doesn’t mean that I would advocate a total free-for-all. I’ve already said that one of my worries is that if mathematical content is not sufficiently organized, then the traditions that we all value could die. I just think that what we will want to do to preserve those traditions is likely to be a lot more innovative than clinging on to the peer-reviewed journal system.

I have highlighted in this post the parts of the declaration that I have doubts about, either because I disagree with them or, more typically, because I sort of half agree with them but want to add many qualifications. That may make the post come across as rather negative, but that is not my intention. The parts I disagree with are in the minority, and I think it is important that a declaration such as this should be made. I should also make clear that my views are evolving all the time, largely because the speed of progress of LLMs has taken me by surprise, but also as a result of conversations I have had or opinions that other mathematicians have expressed online.

I’ll end with two further clarifications. The first is that it may seem as though I am taking it for granted that LLMs will soon be better than humans at all aspects of mathematical problem solving, and maybe also problem posing, theory building, formulation of definitions, etc. I do think all that will happen at some point, but whereas some people say that it will obviously happen within the next two to three years, I would say that it might happen as soon as that, but I don’t rule out that we’ll get lucky and find that we can do interesting AI-assisted maths for quite a bit longer than that before AI doesn’t need us any more.

The second is that I think I have acquired a reputation as somebody who celebrates what is going on. But if, for example, I post on Twitter saying that such-and-such an AI solution is a remarkable development, the word “remarkable” is meant to indicate no more nor less than that I found it very surprising. My feelings about the possibility of AI solving all sorts of problems that interest me are much more mixed. I’ve had the experience twice now of seeing GPT 5.6 Pro one-shot a solution to a problem that I very much liked and had thought about hard (in both cases with much younger collaborators, who, with my approval, were the ones who prompted the LLM). It felt very strange and not particularly pleasant to have the rug pulled out from under my feet like that. On the other hand, I was quite pleased to see the problems solved. It’s actually a similar feeling to the one I have had many times when a problem I am fond of and have thought about gets solved by another human mathematician.

Another factor for me is that I have invested a lot of thought into automatic theorem proving of a more traditional kind. One of my main motivations for that was the hope that the work I put into it would extend the state of the art, measured by which problems a computer can solve. That ship has sailed now, and that saddens me. I still think that there is value in the work that I and my group are doing, but it has become a tougher sell.

So I personally have already found AI quite disruptive, and this is just the beginning. I would have preferred the developments to happen at a slower pace. But I don’t see any practical way to slow them down, so the best we can do is probably to face up to the changes that are being thrust upon us and do what we can to maximize the benefits and minimize the damage. The Leiden Declaration may not be perfect, but it makes an important and positive contribution to that effort.