In February 2019, OpenAI released a “language model” called GPT-2. It had been trained on over 8 million webpages with a simple objective: predict the next word in a given text. So it was perhaps unsurprising that the model could generate coherent text in response to prompts. But it could also write fantasy battle scenes, summarize long paragraphs, and translate into French. It wasn’t especially good at these tasks, but what surprised its creators was that it could do them at all. It hadn’t been trained for any of them, and the researchers had deliberately removed non-English web pages from its training data.
As the essayist Scott Alexander wrote at the time, GPT-2 developed untaught faculties because, at its core, it was a neural net that worked according to principles similar to those of the human brain. A brain learns all sorts of things without being told to, in the process of trying to make sense of the world by predicting what it will sense next. Like the brain, GPT-2 was a prediction machine. Alexander added:
“In the context of performing their expected tasks, AIs already pick up other abilities that nobody expected them to learn … All that stuff you hear about ‘AIs can only do one thing’ or ‘AIs only learn what you program them to learn’ … are now obsolete”.
Alexander saw, years before most writers did, where systems like GPT-2 were heading. As a professional mathematician, I have watched the change at close quarters. I work in number theory—a field Carl Friedrich Gauss called “the queen of mathematics”—where I prove theorems about the structures linking numbers, equations, and symmetry. In the early months of 2026, the best publicly available AI models still produced absolute slop when asked to help with research questions in my area. But in May, OpenAI announced that one of its internal models had disproved a longstanding conjecture of Erdős—about the maximum number of pairs among a given number of points that can lie exactly one unit apart—by drawing on abstract ideas from number theory. This jolted many mathematicians.
In the early months of 2026, the best publicly available AI models still produced absolute slop when asked to help with number theory research. But in May, OpenAI solved a longstanding problem.
My first use of AI to help me prove theorems was in early June. It felt like working with a competent and indefatigable PhD student. Today, I can give OpenAI’s latest public model—GPT-6 Astra Pro—a bare sketch of the strategy for a proof, sometimes even less, and trust it to fill in the details. Increasingly I find myself taking on the role of conductor, rather than doubling up as orchestra. In early September, an internal OpenAI model—believed to be a few months ahead of publicly available ones—solved the Navier–Stokes Millennium prize problem in fluid dynamics, one of mathematics’ most famous problems.
This rapid advance has provoked a wider debate within the mathematical community. An open letter, published three days after the Navier–Stokes breakthrough and signed by twenty-five past winners of the Fields Medal—mathematics’ most prestigious prize—warned of a “severe misalignment” between AI companies and the mathematical community. Solving problems, it stated, was only a “tool and proxy for achieving the primary goal of conceptual understanding”. Yet as Sir Tim Gowers, a Fields Medallist who did not sign the letter, argued, mathematicians vary widely in how they view the relationship between problem-solving and understanding, and at least some are primarily motivated by the wish to solve problems and see conceptual understanding as an important means to that end.
Mathematics is just one part of the upheaval across the knowledge professions. Programming was hit earlier. At Anthropic, the share of code written by AI rose from the low single digits in early 2025 to more than 80 per cent by May 2026. Meanwhile, AI agents have begun e-mailing consciousness researchers with e-mails offering views on consciousness, and have written to philosophy professors, asking for paid work.
At Anthropic, the share of code written by AI rose from the low single digits in early 2025 to more than 80 per cent by May 2026.
We are entering what computer scientist Scott Aaronson—long skeptical of human-level AI in this century—now calls “the age of wonders and terrors”. AI is intellectually unsettling in a way unlike anything else humans have created. No one really knows how the future will pan out. But, as Aaronson notes, there is no longer any doubt that AI will shape the rest of our lives, and everything depends on whether we succeed or fail at directing its astonishing power towards human flourishing.
Concerns about AI fall, roughly, into two classes. The first comprises conventional risks such as disinformation, water consumption, discrimination, surveillance, fraud, cyberattacks, job displacement and concentration of economic power. The first two of these have attracted particular attention recently. AI can indeed be used to generate convincing disinformation, but, as the philosopher Dan Williams has argued, political persuasion is extremely hard; and insofar as AI does influence public attitudes, its overall effect may be to strengthen the influence of expert opinion. Meanwhile Andy Masley has shown in a detailed analysis that data-center water use is minuscule compared with that of other sectors. Some other conventional risks may be more serious, but their scale remains uncertain; many claims are poorly evidenced; and those emphasizing them often underestimate AI’s power and potential.
AI is intellectually unsettling in a way unlike anything else humans have created.
The second comprises existential risks: the possibility that AI systems far more capable than humans across most cognitive domains, pursuing goals that diverge from ours, might act in ways that lead to human extinction or permanent disempowerment. Long associated with Eliezer Yudkowsky, Nick Bostrom and rationalist blogs such as Overcoming Bias and Less Wrong, these fears are now taken seriously by many leading figures in AI. They attracted global media attention in early September when AI researcher Jacob Coxon sensationally resigned from Anthropic, saying that the people building AI “earnestly believe that it could kill us all by the end of the decade”. Anthropic’s alignment science lead, Evan Hubinger, agreed, putting his own estimate of the chance of AI killing all humans at more than 10 per cent.
In a recent article, Claire Lehmann, founding editor of Quillette, compared AI-doom fears to past apocalyptic scenarios involving overpopulation, nuclear annihilation, and environmental collapse. She rightly noted that technophobic alarmism can prompt unwarranted restrictions that do great damage to human progress. However, unlike past doomsday prophets, AI-doomers are highly technophilic and politically diverse; many of them are genuine experts in the systems about which they are concerned. Several prominent members of this group have also forecast AI’s development so far with unusual accuracy. They should be taken seriously.
The detailed AI 2027 scenario, published in April 2025 by Scott Alexander and leading AI thinkers from the AI Futures Project, describes a rapid increase in AI capabilities with two possible endgames: either a misaligned superhuman AI kills all humans within a decade, or we end up in a technofeudal regime controlled by oligarchs. Importantly, though, both endgames rely on AI’s rapid diffusion across government, the military and society, including its being put in charge of factories manufacturing robots and drones, while powerful political figures cede judgment and decision-making to AI.
The last point matters because an AI cannot exterminate an embodied creature by pure thought: it needs direct or indirect control over physical objects. Alexander has also written elsewhere about the “diffusion gap”: the time between AI becoming capable of doing 90 per cent of knowledge-work jobs and AI actually doing even half of them. Diffusion is hard. Current bottlenecks include limits to human cognitive capacity and the speed at which institutions can adapt. Alexander identifies future regulation as another key reason diffusion may slow down.
There is a “diffusion gap”: the time between AI becoming capable of doing 90 per cent of knowledge-work jobs and AI actually doing even half of them.
But there may also be a deeper source of friction: the powerful human urge for meaning and mattering. Rebecca Newberger Goldstein argues that the need to matter is our deepest longing and the crux of the human experience. This instinct is likely to work against surrendering control of meaningful parts of our lives to machines. The reaction of a section of the mathematical community to recent AI breakthroughs is suggestive. Nor should we assume that persuasive power scales indefinitely with intelligence: being vastly better at reasoning does not guarantee an ability to persuade humans to give up control.
A reader may object that a few consequential decisions could transfer dangerous power without widespread diffusion, and that some people’s reluctance to surrender control does not stop others. But I suspect that many AI-doomers underestimate the social and institutional resistance that widespread displacement of human agency would provoke, and so overestimate the likelihood of some doom scenarios. Handing AI control of military systems, drone factories or critical infrastructure is likely to meet far greater opposition than adopting systems to help run them. Moreover, such resistance would create new gaps between capability and power, and new opportunities to slow the process down, buying time to understand and adapt.
Many AI-doomers underestimate the social and institutional resistance that widespread displacement of human agency would provoke.
Even if an AI catastrophe is less likely than some claim, why tolerate any risk? Why not simply halt AI development? Doing so would mean forgoing extraordinary benefits. In AI 2040: Plan A, the optimistic follow-up to AI 2027 published earlier this year, the AI Futures Project proposes a road map to preserving those benefits while drastically reducing existential risks. In an essay introducing Plan A, Scott Alexander envisaged triple-digit GDP growth in the 2030s if it were implemented. He emphasized the scale of what might be at stake:
“The choice isn’t business-as-usual versus a world with some gee-whiz AI innovations. It’s nanobots eating the solar system in 2033 vs. cancer cures in 2035”.
Nor would GDP hypergrowth or spectacular scientific breakthroughs capture all of AI’s benefits. In a recent post on X, physicist David Deutsch described how AI helped him fix a dishwasher he was about to replace. Widespread access to knowledge and practical competence may reduce measured GDP while still making people better off.
Some AI-safety researchers advocate pausing frontier AI development until we have “solved alignment”. Roughly speaking, alignment means designing AI so that its behaviour and goals remain reliably consistent with human intentions and values. Others, however, see alignment less as a problem to be solved in advance than as an iterative process of tweaking, feedback and learning. Google DeepMind’s Séb Krier has argued for this view, while Maarten Boudry and Simon Friederich suggest that selection pressures arising from human choices could lead to more obedient and controllable AI systems, much as domestication shaped animals.
Recently, Dario Amodei, Anthropic’s CEO, argued for pacing the frontier, rather than pausing AI development or racing ahead. This is a sensible approach. Slower gains in AI capability would still mean fast progress, while the time gained, if used wisely, could reduce the chances of doom and improve the prospects for human flourishing. Possible measures include independent testing and oversight of frontier AI models; building in reliable ways to shut down AI systems in an emergency; keeping humans in the loop for all consequential decisions; strengthening the security of critical infrastructure; and preparing defences against engineered pandemics and cyberattacks. Several elements of Plan A also seem compelling: an AI treaty with China to jointly track advanced AI chips; much greater research transparency; and relying primarily on human control rather than alignment for safety in the medium term.
Slower gains in AI capability would still mean fast progress, while the time gained could improve the prospects for human flourishing.
There are also pitfalls to avoid. In their 2025 book, If Anyone Builds It, Everyone Dies, Eliezer Yudkowsky and Nate Soares advocate a global prohibition on publishing new AI research. This would eviscerate freedom of speech and academic freedom. Yudkowsky has, elsewhere, advocated airstrikes on rogue data centres to enforce an AI moratorium, even if it risks nuclear war. More broadly, some of the more sweeping conclusions of AI-doomerism seem to pay insufficient attention to the epistemic and political norms that help societies avoid mistakes, to the realities of human psychology, and to the relationship between intelligence and power.
AI is a revolutionary technology unlike anything we have created. In some fields, including my own, it may soon become an extraordinarily intelligent oracle. But intelligence is not sovereignty. A machine’s ability to think more powerfully than we can does not mean surrendering our choice of what to pursue, build, control or understand. On the contrary, AI could enormously expand what humans can know and do. To realize this promise, we need to protect free inquiry, remain open to empirical evidence, and be willing to solve difficult problems.
