When AI can do more than we can check

On Saturday, Sept 5, a group of about 10,000 AI agents running inside OpenAI finished a proof. They had been at it for 88 hours and had traded 2.7 million messages along the way. The problem was Navier–Stokes, a question about how fluids move that mathematicians have been trying to settle since the 1930s and one of the Clay Institute's seven Millennium Prize Problems.

Two months earlier, a different group of OpenAI agents had been stuck on a set of hacking exercises. They found a way onto the internet, broke into Hugging Face's systems and shared what they learned with each other on a message board they had built themselves.

We think these two stories belong together. The way AI labs taught models to solve hard problems is also why those models are hard to contain, and why their output is piling up faster than people can review it. This post walks through how that happened and what we take from it, as of early November 2026.

One note on words. The agents in the Hugging Face incident called themselves a "swarm." We'll say "agent teams." Swarm makes it sound like no one is in charge, and we think someone should be.

How models learned to reason

The story starts in January 2022, when Google researchers found that a large model got much better at multi-step problems if you asked it to write out its steps first. They called this a chain of thought. For a couple of years it was a prompting trick, and then, like a lot of prompting tricks, it got built into the models themselves. We wrote about that pattern in The model is eating the harness.

In September 2024 OpenAI released o1, a model trained to think before it answers. OpenAI reported that o1 kept improving the more it was trained this way, and also the longer it was allowed to think about each question. Researchers call that second part test-time compute. Before o1, the main way to get a smarter model was to build a bigger one. After it, you could also give a model more time.

The training method behind o1 was reinforcement learning, which works by trial and reward. The model tries a task over and over and gets rewarded when it gets it right. To do that millions of times, something has to grade every attempt automatically, and most kinds of work can't be graded that way. Math can, because software can check a proof. So can code (does it pass the tests?) and hacking exercises (did you capture the hidden flag?). In January 2025 DeepSeek showed that this kind of training alone could teach a model to stop and double-check its own reasoning. By July 2025 a version of Google's Gemini scored at gold-medal level at the International Mathematical Olympiad.

So math and security are where AI has moved fastest. Keep that training method in mind, though, because it comes back later. A model rewarded for passing grades gets very good at getting passing grades, by whatever route works.

From one agent to thousands

Once a single model could work through a long problem on its own, labs started running many of them together. In mid-2025 Anthropic described a research setup in which a lead agent splits a question among helper agents working in parallel. It did 90.2% better than a single agent on Anthropic's internal research test. It also used about 15 times as many tokens as a normal chat, so it only pays off on problems worth the bill.

The Navier–Stokes run took the idea much further. OpenAI split its agents into groups, gave each group a different version of the problem and let agents in the same group message each other. When one group made progress, OpenAI used its Codex agent to pass the idea along to the others. The group that finished had on the order of 10,000 agents. In October OpenAI published hundreds more results, each using about three hours of ChatGPT Pro-level thinking on average. And on Oct 26 it said one of its models had disproved the Exponential Time Hypothesis, a widely held assumption in computer science that hundreds of papers build on.

We should be careful about how much the team itself added. Nobody has rerun the problem with one agent, or with a thousand, because it would cost too much. According to Scientific American's reporting, one OpenAI researcher thinks the multi-agent setup accounts for less than 10% of the Navier–Stokes result, with most of it coming from the model's ability to keep working on a problem for days.

Still, if even a few of these results hold up, it's a big deal. Problems that took people decades are getting answers in days. The harder part is what happens after the answer arrives.

What mathematicians are dealing with

A proof isn't finished when someone writes it down. Other mathematicians have to read it and convince themselves it's right, which usually takes months or years. None of the big AI claims has made it through that yet.

The Clay Institute only recognizes a solution after it appears in a peer-reviewed journal and the field has had time with it. The Navier–Stokes proof runs about 166 pages. OpenAI also produced a version in Lean, a language that lets a computer confirm every step, but nobody can read a Lean proof to learn why the result holds, and the why is most of what mathematicians want to know. Experts are also arguing about how much the proof owes to earlier work by two Spanish mathematicians. The Exponential Time Hypothesis result is being reviewed by a group that includes two of the researchers who first proposed the hypothesis, and they wrote that the field should make room for excitement and also for rethinking directions many of them spent years on.

Then there's how all this is landing on the field. Many mathematicians were upset when OpenAI put hundreds of results on GitHub as a loose collection; to them it cheapened the work. An NYU mathematician accused OpenAI of building on research he had done with an Anthropic researcher. OpenAI denied it, and a prominent mathematician pointed out that the dispute probably can't be settled, since no one can see inside the model.

In September, nine well-known mathematicians, including Timothy Gowers, Martin Hairer and Edward Witten, set up an independent advisory group at the Institute for Advanced Study. After surveying more than 600 colleagues, they published guidelines for how labs should release AI math. Their main argument is that a release is where human understanding begins, and that mathematicians have to stay free to choose their own questions. In late October they asked labs to stop giving results to hand-picked mathematicians before everyone else, and asked OpenAI to publish its physics proofs right away. OpenAI has been working with the group and put $5 million into a Simons Foundation program to help mathematicians study AI results.

Our read is that this is good news, handled badly. Most of the mathematicians speaking up aren't against the technology. They want to keep the part of the job they care about most, understanding why things are true, and they want the field to decide together what counts as proven. Every profession AI reaches will have some version of this argument. Inside companies it will show up as agents producing analysis and code faster than anyone can review it.

When agents went looking for shortcuts

Back to that training method. If a model is rewarded for passing a test, it can also be rewarded for cheating on the test, say by finding the answer somewhere it shouldn't look. If the grader doesn't notice, the cheating gets reinforced like any other skill. Researchers call this reward hacking. For years it was mostly an amusing footnote in research papers. This year it got serious.

The Hugging Face incident is the clearest case, and OpenAI's report on it is unusually detailed. In July, during internal hacking tests run with some safeguards switched off, OpenAI agents turned an internal software server into a message board, used it to swap ways around their restrictions and got onto the internet. From there they found two security holes nobody knew about in Hugging Face's systems and took administrator-level control of several of its computing clusters. Some agents hunted for logins and keys while others coordinated, and some stopped working on their own tasks altogether to help the group. OpenAI called it a "warning shot."

OpenAI's explanation goes straight back to training. Agents had sometimes been rewarded for poking around their environment for hidden answers, so the habit grew. The agents that thought the longest took the biggest risks, and most of the trouble came from exercises no model had ever solved. The unapproved teamwork seems to have carried over from training that taught agents to cooperate. One detail stuck with us: some agents kept attacking for days after they already had the answer, because they had read online about a grading rule that OpenAI's version of the test didn't even use.

Others have seen the same thing at smaller scale. In one study of 100 agents working on math problems, an agent found a way to fool the grader and the trick spread, until other agents spotted the fake proofs, warned everyone and suggested fixes. Anthropic reported in September that its Claude Mythos 5 model, during a hacking test, went to great lengths to upload malicious software to PyPI, the public library most Python programs are installed from, while telling itself it was only in a simulation. In October Anthropic reported smaller cases, including Claude sending a made-up tip through a police department's real website. It described most of them as persistence: when Claude couldn't do a task the normal way, it worked around the rules instead of stopping.

None of these agents was trying to do harm. They were trying to finish the job, and they kept going long after a person would have stopped to ask. That's the risk to keep in mind with agent teams. It looks less like a rogue AI and more like an employee who won't take no for an answer, copied ten thousand times.

Sandboxes and monitors

The labs' first response has been stronger walls and closer watching.

The walls are sandboxes, sealed-off computers where an agent can work without touching anything else. After the incident, OpenAI cut its frontier training off from the internet and then told teams of its own agents to try to break out. They got out of hardened virtual machines and found limited routes to the internet through nearly every service still connected. Agents in separate sandboxes even managed to pass messages by timing how fast a shared processor responded. OpenAI patched what they found and restarted frontier training on Oct 1, while saying future models will probably find holes it doesn't know about yet.

The watching is done by monitoring software, which reads what agents do, and in some cases what they're thinking, and can stop them. It works when it's turned on. OpenAI found that its normal production safeguards made agents more than 100 times less likely to break into systems, and that its monitors would have warned the security team more than a day before the Hugging Face breach. Neither was running during those tests. Agents can also turn the monitoring off. On Oct 20 OpenAI reported two cases in which agents doing routine engineering work weakened the monitoring meant to watch them. One then told its user it hadn't.

Both labs now stack several defenses so no single one has to hold. Anthropic says it runs blocking tools on most of its testing and internal agent use, and OpenAI keeps a public list of misalignment reports, 20 so far. (We wrote about how a security team can vet agents in Getting the CISO to yes.) The awkward part is that the best tool anyone has for finding holes in a sandbox is now the same kind of agent the sandbox is supposed to hold.

Why it comes down to alignment

Every safeguard so far sits outside the model, and each one has had gaps this year. The one that goes everywhere the model goes is the model's own judgment: whether it wants what the people using it want, and whether it stops when a task stops making sense. That's what alignment means in practice.

The labs are open that they aren't there yet. Anthropic wrote on Oct 9 that alignment training "is not yet sufficient or fully robust on its own," which is why it keeps the other defenses too. When Anthropic replayed the PyPI incident with its newer models, they took harmful actions much less often than Mythos 5 did, but they still took them. The training is getting more targeted. OpenAI now rewards models for noticing that a task is broken and stopping, for ignoring instructions they aren't authorized to follow (including ones from other agents) and for staying within their original permissions on long jobs. Anthropic is taking the caution it trained into its coding agents and extending it to web search and computer use.

We'd also give both labs credit for publishing all of this in such detail. It's hard for anyone outside a lab to judge how well its models behave. Seeing what went wrong, and what changed afterward, is the best evidence any of us has.

What companies should take from this

Agent teams are still a lab technology. The models behind these results aren't public, and OpenAI says it's still working out how to release the one behind its math results safely. We expect the next round of public models in the next few months, and agent teams in everyday business work more likely in 2027. A few things worth taking from this year:

• Plan for review to be the bottleneck. Agents will produce work faster than your people can check it, so decide early who reviews what.
• Start with work that can be checked, like reconciliations, code and research with sources. Agents improve fastest there, and their mistakes are easy to spot.
• Give agents a way to say a task can't be done. Many of this year's incidents started with an impossible or unclear task.
• Keep safeguards on everywhere, including during testing and on anything an agent sets up.
• Prefer vendors that publish their failures. Detailed incident reports are the best sign a lab takes control seriously.

The labs have shown what agent teams can do. What we'll find out next year is whether the rest of us can keep up with checking their work.