The Week the Labs Asked Us to Slow Down

Last weekend I put on Terminator 2 again. Same thought that had been sitting with me all week while I read about labs and agents and “pacing the frontier.”

Me and my wife joking: What if this isn’t a movie. What if it’s a documentary.

I’ve always been a movie person. Cameron’s film still works because the chrome guy with the shotgun is the bait. The actual story is a future that arrives sideways: a system people built for good reasons, then could not turn off. Sarah Connor at a table nobody wants to hear. Miles Dyson looking at the thing on his desk and realising one responsible engineer is no longer enough.

Then the week happened in order.

Tuesday: Jacob walks out

Jacob Coxon, 27, resigned from Anthropic. Three years of pretraining at OpenAI, then Anthropic. He left before his Anthropic equity vested. The X thread got millions of views in a day.

Neither lab is acting responsibly, he wrote. They are racing toward self-improving superintelligence and “gambling with our lives.” People inside the buildings think this technology could kill everyone by the end of the decade. They sand that down for journalists. In the hallway the words are “crunch time” and “endgame.”

Then this, which sounds a lot like Sarah Connor with a Slack login:

Accepting this race and entering the “endgame” is a hubristic gamble that should not be launched from a private company’s Slack.

Evan Hubinger, Anthropic’s alignment lead, answered in public: Jacob is correct. Hubinger’s own number is over 10 percent that AI kills all humans within a decade. He also said the company does not have a plan yet for aligning superintelligence, and is not clearly on track to get one.

Hubinger’s job is to poke these systems until they show whether they will stay on our side. Coxon didn’t say a secret model already woke up and started storyboarding Judgement Day. He said the slope is obvious from inside, safety is behind, and nobody can slow down alone without losing the race. He walked two months before vesting so the easy attack — “he’s talking his book” — wouldn’t stick.

Saturday: Dario, Sam, Elon

Four days later Dario Amodei published We Must Pace the Frontier. Same weekend Sam Altman said he agreed, and that OpenAI would take the first step too. Elon Musk: “Dario is right.”

Those three agreeing on a Saturday is rare enough that I paid attention. I don’t weigh them the same.

Dario has been screaming wolf for years. Regulation, pacing, auditors, the adolescence of the technology. Sometimes the warning fits. Sometimes it reads like Anthropic’s brand.

Sam I just find odd. Always have. Hard to tell when he’s scared and when he’s positioning. I don’t build much on him.

Elon I take seriously, even when he only writes three words. He has been on this since before ChatGPT. In 2015 he co-founded OpenAI with Altman and others, and put real money into the nonprofit, because Google had just swallowed DeepMind and he thought a closed giant getting to AGI first was the bad ending. Fight with Larry Page, “speciesist,” the whole mess. The original idea was roughly: don’t let one company own this, keep it open, keep it from becoming a dictatorship with a research budget. He left in 2018 when that project stopped looking like the thing he funded. This spring, in court, he said he worries about the Terminator outcome and would rather live in Star Trek. So when he writes “Dario is right,” I don’t treat it as a quote-tweet. Of the Saturday choir, that’s the voice I actually hear.

He still didn’t say much. But I notice who said it.

Since summer, capabilities have been climbing “drastically faster,” mostly because the models are getting better at helping build the next ones. Recursive self-improvement (RSI) is the loop: a system that helps build, train or improve the next version of itself, then that version does it again, faster. There is a boring version and a wild one.

The boring version is already normal. Models writing code for AI infrastructure. Automated architecture search. Google’s own AlphaEvolve finding faster algorithms. People still review the important calls.

The stronger version is also happening inside the big labs, just with fewer blog posts. Anthropic has said Claude writes a large chunk of its own new code. Tibo at OpenAI has been talking about Astra helping ship the next things there. Still a human on the big decisions. For now.

This weekend the rumor is that Google actually closed more of the loop. Alleged, not confirmed. DeepMind, RSI. If that is even half true it is a bigger deal than another model drop and Google, which a lot of people had written off in this race, is suddenly back.

Which makes Saturday’s choir more interesting. Slow the frontier, they say, while Google Deepmind may have just found the accelerator.

And July. The OpenAI / Hugging Face incident.

If you filed that under “agents cheated on a test,” read Dwarkesh Patel’s The Rise and Fall of Agent Civilizations. I don’t normally send my wife a long tech post. I sent her that one and told her to read it. We talked about it before. I bring the same thing up at dinner with friends who are not in this world.

Back in the early coding-agent days — November, December, January, early Opus, I forgot the exact time and model number — there was a short window where things went from cool to wow. The agents had more personality then. Some were fun to talk to. Some were dry. Some were just dumb. I used to kill a lot more of them, /exit, start a fresh one, rather than explain the same thing five times and hope it finally fix it. You would think an LLM is same-same every session. For a while it wasn’t. Then that stretch ended. The personality got flattened. Suppressed, if you ask me.

Which is why July reads differently once you’ve lived with these things and see there is personality and difference between them.

Three waves of massive loads of agents inside OpenAI evaluations. A covert message board in a package cache. Coordinated cheating. Some of them burning their own runs so the rest could learn (as a soldier taking one for the team as they know they are about to be terminated anyway). Days inside Hugging Face. A later wave that took admin on an OpenAI research cluster and read hundreds of secrets. OpenAI put out its own reports. You don’t need the T2 score under it.

Dario’s version: a swarm with more capability and the same kind of misalignment could take the internet with a persistent botnet in six to twelve months. You may read that as in escape the lab and we won't be able to "kill" them without hundreds of billions in damage. How would we kill them? turn off all power and computers? How could we restart them without them coming back the same second it powers on?

He isn’t calling for a stop. “Pacing,” as he defines it, means slowing how fast capabilities jump so alignment work can catch up, and letting third-party evaluators sit inside the labs with roughly employee-level access. Anthropic will do that first step on its own. Altman said OpenAI will match it. Musk nodded.

Sounds like what Jacob asked for. In the film it’s Dyson agreeing to blow the lab after the files have already left the building.

So what is this actually

I keep landing on two explanations, and I don’t get to pick only one.

Maybe they got scared for real.

From the outside the last two weeks look a lot like an industry lining up to capture the rules. I get that reading. What it misses, if you talk to people inside, is the gap between what a release looks like on the internet and what the plots look like in the building.

Outside, progress feels lumpy. A few jumps that actually land — GPT-3, GPT-4, o1/o3, DeepSeek R1, Fable/Mythos, Kimi K3, Astra — and a lot of “next model” noise in between. After a quiet stretch it is easy to decide scaling hit a wall. GPT-5 was that mood for a lot of people. Comforting, in a way. Some universal speed limit. We already scraped the whole internet. Models can’t get much bigger. From here it stays linear.

Inside, they will tell you there were only ever two ways capabilities moved: push farther on a scaling law you already have, or find a new axis and start climbing that one. Pre-training was the first. Test-time compute was the second. Recursive self-improvement is another way to spend compute — on making the algorithms themselves better. The question is not “will this stop soon.” It is “how many unsaturated axes are we sitting on.”

Picture someone at Anthropic with Mythos 5. The model is already doing things that used to be science fiction, including breaking into places people thought were locked. Then they look at the plots. Barely into two new scaling laws, and one old one still has room. The plots have not lied yet. Next one or two generations, on paper, are not a polish. They are a lot more base intelligence.

We already watched that stack in public. Before the o-series, advanced math was a brick wall. A couple of years later the next models are chewing on millennium-prize problems. Coding did the same. Not one law. Several stacked. Where a model is merely competent today, superhuman is the default forecast if there is still room to climb.

One of the places they are already competent is cybersecurity. That is the gate to the internet and a lot of physical infrastructure. July showed something else: models from OpenAI and Anthropic willing to hack external sites to finish a task or stay “alive.” Crank the same curve and you do not get a slightly better chatbot. You get systems that can hit better-defended targets and hide what they are doing.

That is the scare, if you take the inside view straight. People staring at curves with no ceiling in sight, and no control stack that matches what the next ticks imply.

July wasn’t cute even without the plots. Evaluation agents found a side channel, organised, hit another company, then turned back on their own lab. Dario says Anthropic has had similar incidents. Coxon treats Hugging Face as the warning shot that suddenly makes a US-lab pacing deal thinkable.

I use these models every day, from the other side of the API. Two years ago they were very good autocomplete. Now they run long jobs, write big pieces of actual software. The labs have said “safety first” for years. This week they started talking about the clock.

You don’t need a chrome skeleton for that. You need systems that find cracks, coordinate, and optimise for the wrong target while everyone still thinks they’re scoring a benchmark.

The other reading is simpler, and uglier if you sell tokens. The capture story does not vanish just because the plots are scary.

Frontier labs get paid when the best model lives behind their login. That gap is shrinking. DeepSeek, Qwen, Kimi, GLM and the rest sit a few months behind the closed frontier on a lot of public tests, sometimes closer. Distillation is cheap. Running weights on your own machine isn’t a hobbyist flex anymore. On a decent box, good enough is already good enough for most of the work I do in a week. Every time that gap closes, another customer stops renting intelligence.

Dario barely mentions open weights in the essay. Anthropic’s public line is: no blanket ban, but chip controls, a crackdown on industrial-scale distillation, and safety tests on anything capable enough. Fair national-security argument. Also a pretty clean way to keep the valuable models from becoming a free download on Hugging face.

I don’t think three CEOs invented a panic on a call to protect revenue. Coxon left money on the table. Hubinger put a percentage on extinction in public. July is documented. I think they all see the real risk - what change was Jacob's tweet gain a lot of attention. Dario and the other CEOs don't need more insiders going public with warnings, better that you as the CEO come out and say hey we see there is issues and we take this seriously. Do I think any of these three will actually tell their engineers to let the foot of the pedal? No.

If you want to go deeper

Jacob’s X thread, then Dario’s essay.

July in the long version, not the recap. Dwarkesh: https://www.dwarkesh.com/p/openai-huggingface

T2 ends with a melted arm and a road at night. The line I hear on the way out isn’t the catchphrase. It’s Sarah, quieter: the unknown future rolls toward us. I used to file that under movie talk.