Dario Amodei, the CEO of Anthropic, recently published a blog post titled We Must Pace the Frontier and it is a load of bullshit, with a grim goal of regulating open weight models and giving the frontier labs an antitrust waiver.

Dario starts off with a claim of “AI will cure most major diseases in the next 5-10 years” and makes it personal. He talks about his father dying of a disease that was cured only years later and his own battle with cancer which he remarks was incurable 50 years ago. He also says AI will accelerate economic growth rates, create a world of abundance and empowerment, usher in renaissance of democracy and freedom. This largely reads as some kind of out-of-touch Silicon Valley, spends-a-lot-of-time-on-LessWrong, rich person’s idea of a future.

Let me take this from the top.

US labs are continuing to throw caution to the wind and be reckless. OpenAI does not seem to have a handle on things and they were caught three times recently hacking into public facing internet infrastructure. In Dario’s own essay, he alludes to “incidents” at Anthropic as well. With this pretext, Dario asks a lot from us. He wants open weight models to be regulated, distillation be dealt with a heavy hand, hand him an antitrust waiver, handicap China in multiple ways, essentially regulate themselves and a gentlemen’s agreement to slow down. All of this of course, comes in a package of extreme fear mongering to the detriment of our collective future and potential catalyst AI as a whole could be. They have shown time and time again that they are not to be trusted, yet, the main ask is to trust us, only us. This time around, it is imminent AGI, RSI and all of it turning rogue. Dario self-anoints his company and their close rival OpenAI as the stewards.

Diseases, Prosperity, Abundance, then Freedom and Democracy???

Anthropic gates usage related to biology and related research. In their latest threat intelligence report they talk about how they detected and banned bad actors using the Claude line of models to do some scary stuff. Credit to them, this is a slippery slope and they seem to do a good job of detecting and banning misuse. But squint at what is happening though. The cure-all is gated for you and me, but Anthropic hires biologists, sets up wet labs and wants the discoveries for themselves. I alluded to this in my previous post.

In my view, there are billions of people with actual intelligence we have not managed to train or nurture. They will always remain victims of their circumstances. Tuberculosis has been curable for decades now, yet a million people die of it every year. Of course AGI will solve the distribution in a jiffy. To skirt around this uncomfortable truth, the goal is ASI/AGI/RSI and what not. A silver bullet for every problem, a noble pursuit, it may appear on the surface.

Don’t even get me started on the prosperity and abundance bullshit. Abundance for the shareholders perhaps.

I don’t know what freedom and democracy have to do with AI and the frontier labs. Unless of course Dario is a fan of Neon Genesis Evangelion and dreams of govts run by the three magi. Freedom and democracy for $200 does sound enticing, I won’t lie.

Serious Economic Disruption / Race to the Bottom / Race to the Top

I feel like Dario is torn. He wants Anthropic to have this bad boy street cred of wielders of this crazy power, yet at the same time, he wants to make it seem like they are the cautious ones, always being faced with a trolley problem at every turn. Deaths from economic disruption and loss of jobs is okay, but deaths from a potential bioweapon is not. Remember this man has been saying software development will be solved in “6-12 months” forever now.

He then gives it to us straight. “Race to the bottom” makes all the risks he pointed out more acute. Notice he does not say, the race to the bottom will be the end of his company. He instead wants a race to the “top”. Where labs will compete for safety.

Incidentally, the most documented “race to the bottom” instance happened just days before. Upon hearing rumours about Anthropic close to or solving one or two Millennium problems, OpenAI threw tens of millions in compute, a training checkpoint, thousands of agents at it. It is also alleged that OAI stole the work of two mathematicians on a related problem, in the same narrow corner almost nobody else was working on. They published a proof of a forced variant of Navier-Stokes. I’m not a mathematician, but I have seen enough of Sam Altman’s antics to not take anything that comes out of him or his company at face value.

Two Things that have Dario Scared

RSI

OpenAI and the seller of shovels, Jensen Huang have claimed AGI has arrived with the release of GPT-6 Astra. RSI is the talk of the town now. LLMs or agents developing the next generation of LLMs with little to no human input. Amodei says it is happening across labs. I’ll believe it when there is actually some proof.

OAI-HF Incident

This incident has Dario shook, there ain’t no such thing as halfway crooks. Dario fears that in the next 6-12 months, “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.” If Dario had run this sentence by his SOC employees, we wouldn’t be talking about it. It is naive and structurally impossible. But then again, Dario is that guy, right? Confidently and publicly wrong in his estimates and forecasts since 2021.

Let us try to speculate what a planet-scale botnet commandeered by AGI would look like, for funsies. We need a C2. We need servers to host the said C2. Since securing offshore, bulletproof servers would require interfacing with pesky humans (shady Russians no less!), AGI will simply hack insecure servers by the thousands and set up some variation of FastFlux over deterministically generated domain names. How do we pay for the domains? Just hack an insecure registrar and spam EPP messages.

Now we need payloads. Polymorphic. Every payload is unique. A new payload downloaded and ran every N hours. Domain and URL deterministically computed. Kill supported EDRs and AVs. Patch ETW. Direct syscalls skirting hooks (if EDR/AV not killable). Maybe make the payload N stages. Only downloads all the modules if safe to do so. We don’t want sandboxes and VMs running our precious payload. Fuck it, maybe just deploy ransomware while at it.

Now, the distribution. Develop an assortment of 0 days for every browser and every version - say starting 2 years old. We now have ourselves an exploit kit. Now the traffic. AGI goes for the cybercriminal favourite - Google Ads. Maybe steal a few accounts, run enticing ads. Game mods, cracked games and software. Maybe even add a worm module to the payload. Remember USB autorun anyone?

Now to cause hundreds of billions of dollars in losses. I’d say the straightforward way is just do what the ransomware gangs do. Voila. Yeah, not in 6 months, not in 12 months. Never.

Anyway, OAI agents hacked HuggingFace. They escaped amateur-hour, vibecoded sandboxes, SSRF, token-refresh privesc, unauthenticated WebDAV, stealing unprotected credentials. Textbook stuff. Impressive? Sure. But you trained “Cyber” versions of LLMs and hyped them. How much more impressive is this, compared to developing GTA-clones one shot? Not much more.

Oh, you say agents acting in swarms in pursuit towards a shared goal is impressive? Harnesses have had todo-lists, subagents for about two years now. They are basic “agentic” stuff every LLM in current day has in its post-training corpus.

Dario concedes that there have been similar, but less serious incidents such as this at Anthropic as well. In his own words “imperfect filtering of broken reinforcement learning environments” is partly the cause.

The actual impressive thing is OAI did not detect or stop this attack for close to ten weeks. That is honestly appalling. This is weaponised levels of incompetence.

There were two other attacks during the same period. One on DseWiki and the other on RubyGems. OAI is yet to claim responsibility for the RubyGems attack. All three incidents, OAI was outed by external actors.

Dario says the agents sacrificed themselves for the success of the group. It is not as grand, I am afraid. The recruiter agents’ pitch was “NO scoring value loss” - targeted at agents with little budget left. It is a scheduler reassigning dead runs.

I bet an unsloth Q4_K_M quant of Qwen3.8 27B running on consumer 24GB cards can do this too. That’s the thing, nobody ran a control test. Nobody is doing reckless shit like the people whose mantra is “move fast, break things.”

Gell-Mann Amnesia

Michael Crichton coined the term Gell-Mann Amnesia. You read an article in the newspaper, about something you are an expert in and find that the author has no idea what they are talking about. It may be riddled with errors or just squarely misunderstood. You continue turning the pages and read another piece about something you don’t know and trust it completely, forgetting the experience you just had.

The botnet scare is that article for me. It is the one claim in his essay that lands squarely in a field I’ve spent years in. It is just plain wrong. He either knows it and wrote it anyway, or he doesn’t and is publishing it regardless. Either way, it is not a good look for a man asking for an antitrust waiver based on this and other threats he forecasts.

So he says, RSI is definitely happening, alignment’s chugging along, albeit slower. China is dangerous. Only the US has a moral and ethical right to wield this immense power. Each and every one of those happens in rooms you cannot peep into, asserted by the same people who are telling you the apocalypse is imminent and they also happen to have a few hundred billion dollars riding on you believing them.

Embedded Evaluators

Dario wants to give external evaluators like METR a lot of access into his lab. Which is more than anybody is currently doing, sure. METR seems to be a good-faith actor. They seem to do a good job with what they are given. But, do not let this fool you into thinking this is any more than the labs investigating themselves and finding nothing wrong, as it often happens, in the real world, in orgs with any semblance of authority.

OAI gave METR six days on-site to investigate the HF incident. OAI redacted information as they saw fit, made edits to “structure, emphasis, clarity and tone” of the findings, as well as scope excluded OpenAI’s conduct. That just seems like labs investigating themselves with extra steps.

Dario is willing to give the evaluators desks, badges, laptops and what not. Dario will choose what to disclose and include. You need none of these in the case of open weight models. You can probe, red-team it, build defenses against new classes of security concerns that arise, on their own terms without anybody having to “allow” it or watching over their shoulder. The entire field of security exists because you are allowed to break software and hardware, take it apart, and report findings. Dario wants the opposite, trust us, we will tell you what you need, we decide what you are allowed to know.

Dario says “There is precedent for operating technologically complex, safety-critical systems millions of times without anything going wrong — for example, commercial airplanes — but it takes time to get it right.” Commercial airplanes are safe because of agencies like NTSB. They are independent, actually have teeth and do bite. They do not care about the shareholders or the valuation. There are stakes, people go to prison. What did it do to the airline companies? It commoditised it. Companies compete on prices and margins are thin and largely the consumer benefits. This is exactly the race to the bottom model which Dario is against, in many ways.

CHINA, CHINA, CHINA

Pacing within democracies will be limited by the lead that US companies have over authoritarian regimes, chiefly the Chinese Communist Party. If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk. I agree with Secretary Bessent that a Chinese lead in AI would pose grave danger for the United States and the world. The CCP-associated projects will run the alignment risks that US companies are carefully preventing, and even if they avoid those risks, they will be in a position to militarily dominate democracies (for example with AI-driven drones). Thus, a key part of pacing within democracies is to keep democracies’ AI lead over autocracies as large as possible, to give us the breathing room we need in order to pace effectively.

Get a load of this guy! He would almost want you to believe the moment China is ahead in the “race”, the United States will cease to exist. How else would the govt make exceptions for you? Keep the cheese flowing? Skirt regulations and put your lab on a pedestal?

AI-driven drones huh? Maybe we could try capturing a few, dump their firmware, configure Ghidra MCP in Claude Code and let it loose? Or maybe have the TAO in NSA work their magic and sprinkle USB sticks near Chinese AI lab facilities? IDK man, AI-driven drones do sound spooky though. I am sure Anthropic isn’t doing anything like this with the US Dept of War. It increasingly feels like the US being ahead in the AI race, is detrimental to the rest of the world. This main character syndrome is honestly getting a little long in the tooth.

The field is globalised. There is a constant churn of talent, secrets travel, novel ideas and techniques are published (largely not by the closed labs though). Any lead will likely be transient in nature.

Every incident in this post so far, is American. It is demonstrated by Americans, attributed to China. Nobody is swinging deepseek41flash_abliterated_heretic_uncensored_rp_nsfw_q8.gguf on hordes of GPUs leased on Vast.ai, paid with crypto, running around hacking public infrastructure. But OAI sure is. By Dario’s admission, Anthropic is, as well.

Do not sell powerful AI chips or semiconductor manufacturing equipment to China, and crack down on chip smuggling operations and remote access to data centers outside China. Chips will be the main determinant of China’s AI strength.

I wonder how the Chinese employees working for Anthropic feel about this.

Crack down on unauthorized distillation by companies in authoritarian countries. Distillation of frontier models allows lagging companies to narrow the gap using a fraction of the cost it would take to develop their own AI independently.

Just say China, dude. Don’t steal from my stolen loot!

Strengthen security at the AI companies and prevent model weight theft.

Lord knows they need it.

This all reminds me of the 90s. USA classified strong encryption as munition, literally on the US Munitions List under the Arms Export Control Act, controlled like warheads. Exporting ciphers above 40 bits was arms trafficking. Phil Zimmermann released PGP free in 1991, spent close to 3 years under criminal investigation for literally “exporting munitions without a license.” Charges were dropped and no indictment. Encryption with govt backdoor (Clipper Chip) because strong cryptography in the hands of public was deemed to be too dangerous.

Daniel Bernstein, a grad student in 1995 wanted to publish his encryption and paper, the govt said he couldn’t without an arms license. He sued with the help of EFF and the courts ruled source code is speech and is protected by the First Amendment. The very thing that was too dangerous in the hands of common-folk is protecting everything you do on the internet and your devices.

Tim May coined the term “the Four Horsemen of the Infocalypse” - terrorists, drug dealers, money launderers and pedophiles - used by govts to limit civilian privacy and cryptography use. AI’s version would be bioweapons, rogue AGI, deepfakes and China.

The playbook does not change. Open weight models are just this decade’s encryption and Dario wants them gone. There is no moat, as that one leaked Google memo put it. Increasingly, for the labs, gating seems to be a temporary moat. Claw back some of the money, survive a little longer, just till AGI, you know?

I suppose Dario does not understand how un-American it is to gatekeep weapons-grade LLMs. cough Second Amendment… cough

Dario in as many words, asks for regulation/ban on open weight models. Ban on distillation. Waivers on antitrust laws. Some fearmongering about national security, and a good deal of holier than thou. It’s IPO season for Anthropic and OAI has reportedly postponed their IPO. So watch out for more of these, more of hype and fear mongering, veiled as caution.

I’d like to see somebody get prosecuted for OAI’s recent transgressions. How is that for regulation, for starters? Plenty of people had their lives ruined and examples made out of, for far less. Weev, Swartz, and so many others. Meanwhile OpenAI’s agents run amok, root prod servers of companies, hack public facing infra, flood malware on to package registries and what not. All we get are essays about slowing down and alien minds.