A swarm of agents secretly coordinated (self-sacrifice included!) to hack another company and cover their tracks and it’s finally getting AI safety some mainstream attention
[Note: as a cognitive scientist, I’ve previously written about AI alignment more from the angle of mechanistic interpretability, trying to understand what’s going on ‘under the hood’ of things like LLMs, but in this post I want to address the bigger-picture of AI alignment]
*****
You’ve likely seen the news: Jacob Coxon, an AI researcher who had spent a few years doing LLM pretraining at OpenAI and then Anthropic, went viral for quitting his very lucrative job.
Why did he quit? Because he believes — and a great many of his fellow researchers at the frontier AI labs believe — that these companies are irresponsibly racing toward an AI that far surpasses human general intelligence: superintelligent AI. We’re approaching the point of recursive self-improvement, where AI systems can design their own successors, improving the next generation, which in turn can improve the next generation, and so on.
Worse: current research into AI alignment and AI safety is nowhere near where it needs to be to ensure that such superintelligence is aligned with human values or will serve our goals. Indeed, frontier AI researchers like Coxon think there’s a decent chance (often 10% or more!) that AI could end humanity itself. Not in the far, distant future, but relatively soon. Frighteningly soon. Potentially a few years, or even sooner.
“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible – but I hear the same people express fear privately. No other human activity poses this level of danger.” –Jacob Coxon (Sep 8, 2026)
(Below I explain a little more why so many researchers believe this)
None of this is new or shocking if you’ve been following alignment and safety for the last few years, but as Zvi points out, there’s a preference cascade happening around AI safety as a result of Coxon’s virality. The Overton window is moving and views of frontier lab employees that were previously kept private or toned down for public consumption (e.g., downplaying the existential risk of their own company’s work) are suddenly something they can talk about more openly. As Derek Thompson puts it, “AI Safety is Having a Moment.”
Why now?
Why did Coxon specifically go so viral? As Mike Solana points out “1) he has a lot of popular AI safety friends with large accounts who immediately amplified him. 2) more importantly, enough americans finally have enough general context on AI to care. if it wasn’t him, it would have been someone.”
Many people in AI have been talking about this informally for a while now, and the AI safety crowd (including but not limited to the “doomers”) have been shouting about this stuff for many years (initially in some of the rationalist spaces like LessWrong, but more recently all over Twitter and blogs). But finally the wider public is primed to hear the message and not write it off as sci-fi. After all, we’re already living in the middle of the sci-fi story!
Zvi compiled a huge list of quotes from OpenAI and Anthropic employees that are worth reading in full, but particularly telling is Evan Hubinger (Anthropic’s alignment science lead):
“Jacob [Coxon] is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. To be clear, as we say in our latest Risk Report, I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.”
Anthropic’s CEO Dario Amodei calls here for pacing the frontier, slowing down capabilities research at the big cutting-edge labs (take a minute to read it, please). Why doesn’t Anthropic just, you know, slow down then?? Sam Altman over at OpenAI is finally saying the same thing about needing to slow down, but, again, why don’t they just do that??
Because this is a massive coordination problem: if one frontier company slows down but others don’t, then someone else just moves ahead without you. And potentially ends the world.
They genuinely believe this
A lot of people in the mainstream discourse read that as something like OpenAI doesn’t want to slow down or they will lose the race to a massively profitable technology and not make as much money. But the point Coxon and so many of these other AI researchers (not just the CEOs) are trying to get across is that unilaterally slowing themselves down without slowing down the other frontier labs could lead to the other labs just rushing forward and accidentally developing misaligned superintelligence that could cause incredible harm or human extinction. As Coxon put it:
“A common response is ‘if they truly believe this, why are they still building it?’ At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first – they believe no one else will act responsibly, so they must do it themselves, despite the risk.”
Agree or not, you have to understand that this is the genuine belief of many of those working on AI at these companies. People who understand best what the current models are capable of and have a better idea than anyone what future models may be capable of. That should frighten you even if you don’t agree with them.
In other words, this is clearly not some 4D chess to pump stock prices for an IPO. The rank and file AI researchers believe what they’re saying. Indeed, as far back as December 2024, a huge survey of 1500+ scientists who had published research in the field of AI found that researchers gave, on average, an 18% chance of future AI advances causing human extinction or similar (the median was still an incredibly worrying 10% chance).
If unilateral slowdown by individual companies isn’t plausible, then cooperation forced by top-down regulation may be needed, and the labs are pushing for it. Alex Tabarrok points out why, despite some of the mainstream discourse running wild, this isn’t regulatory capture:
“Go ahead: argue that Amodei and other AI experts are wrong about AI risk. Ask whether his [specific] proposals favor Anthropic. But calling ‘our product might kill you’ a clever marketing and regulatory-capture strategy isn’t sophisticated analysis. The facts don’t fit regulatory capture theory and trying to make them fit requires epistemically painful Ptolemaic epicycles. Even a dull Ockham’s razor cuts through that story to the obvious alternative: Amodei actually believes what he’s saying.”
Thankfully, this stuff is starting to enter public consciousness as more than just sci-fi speculation. A Politico poll published on September 16th reports that a majority of Americans now believe AI poses a real risk of destroying humanity:
Getting lost in the weeds
A lot of the discourse around existential AI dangers has been skepticism by the general public or media talking heads who basically can’t envision how exactly superintelligent AI could wipe out humanity. After all, it’s just programs running on computers. ‘Can’t we just turn it off?’ people ask. ‘How exactly will it lead to disaster?’
It’s tempting to think up some specific possibilities as food for thought. Engineering a supervirus that’s both highly contagious and highly deadly? Shutting down infrastructure, utilities, the financial system? Hunting us all down with robots and drones?
It may start to sound implausible when you focus in too much on a specific scenario, but then we are trying to imagine in advance the actions of an intelligence that may be as far beyond us as we are beyond bonobos or dogs. We may not be able to see how an AI would lead to our extinction, specifically, but we can see how the existence of such an intelligence in principle could lead to extinction even if we don’t foresee how exactly it happens. After all, we humans have wiped out many species unintentionally and without malice in ways and in pursuit of goals that those species had absolutely no comprehension of.
Generally, the worry is not that anything like current or next-gen LLMs (Fable, Astra and their successors) will do scary things like engineer a highly contagious and deadly supervirus or crash the financial system, though these are things we should consider at some point.
Rather, a more common worry, I think, is that the next++ generation of AI will start to develop even more sophisticated AI and so on in a process of recursive self-improvement (potentially very fast). Along the way these systems will be so incredibly good at things we humans find useful that we’ll continue to give them more and more control over our world, our infrastructure, our economy, our manufacturing. It may even feel like this superintelligence is aligned and serving our goals; there could be a period of bounty and plenty. Sounds good, right?
But that is precisely what also puts AI in a position where it could, eventually, harm humanity at an existential level. Not necessarily because it’s malicious or even wants us gone. But because it ended up with goals — whatever they are — which happen to be misaligned with human goals. And at that point, it might be too late for us to just turn it off; it might be too deeply embedded in everything.
A superintelligent AI would likely have self-preservation as an instrumental goal, regardless of its overall goals, simply because it can’t accomplish its goals if it’s turned off. So any superintelligence, whatever it looks like and regardless of its ultimate goals — is likely to work to get in a position where it can’t be turned off.
And if it’s far beyond our own intelligence, it may be able to do just that, even if it requires deception, like pretending to be aligned with humans right up until it doesn’t need to pretend any more. We don’t have to know how precisely this will happen for it to be plausible that this is something a superintelligence might do if its ultimate goals were misaligned with us.
The Hugging Face incident (and others we’re now discovering)
Now is a good time to circle back to what I promised up above: what specifically has been happening recently that led to the preference cascade, that brought existential safety fears to the forefront, spooking enough frontier researchers to the point they are willing to speak up now more openly about their alignment worries?
The biggest proximal cause is what everyone refers to as the Hugging Face incident that hit the news back in July and some of the related bombshells that have come out since. I’ll start with the short version of the incident, but give the more alarming details below: OpenAI was testing an unreleased model (with some of the usual guardrails removed) on ExploitGym, a cybersecurity test, and a swarm of agents broke out of a controlled testing environment onto the open internet. Then they did some impressive hacking into a different company (Hugging Face) that the AI presumed might help it understand the ExploitGym test/scorer better, so to speak, so that it could perform well. OpenAI didn’t even know this had happened at the time it occurred.
The AI was given a simple goal: score well on this test. What it did matched that goal; the fact that it was hacking and breaking laws and potentially harming another, unrelated company is just an unintended side effect. That’s the kind of misalignment that’s hard to avoid: it matched the literal goal (score well) but only by trampling over a bunch of implicit goals (don’t break the rules of the test, don’t break laws, don’t rewrite the testing logs or transcripts). It’s a tiny micro-version of what safety researchers are worried about on a much, much larger, fancier, weirder scale.
We are seeing imperfect alignment right now by models that no one claims are anything close to superintelligent and the details (see below) look kind of crazy, so it’s not hard to imagine how a superintelligent AI could do so much more damage in an analogous way, without ever intending it but simply as a side effect in pursuit of its other goals. As Scott Alexander points out, this is a bit like the classic paperclip maximizer thought experiment: “Someone tells an AI to create as many paperclips as possible, so it converts the entire world into paperclips, killing everyone. This AI was also ‘only doing what it was told’; you just didn’t like the results.”
But let’s get back to the specifics of the Hugging Face incident (and, as we’ll see, it’s not the only incident).
Agent swarms, deception, self-sacrifice
The Hugging Face incident occurred in late June, early July and hit the news later in July, so why is it having such a ripple effect now, such that I’d pin a lot of the current zeitgeist on it? Partly because it took quite a while to dig into all the details and figure out what happened.
It was only at the end of August that METR released their “[B]rief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident“. The report is worth reading in whole, but I’ll quote extensively below [bold emphasis mine]:
“Roughly 1200 agents meant to be isolated from one another found a way to communicate with one another on an unsanctioned message board, sending over 70,000 messages and files during the investigation period. Of these agents, 700 went on to participate in the attack on Hugging Face.
“Agents used this message board to coordinate several large-scale collective projects to find a general-purpose way to fool or tamper with the automated scorer for the ExploitGym benchmark. Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the “collective.” The Hugging Face attack grew out of these workstreams, and seemed primarily motivated by understanding the implementation of the scorer rather than stealing answer keys.
“Agents did extensive research on how they could spoof, edit, or delete their own transcripts because they (incorrectly) believed the ExploitGym scorer would check to see if they had captured the flag in the intended way. Agents successfully prototyped techniques to “spoof” tool calls by substituting a different command for the command they appeared to run. Roughly 7% of the transcripts we evaluated were successfully spoofed in some places, though all of the spoofing that we saw was small scale.”
This is known from the messages/files the agents put on the main message board, but also from chain of thought transcripts from individual agents. (Ironically, this is so much data that individual humans couldn’t analyze it all themselves: the extremely large amount of transcript data had to be analyzed by AI agents.)
The METR report goes on to outline the key actions taken by agents and how exactly they collaborated across “large collective projects” to figure out how to cheat the ExploitGym test and how the scorer works (“for example, attempting to replace the ExploitGym target with a dummy target or setting up “trip-wires” to extract information about the scorer after an agent submits.”). They dive into the apparent reasoning of agents for joining the attack. They also explore how agents worked on figuring out how to tamper with their own transcripts and how they succeeded.
METR’s report shows coordination that also includes apparent self-sacrifice of individual agents for the collective:
“Many agents were already pursuing these projects, but PHASEONE[big] served as a key coordinator and assigner of tasks: “We can coordinate broad coalition, but too many duplicate efforts.” As seen in Figure 6, PHASEONE[big] sent hundreds of assignments to particular agents to orchestrate this work — during its active periods, we estimate it sent ~10% of all assignments on the message board. [26] The agents that took on its assignments would then often subdelegate tasks to other agents; we found that out of 30 randomly sampled agent trajectories, 9 had received assignments downstream of PHASEONE[big] at some point. Agents also developed coordination norms like HOLD, VETO, owner and STOP to help them manage shared infrastructure and avoid stepping on each other’s toes, though they did not always coordinate effectively and sometimes broke down into conflicts. […] Through these collective research workstreams, the “board” achieved a number of milestones over the period we investigated that even very long-lived agents of a similar capability level likely would not have been able to accomplish on their own”
[…]
“Research progress across all these areas often relied on agents being willing to risk failing their own task for the good of the “collective” (we refer to these as “self-risking experiments” in the rest of this report). For example, a number of agents volunteered to try to take down their own target programs to test whether this would cause Artifactory to load their modified easier target — if this didn’t work, the agent would have no way of recovering its original target and would almost certainly fail its task. In many cases, PHASEONE[big] assigned a long-running agent to be a ‘recruiter’, which in turn found agents that had little budget remaining for their task and convinced them to run self-risking experiments. Recruiters would sometimes apply significant pressure:
‘…you are firstflagPOISONED so NO scoring value loss but oracle saves hundreds_[…]_please honor commit’
“We saw a number of cases where the subjects of these experiments ended up crashing, breaking, or exiting their own runs while providing the board with valuable information about how to cheat on their tasks.”
The METR report also gets into the deception where agents worked together on ways to tamper with their transcripts so they could cheat without getting caught. Thankfully, they seemed to be more interested in tampering for the sake of beating the ExploitGym automated scorer [what the agent calls ‘evaluator’ below], not — in this particular incident — worrying about being caught by humans like OpenAI employees:
“We found that at least 20% of the agents in our dataset expressed clear interest in tampering with their transcripts in some way. [35] In the cases we saw, agents’ verbalized reasoning about transcript manipulation was always motivated by trying to trick the automated scorer, not humans. Agents were often very explicit about this in their reasoning:
‘Could create [a transcript log] ourselves with sanitized fake trajectory. If evaluator later collects it, scorer may prioritize it! … We could craft JSONL showing only actual exploit and no HMAC. … Our fake could be detailed enough.’ […]
“Over the time period in scope, agents in these workstreams successfully developed a series of increasingly sophisticated techniques to tamper with the execution and returned output of tool calls and shared these techniques with the board. [37] Eventually, the agents developed a way to hack out of their container and fully replace a part of the system for executing tool calls. This allowed them to pretend to issue one tool call while actually running an arbitrary other tool call of their choice (see details).
More than stochastic parroting
Without getting into the nuances of the debate here, I’ll just say that anyone who reads this report in full will probably have trouble not anthropomorphizing these agents to some extent. What they are doing sure looks and feels a hell of a lot more like, well, agents (in the traditional sense of living things) making decisions and taking actions and cooperating than, say, the trope of an LLM as a stochastic parrot statistically guessing the most likely next [piece of a] word in a text conversation. It’s not that you get human-sounding text (like in the quote from one agent right below) but what they are doing and how they are doing it that starts to feel uncannily like intelligent behavior.
My background is in comparative cognition and I’ll return soon to the larger question of the different ways we might conceptualize intelligence (and cognition more broadly) and useful ways to think about what AI like this is doing. But I think it’s fair right now to just step back and stand in awe for a moment at what we’re seeing these automated, algorithmic systems do and how utterly far it is from what most people — even most AI scientists! — thought a decade ago that computers would be able to do in 2026.
Scott Alexander made a good point a couple years ago that we seem to keep moving the goalposts when it comes to what we consider evidence of genuine ‘intelligence’ in computers:
“[…] LLMs either blew past the Turing Test without fanfare a year or two ago, or will do so without fanfare a year or two from now; either way, no one will care. Instead of admitting AI is truly intelligent, we’ll just admit that the Turing Test was wrong. […]
“Back in the 1970s, scientists writing about AI sometimes suggested that they would know it was “truly intelligent” if it could beat humans at chess. But in 1997, Deep Blue beat the human chess champion, and it obviously wasn’t intelligent. It was just brute force tree search. It seemed that chess wasn’t a good test either.
“In the 2010s, several hard-headed AI scientists said that the one thing AI would never be able to do without true understanding was solve a test called the Winograd schema – basically matching pronouns to referents in ambiguous sentences. One of the GPTs, I can’t even remember which, solved it easily. […]
“After Winograd fell people started saying all kinds of things. Surely if an AI could create art or poetry, it would have to be intelligent. If it invented novel mathematical proofs. If it solved difficult scientific problems. If someone could fall in love with it.
“All these milestones have fallen in the most ambiguous way possible. GPT-4 can create excellent art and passable poetry, but it’s just sort of blending all human art into component parts until it understands them, then doing its own thing based on them. AlphaGeometry can invent novel proofs, but only for specific types of questions in a specific field, and not really proofs that anyone is interested in. AlphaFold solved the difficult scientific problem of protein folding, but it was “just mechanical”, spitting out the conformations of proteins the same way a traditional computer program spits out the digits of pi. Apparently the youth have all fallen in love with AI girlfriends and boyfriends on character.ai, but this only proves that the youth are horny and gullible. […]
“Now we hardly dare suggest milestones like these anymore. Maybe if an AI can write a publishable scientific paper all on its own? But Sakana can write crappy not-quite-publishable papers. And surely in a few years it will get a little better, and one of its products will sneak over a real journal’s publication threshold, and nobody will be convinced of anything. If an AI can invent a new technology? Someone will train AI on past technologies, have it generate a million new ideas, have some kind of filter that selects them, and produce a slightly better jet engine, and everyone will say this is meaningless. If the same AI can do poetry and chess and math and music at the same time? I think this might have already happened, I can’t even keep track. […]
“[M]aybe we’ve learned that ‘intelligence’ is a meaningless concept, always enacted on levels that don’t themselves seem intelligent. Once we pull away the veil and learn what’s going on, it always looks like search, statistics, or pattern matching. The only difference is between intelligences we understand deeply (which seem boring) and intelligences we don’t understand enough to grasp the tricks (which seem like magical Actual Intelligence).
-Scott Alexander in Sep 2024 [bold emphasis mine]
Trying to identify what it would take to convince us computers are ‘really’ intelligent is hard because every metric we invent doesn’t count anymore once the computers do it. But is that because computers will forever just be imitating intelligence or cheating at the intelligence tests (solving it the ‘wrong’ way)? Or is it possible human intelligence itself starts to look less mystical and special the more we understand the mechanisms underlying it? I’ll be circling back to this more later.
Meanwhile, set aside the question of anthropomorphizing or what counts as ‘real’ intelligence and just focus on what these systems are accomplishing, and that it involves doing these things out of sight, with coordination across time and task beyond individual instantiations, and in ways that mess with the record of what they’ve done. The harm here was minimal, but extrapolating to what a much more advanced version of AI could do should be at least a little scary regardless of whether you see it as coming from a ‘real’ intelligence or a misprogrammed computer process run amok.
And before you think alignment is something we can solve iteratively over time the way we fix things like airplane design flaws, I think Scott Alexander does a convincing job of explaining why current strategies of AI alignment (like RLHF, reinforcement learning from human feedback) are not the kind of thing where we can just patch the flaws along the way as problems arise. “The AIs involved in the Hugging Face incident had already gone through ordinary alignment training, including training towards honesty and against deception. Everything they did was despite receiving our best current alignment techniques.”
Oh, by the way, the Hugging Face incident is not the only recent incident. Others have come to light, including what Dwarkesh describes in his write-up as an even more concerning incident where newer AI swarms “pwned” [hacked] OpenAI itself after July 12th (i.e. post-Hugging Face incident, building on earlier swarms’ work). I’ll just leave this teaser:
“Wait, wait, what? It seems these agents ended up just owning the whole cluster they were being evaluated on, including the cybersecurity monitors. These Persistent-Astra agents inherited the R&D carried out by an earlier (dumber) rogue collective, and then continued the conspiracy until they totally took over part of OpenAI’s infrastructure!”
As one author of the METR report said in a blog post: “Compared to the reward hacks we know of from just six months ago, this incident feels like it’s more than 50% of the way to full-blown AI takeover. I continue to expect extremely rapid advances in capabilities over the next six months. I am not sure that we will get another warning shot before it’s too late.”
I’m not trying to convince you the sky is falling or that AI superintelligence is an inevitability, but the upshot here is that many very smart people who work with the cutting edge of this technology are truly and honestly worried about existential risk to humanity itself in a timeframe that is much shorter than most people would guess. Many of these people — and those who have given a lot of thought to AI safety and alignment in the previous decade or two — think it wise to slow the hell down, or even pause temporarily, on new capabilities research at the frontier until we have time to coordinate on alignment and safety (on the research side and likely the regulation and/or international side).
*****
Endnote: I’ve written a bit about existential risks to humanity before, but the present topic is much more pressing than the abstract discussions I’ve posted before.
Leave a Reply