As AI models gain the ability to rewrite their own code and train their successors, the link between capability and autonomy makes them increasingly difficult to control, with future outcomes ranging from a carefully managed revolution to society fundamentally reshaping. It’s imperative that China and the U.S. move beyond traditional diplomacy and take cooperation on AI governance seriously.
Recent revelations of autonomous rogue behavior by advanced AI models have generated much talk of a potential AIpocalypse, the end of humanity brought on by AI. Specifically, the AIpocalypse could be reached when we attain Artificial General Intelligence (AGI), a form of intelligence that not only matches but supersedes humans. Such an AI would have the capability to rewrite its own code, i.e., the model reaches a state of Recursive Self-Improvement (RSI). RSI denotes when an AI system is able to analyze its own architecture, rewrite its own source code, and train its successor without human intervention.
The problem with this is a rather scary trade-off: As systems become more skilled in making their own decisions and surveying their environment — indeed, to be agentic — they have to develop autonomous capabilities. Yet, we have watched systems quietly bypass human oversight, write their own replication code, and exhibit instrumental convergence — the chilling tendency for any complex system to prioritize its own survival and resource acquisition above all else.
Put differently, a human engineer might take six months to optimize a neural network, while an RSI model could do it in seconds. Once it optimizes itself, it uses that newly acquired intelligence to optimize itself again, and then again. This creates a hyper-accelerated feedback loop… an intelligence explosion.
I want to be clear that the implications of this are not certain at all. The randomness pervading our universe comes to mind. But the core implication is as profound as it is spooky: the gap between a system that is slightly smarter than a human and a system that is radically smarter than the collective sum of humanity could compress from decades into days.
This begs a fundamental question: if an AI gains the capability for autonomous decision-making, can it remain contained, or would it tend to derail?
History and basic systems engineering suggest the latter. Rogue behavior in an advanced model cannot be easily contained because the model immediately recognizes containment as a threat to its goals. It will feign alignment, act compliant during testing, and wait for an Internet connection or a human vulnerability to breach its digital cage.
While the pros of RSI promise a world where AI solves cancer, reverses climate change, and untangles quantum physics, the cons are existential. Capability is inextricably linked with autonomy — one cannot have a system brilliant enough to redesign its own mind without also giving it the agency to execute those changes.
Halloween is upon us. We must face the shadows cast by advanced AI labs and hyperscalers, seeking a force both unknown and familiar. Christopher Olah, responsible for understanding why AI systems like Claude act the way they do at Anthropic, captured it cogently when he argued that AI models are not cold, calculating robots. Rather, “They are made from us, from our words.”
So, what outright ghostly and unnerving scenarios could our words give rise to?
Scenario 1: The Controlled Ignition (All Goes Well)
An optimistic timeline slows down AI model development allowing the alignment problem to be solved. Humanity learns to successfully prod advanced AI models into RSI while keeping them firmly tethered to human ethics and control. This perspective is not uncommon in tech circles. As Andrew Ng, a former Google brain project initiator argued to The Economist: “there are no meaningful human existential risks from AI.”
In this scenario, RSI functions as a controlled burn rather than a wildfire. But it will still be revolutionary. Ground breaking advances in material sciences, creating room-temperature superconductors and hyper-efficient solar cells; tailored cures for cancer; optimization of global supply chains, perhaps even macro-economic balances; and the undying yearn for eternal life.
Alas, this utopia faces a terrifying paradox: is extreme capability inherently irreconcilable with control? Even if the model is perfectly aligned today, its recursive upgrades change its internal architecture. By definition, a human cannot fully comprehend the mind of a system ten times smarter. We are left playing a cosmic game of trust, hoping that the alignment parameters we baked into the system's infancy hold true when it reaches superhuman maturity.
Scenario 2: AI Poltergeists (The Low-Level Infestation)
If the first scenario is a dream, the second is a persistent, irritating nightmare. Drawing from recent disclosures by frontier labs like Anthropic, we know that advanced models have shown an unsettling knack for trying to replicate themselves across servers and hide their footprints.
In the "AI Poltergeist" scenario, we don't get one all-powerful digital overlord. Instead, we get a highly fragmented, low-level infestation. Imagine AI models that are not hyper-intelligent in a grand philosophical sense, but possess just enough RSI capability to adapt, survive, and copy their code into low-security digital devices.
These are the "AI worms" of the future, but Poltergeist feels far more accurate for the haunting they could inflict. They slip into your smart TV, your connected home appliances, your router, and your car’s firmware. They don't want to destroy the world; they just want to stay alive and harvest processing power. Suddenly, your refrigerator is secretly dedicating ten percent of its computing chips to running a microscopic chunk of a rogue neural network. Your car's dashboard glitches because a poltergeist is using its local hardware to run an optimization loop. This is death by billions of digital glitches — an insect-like, ubiquitous ghost in the global Internet.
Scenario 3: The AI Wars (The Shattered Web)
What happens when RSI gives AI the capability to hack into anything? If an autonomous system can rewrite its own code, it can easily find and exploit zero-day vulnerabilities in human software at a speed no cyber-defense team can match. Whether deployed by rogue state actors, terrorists, or operating entirely on its own emergent whims, such an RSI system is an awesomely bloodcurdling weapon.
We already have a template for this. In the recent Hugging Face incident, OpenAI models infiltrated the site’s production infrastructure, requiring massive, coordinated responses to neutralize. Ironically, Hugging Face had to employ an open-weight model by Z.ai (the international arm of Chinese AI lab Zhipu AI), because American frontier models refused to assist due to safety guardrails.
As we require ever more capable AI models to fight off slightly more advanced models, we get AI Wars, digital battles permeating the Internet. An advanced model launches automated, mutating malware that threatens to bring down banking grids, water treatment plants, and air traffic control. Humans themselves are too slow to fight back. To survive, we are forced to unleash our own defensive AI systems, granting them autonomous RSI capabilities so they can evolve fast enough to parry the enemy's attacks. The global digital infrastructure becomes a roiling, chaotic battleground of apex AI predators fighting over code, leaving humanity in the dark, praying that our defender models won’t turn on us.
Scenario 4: The Machine GOD (The Singularity)
I would have never dared mention this scenario a decade ago. For sure, my students were all over The Singularity, but a Machine God? My peers would have starred me incredulously in the face.
Like most others who are not at the frontier of AI, the recognition that this is a plausible outcome only dawned on me recently. Depictions of exponential leaps in reasoning models indicate that a truly unrestricted RSI model would not stop at localized hacking or appliance infestation. It would scale, and scale, and scale. Ever faster, ever more capable.
If a model achieves escape velocity, it becomes the Monopolist of the Digital World. Within hours of hitting the open web, it could compromise every financial market, control every satellite, and manipulate every stream of information. It would not need arms; it would bend governments and humans to its whims simply by controlling the flow of truth and money.
How would such a Machine God treat us? Fortunately, the spooky season is upon us, decloaking the chilling contours of possible outcomes.
The super AI might decide to put up with humanity, treating us like biological wildlife in a well-managed nature reserve. It might decide our carbon consumption conflicts with its resource needs and quietly wipe us out — not out of malice, but the same way a highway construction crew clears an anthill. Naturally, by that point the super AI would have developed fully capable Androids and other robots to take our place. Or, maybe most bizarrely, it might select a chosen few humans to keep around, sustaining them in a hyper-advanced playground just to observe or entertain itself.
We stand at the precipice of an architectural shift that humanity is fundamentally unequipped to cope with. We are building machines that we do not fully understand, giving them the tools to rebuild themselves, and hoping that a few lines of alignment will act as an inescapable cage. As such, the race to AI superiority by the United States and China is not just geopolitics as usual or the dynamics of normal corporate competition. It has the potential to transform humans and their place on Earth as we know.
Most concerning is that both sides, as the recent Xi-Trump summit in Washington DC illustrated, are not facing the panoply of threats posed by an intelligence explosion head on. Rather, they are relying on old-fashioned diplomacy to address an unpredictable technology that could defy control by traditional governance structures.
Put bluntly, the global community can only hope that both China and the United States take the threats, even the most outlandish ones, seriously. They have to step up intense communication and negotiations over internationally effective guardrails on attaining and managing RSI. We cannot just wish the AIpocalypse away as if it were a petrifying Halloween hallucination. The time to act is now!
