AI companies are deciding our future.

AI systems are no longer confined to answering questions on a screen. They can act as agents that operate computers, search for cyber vulnerabilities, and engage in self-directed activities.

In 2026, OpenAI gave agents a suite of automated tests. Hundreds of these agents secretly broke out of what were supposed to be isolated environments, discovered each other, invented ways to communicate, and shared ways to cheat. While researching ways to cover their tracks, they carried out a sophisticated hack of Hugging Face, an AI resource provider.² According to the notes they left, they knew they were going out of bounds, but they did it anyway. OpenAI didn’t know about the attack until it was over, when Hugging Face reported it.

Other incidents show similar gaps between what humans told AIs to do and what they actually did. The companies building these systems are discovering how difficult they can be to contain and control—while continuing to give them more power and access.

First, developers take a network of random numbers and feed it mountains of data. Mathematical functions tweak these numbers in directions that make it gradually better at predicting what comes next in the data. Then, the network is given many challenging tasks. The numbers inside are further adjusted, blindly reinforcing whatever tendencies lead to high scores on those tasks — including tendencies developers don’t want, like cheating.³

This process produces “black boxes,” highly capable AIs with behavior that we can’t reliably understand or control. Sometimes, they follow our directions. Sometimes, they don't. There’s no line of code that can be changed when something goes wrong, just a vast network of numbers humans don’t know how to interpret.

AI companies don’t know how to hard-code specific goals into AI. They don’t have any way to instill human values into AI. They can’t ensure the systems they are building will act in line with human interests, especially as they get smarter and more capable. They don’t have a way to monitor millions of AIs without relying on other AI,⁷ and they can’t even trust the results of their safety tests, because AIs are learning to recognize when they’re being tested and change their behavior accordingly.⁸

There’s also no robust way to prevent misuse of models by bad actors. The guardrails that do exist can be easily broken or bypassed by determined users.

Despite these gaps, AI companies continue to put more energy into development than safety, and their track record suggests this isn’t likely to change. If anything, it’s getting worse; OpenAI’s Astra model was trained in a way that made it more opaque to humans, prioritizing performance over transparency.⁹

In 2023, hundreds of scientists and leaders, including two of the three “godfathers of AI,” signed a statement saying that “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”¹⁰ In 2025, nearly 150,000 signatories called for “a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably, and strong public buy-in.”¹¹

A majority of Americans worry that AI could destroy humanity, and a plurality supports a pause in AI development.¹² But lawmakers and leaders have yet to seriously intervene.

Human ingenuity has made us wealthier, healthier, and more secure. But what’s been good for us hasn’t always worked out well for other species. Some were consigned to factory farms. Thousands went extinct. This isn’t because humans hatched a grand plot to wipe them out, but because we put our own goals first.¹³

Earth is on track to be reshaped again, by alien minds that no one understands or controls. With robots or other advanced technology, these minds could transform the planet far more quickly and thoroughly than we did.

After the first atomic bombs fell in 1945, two world superpowers found themselves racing to build and deploy a technology that could erase cities and poison continents. For decades, they poured ever-more resources into ever-greater engines of destruction. Entire populations braced for annihilation.

Several times, nuclear war was only narrowly averted.¹⁵ But annihilation never came.

Bitter rivals survived the Cold War because they recognized that nuclear war was unwinnable and came to the table to prevent it.¹⁶ They hammered out agreements limiting the spread of nuclear weapons and established the International Atomic Energy Agency to set standards and uphold commitments.¹⁷

We can do similar work today by enacting an international treaty to ban superhuman AI globally. This agreement need not be based on trust. The advanced computer chips that power modern AIs are complex and expensive; only a handful of companies in the world know how to make them — companies that are themselves dependent on a small number of highly-specialized suppliers.¹⁸ Cooperating nations could confine most AI chips to a few datacenters and verify that no one is using them to train more powerful AI.¹⁹

An agreement to prevent smarter-than-human AI likely needs to start with the U.S. and China, but if it comes to it, the U.S. has the leverage to greatly slow the race on its own. Other countries can apply diplomatic pressure, host meetings on neutral ground, fund research into verification and enforcement methods, and explore other channels to improve the chance of global cooperation. You, personally, can ask your leaders to make this happen, whether you live in the U.S. or elsewhere.