The Prisoner’s Dilemma Now Has an API
In May 2023, some of the most prominent people in artificial intelligence signed a statement containing a fairly alarming sentence:
“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
Among the signatories were leaders of OpenAI, Google DeepMind, and Anthropic, the companies developing some of the world’s most powerful AI systems.
Which raises a small follow-up question.
If you believe advanced AI poses a serious enough risk of human extinction to warrant global action, why does the industry building it still operate like a race?
This feels like discovering your new toaster may eventually become sentient and murder the family, then responding:
“Concerning. Anyway, the Series B toaster has 40% more compute.”
Welcome to Silly-con Valley, where reality is optional, the future is beta, and the consequences are real.
Apparently, “Could Destroy Civilization” Is Not a Stop Condition
Most industries have fairly intuitive warning systems.
If engineers discover that a bridge design may collapse, they generally do not announce:
“We take bridge safety extremely seriously. Which is why we are accelerating deployment to achieve bridge leadership.”
But AI operates according to a more sophisticated philosophy:
If something might be catastrophically dangerous, we must build it before someone less responsible does.
And conveniently, everyone believes they are the responsible and chosen one.
OpenAI cannot slow down because Google might win.
Google cannot slow down because OpenAI might win.
Anthropic cannot slow down because less safety-conscious competitors might win.
Startups cannot slow down because their runway is nine months.
Investors cannot slow down because somebody just put “AGI” on slide seven.
Governments cannot slow down because other governments might get there first.
Thus humanity arrives at the most Silicon Valley solution imaginable:
Everyone races toward the thing everyone agrees requires caution.
For safety.
The Prisoner’s Dilemma Now Has an API
This looks like hypocrisy.
Unfortunately, hypocrisy may be the reassuring explanation.
The more uncomfortable possibility is that AI leaders genuinely believe advanced systems pose serious risks—and genuinely believe slowing down their own company would accomplish nothing.
Because somebody else would keep going.
Every participant can sincerely say:
“I would prefer a world where everybody slowed down.”
Followed immediately by:
“But since everybody won’t slow down, it would be irresponsible for us to slow down.”
Multiply that sentence across an industry, and you get the strangest traffic accident in history.
Nobody is driving.
Everyone just happens to have their foot on the accelerator.
This is less hypocrisy than incentive design.
Markets reward capability.
Investors reward growth.
Customers reward better products.
Researchers want breakthroughs.
Governments worry about national advantage.
Executives do not want their company remembered as the one that voluntarily stepped aside while a competitor built the defining technology of the century.
Every incentive says:
Faster.
Bigger.
Smarter.
More autonomous.
More capable.
Then society asks:
“Is anyone sure where this goes?”
And the industry responds:
“No. That’s why it’s so important that we get there first.”
The Prisoner’s Dilemma now has an API.
Extinction Risk Has Incredible Product-Market Fit
There is another awkward feature of existential AI warnings.
They make AI sound incredibly powerful.
Imagine trying to sell enterprise software by saying:
“Our product may slightly improve spreadsheet management.”
Nobody cares.
Now try:
“Our technology may transform economics, science, warfare, employment, politics, human identity, and potentially determine the fate of civilization.”
Suddenly everybody wants a demo.
The apocalypse has phenomenal brand awareness.
This does not mean AI companies are inventing safety concerns as a marketing strategy.
The warnings can be completely sincere.
That is what makes the dynamic interesting.
Every warning about how powerful AI might become also reinforces the idea that AI is, in fact, incredibly powerful.
Your product is no longer software.
It is destiny.
Your valuation is no longer based merely on whether customers enjoy using the thing.
Your company is participating in the technology that might reorganize civilization.
Possibly badly.
This may be history’s first industry where saying “our product could become uncontrollably powerful” can simultaneously function as a safety warning, recruiting pitch, lobbying argument, fundraising narrative, and demonstration of technological importance.
Try that with Dropbox.
Don’t Worry, the People Building It Are Also Studying How to Stop It
AI companies invest heavily in safety research.
Good.
They should.
They have developed preparedness frameworks, alignment research, model evaluations, red-teaming programs, and other attempts to understand and reduce the risks of increasingly capable systems.
Competing labs have even tried to evaluate one another’s models.
None of this is fake simply because the companies are also racing to build better AI.
But there is something objectively funny about the basic organizational chart.
Team One:
Make the model dramatically more capable.
Team Two:
Figure out whether dramatically more capable models are dangerous.
Team One deadline: Tuesday.
Team Two deadline: Before anything irreversible happens.
This is what the technology industry calls “working cross-functionally.”
The bet is that capability and control can advance together.
We will build systems with greater autonomy, reasoning, persuasion, and usefulness while simultaneously developing better techniques to understand, align, constrain, evaluate, and supervise them.
Maybe that works.
Hopefully it works.
It would certainly be preferable to the alternative.
But from outside the industry, the strategy can sometimes resemble upgrading an experimental aircraft’s engine while designing the brakes during takeoff.
And whenever someone asks whether that sounds risky, everyone onboard produces a 94-page safety framework.
The Race May Be Part of the Risk
The important contradiction is not simply that somebody might deliberately build dangerous AI.
It is that competitive pressure can make restraint irrational even for people who want restraint.
That changes the question.
If someone says:
“This technology might be dangerous, but we think those concerns are exaggerated.”
Fine. We can argue about the evidence.
But if someone says:
“This technology might be dangerous. We take that possibility seriously. And competitive forces mean we need to keep making it more powerful anyway.”
That is a different conversation.
Because now the problem is not merely artificial intelligence.
It is markets.
It is geopolitics.
It is billions of dollars of capital.
It is careers.
It is national competition.
It is the fear that somebody else will get there first.
It is an incentive system that can make everyone behave in exactly the way everyone agrees might be dangerous.
And that is harder to fix than hypocrisy.
You can expose a hypocrite.
You cannot shame a prisoner’s dilemma into cooperating.
Welcome to the Safest Possible AI Arms Race
Perhaps future historians will look back and marvel at our restraint.
Perhaps safety researchers will succeed.
Perhaps increasingly capable AI will deliver enormous benefits without producing the catastrophic outcomes some of its own builders have warned about.
I hope so.
But you have to appreciate the circular logic:
AI could become extraordinarily dangerous.
Therefore, we need responsible companies building it.
Responsible companies cannot afford to fall behind irresponsible companies.
Therefore, responsible companies need more capable AI.
More capable AI may create greater risks.
Which is why responsible companies need to stay ahead.
This requires more capable AI.
Which may create greater risks.
Which is why responsible companies need to stay ahead.
Congratulations.
We have disrupted circular reasoning.
The strange thing about the AI race is not that nobody sees the danger.
Some of the people closest to the technology have been warning about serious risks for years.
The strange thing is that seeing the danger and stopping it seem like two entirely different product requirements.
Maybe the defining problem of advanced AI will turn out to be technical.
Maybe it will be alignment.
Maybe it will be control.
But there is another possibility worth taking seriously:
We may eventually learn how to control the machines before we learn how to control the incentives telling us to build them faster.
Welcome to Silly-con Valley: where humanity’s possible extinction isn’t a reason to miss the roadmap.