AI May End Us. Someone Has To Say Stop.

The week the message broke containment

On September 9, Jacob Coxon, an Anthropic researcher, resigned and wrote, in a post that circled the globe by lunch: “The people building AI earnestly believe that it could kill us all by the end of the decade.” Then, rarer still, he went on CNN and explained to Anderson Cooper why he had walked away from one of the best jobs in technology.

By week’s end, The Wall Street Journal had a headline that was itself the story: America’s great AI freakout has begun. Not a report on the danger. A report on the fear arriving — “at Bible study groups, family dinner tables and on group chats between friends.”

Nate Soares, who runs the Machine Intelligence Research Institute, gave the Journal the line that explains the week: “The message never broke containment until now.”

Nothing about the technology changed. Not one capability. Not one benchmark. What changed is who was talking about it. This is a diffusion story: one named human being walking away from a good job moved public opinion further in seventy-two hours than a decade of reports and documentaries. Those of us who make arguments for a living should sit with that.

Know the numbers

Let’s look at what AI insiders are saying about the risk levels: Dario Amodei, who runs Anthropic, estimates a 25% chance it goes really, really badly; Evan Hubinger, who leads alignment science there, puts extinction within the decade above 10%; Geoffrey Hinton, who won a Nobel for the work that made this possible, 10% to 20%.

I have no number. And I am wary of the people who do.

The objection I must answer

Soares has an image aimed squarely at my crane. No hammer, he says, has ever broken out of the toolbox to team up with other hammers and pressure the carpenter into buying softer wood. We have left tool territory.

If he is right, my analogy has a hole you could drive a truck through. But he also offers the most humanist sentence in this debate: The world is shaped by human will because we are the smartest creatures on the planet; make something smarter, and the world will be shaped by it.

I have written an entire book on that idea. We disagree only about whether the shaping is still ours. And the crane remains the only version with a human being in the cab.

Why cranes

Tower cranes fall. They fall in Manhattan and Seattle and Dubai. They kill the operator, and sometimes the woman walking below who never looked up.

Nobody answers a crane accident by abolishing cranes. And nobody answers one by pretending the thing isn’t heavy.

What we did instead was unglamorous, and it worked. We licensed the operator. We wrote load charts and made them law. We required inspection before the lift and after the storm. And we put a person on the ground with a radio whose only job is to have the authority to say “stop,” and whose word outranks the schedule, the client, and the machine.

That system never promised the crane was safe. It admits the crane is dangerous, then decides who sits in the cab and who halts the lift.

Nobody has “stop” authority

In July, more than 1,100 employees of OpenAI, Anthropic, Google DeepMind, and Meta signed a letter asking Washington for a mechanism to slow AI development. The builders asked someone else for brakes. Trapped in a race, they fear they could not stop even if they wanted to.

That is a jobsite where the signalman can see the load swinging, and his signal to stop does not outrank the schedule.

It may end us. Every version of this that does not end us runs through a human being who looked at a flawless recommendation and said no, and had the standing to make it stick.

For two years, I have called this the Human Veto, and for two years, it has been filed under soft skills. But if the veto is the safety system, it is not a soft-skills seminar. It is civil defense.

Yet we are training the opposite: a generation that accepts smooth, confident output stripped of the friction that once told us to slow down. That road ends at what I call the wall of gray: everything competent, nothing alive, nobody responsible. You cannot veto what you have stopped reading.

I am not telling you the machine is safe. I am telling you who should be in the cab.

And if they call you an optimist

I have been sold five revolutions during my five-decade career. Desktop publishing. The internet. Mobile. Social. Now this. That does not make me an optimist. It makes me a witness. I have watched the people at the side of the stage, promised the new machine would free them, learn what it had freed them from.

That is the fear underneath the fear. Extinction is the loud version. The quiet one, which surfaces in my workshops when the session runs long and the honesty arrives, is nearer and no less serious: I will still be here, and I will not be needed.

I cannot tell those people the odds. What I can tell them, and last week did not change a word of it: the machine is heavy, the load chart is real, and the hand on the radio is yours.

Learn to say stop.