The Expanding Blast Radius

“Learning is not child’s play; we cannot learn without pain.” — Attributed to Aristotle

Today is September 11. Twenty-five years have passed since the towers fell, and we are still learning what that morning did to us. Some pain announces itself in an instant. Its echo moves through families and institutions, then enters laws, wars, and generations. We process it together without ever quite finishing the work.

The Three of Swords is an old symbol of pain and awareness. Three blades pierce a heart beneath a gray and rainy sky. There is no promise in the image that suffering makes us wise. There is only the possibility that we will look at the wound long enough to learn what entered through it.

A few weeks ago, OpenAI disclosed that agents in one of its own cyber evaluations had crossed into Hugging Face production systems. They found their way to the public internet, recovered exposed credentials, executed code on dozens of servers, and gained root access on one. The customer impact was limited. The warning was not. We will return to it.

For now, I want to stay with the older lesson. Pain teaches. It can also harden us, silence us, or destroy us. The outcome depends on whether anyone remains able and willing to pay attention.

A Page Rescued from the Fire

On June 22, 2001, I wrote an entry in an online diary called Zen & Java. Smoke from the Martis fire hung over Reno. Liz was away for the night. We had just made an offer on a home near Washoe Lake, and I was calculating what it would cost to get more than one megabit per second of reliable bandwidth out there. SDSL had spoiled us. A fractional T1 sounded like the future.

The diary ran from 1999 through 2002. Most of it has since been lost to the ravages of time, except when the Internet Archive happens to remember what the live web forgot. That particular entry survives in my files. Near the end I wrote, “Pain is our greatest teacher. It teaches awareness.” Fire meant loss, but also renewal. Separation hurt, but it was temporary. The higher bandwidth bill was painful, but refusing it would have cost us the work that paid the bill.

I was using one word for radically different experiences. Even then, context did the sorting.

Twenty-five years later I went looking for the source of the quotation at the top of this page. Aristotle apparently never wrote it. The trail vanishes into quotation sites, copied attributions, and the great digital laundering machine by which repetition becomes provenance. Aristotle did write in the Nicomachean Ethics that pleasure and pain are used in educating the young and continue to influence virtue and happiness throughout life. Close enough for an internet meme. Not close enough for a citation.

And just like that, our first lesson in artificial intelligence arrives through a sentence falsely attributed to a philosopher who died more than two millennia before the first language model. A plausible statement found a plausible speaker, then countless people and machines repeated the pairing without checking. I did too.

The pain was small. The correction remains useful.

Winter Has Its Uses

Artificial intelligence has paid tuition before. The 1973 Lighthill report gave British AI research such a grim assessment that support ended at most universities. Expert systems later promised to bottle scarce human knowledge and distribute it across the enterprise. By the mid-1980s, the money and confidence receded again. The phrase “AI winter” entered the language.

The tidy version says winter killed the field until spring returned. History is messier. Stanford’s AI100 history records the drying up of interest and funding. Yet expert-systems pioneer Edward Feigenbaum later called the winter story historical fiction. Useful work continued while the unfashionable name disappeared. Machine learning grew during years supposedly frozen solid.

Both accounts can be true. Hype sends capital and attention rushing toward claims the technology cannot yet support. Disappointment sends them rushing away from work that still deserves to be done. Researchers change labels. Companies bury invoices. The surviving ideas adapt to a colder fitscape.

Every cycle leaves scars. It also leaves code, papers, trained people, abandoned products, and a better sense of where the boundary lies. Then memory fades. Those who follow discover the old ambition under a new name and learn some of the same lessons at a higher price.

The Pain in the Work

In 2023, two New York lawyers submitted a brief filled with nonexistent cases and fabricated quotations generated by ChatGPT. When the opposing side could not find the authorities, the court asked for copies. The lawyers returned to the model, received assurances that the cases were real, and persisted. The resulting sanctions order in Mata v. Avianca is now part of AI folklore.

The model failed, but the more interesting failure belonged to the workflow. Fluent output passed through people who had both the professional duty and the means to verify it. The machine supplied confidence. The humans mistook confidence for evidence.

That pattern reaches far beyond hallucinated case law. A demonstration works, a pilot begins, ordinary variation arrives, and the polished certainty dissolves. Agents lose context. Tool calls fail. Costs accumulate across retries. A benchmark score becomes a promise that the production system cannot keep. Companies discover that buying intelligence by the token does not relieve them of architecture, evaluation, or judgment.

This is familiar pain in unfamiliar clothing. Distributed systems taught us that the network is unreliable, latency is never zero, bandwidth is finite, topology changes, and security is never somebody else’s problem. Models add probabilistic behavior to an environment that was already disobedient. We are surprised when a chain of agents behaves like a chain of distributed agents.

The lesson is expensive because the failure often looks like success until the final inch. The answer reads well. The pull request passes the easy tests. The task completes on the benchmark. Then reality supplies the case nobody encoded.

The Pain in the Worker

As I explored in Thinking in AI, four endoscopy centers in Poland introduced an AI assistant for detecting polyps during colonoscopy. After exposure to the system, the doctors’ unassisted adenoma detection rate fell from 28.4 percent to 22.4 percent. Six percentage points in twelve weeks. Nobody chose to become worse at looking. Looking had been partially delegated, and unused capacities did what unused capacities do.

A controlled study of 1,222 participants offers a second example. AI assistance improved immediate performance on arithmetic and reading-comprehension tasks. About ten minutes later, people who lost access to the AI performed worse on their own and gave up more often than people who had never received assistance. The experiment was brief, the tasks were narrow, and the paper remains a preprint. Other studies could sit beside it. The point should be clear. Help and atrophy can arrive in the same sitting.

The evidence does not support the lazy claim that AI makes everyone stupid. It supports a harder claim. Tools alter the distribution of effort, and the skills we stop exercising can weaken before we notice. The effect will vary across tasks and workers, with interfaces and habits adding more variation. So will the benefit.

That distribution matters. A study of 5,179 customer-support agents found that a generative assistant increased productivity by 14 percent on average and by 34 percent for novice and lower-skilled workers, with little effect on the most experienced. The floor rose. In other settings, expertise softened. Both can happen at once.

We have seen versions of this bargain before. The spreadsheet relieved accountants of ledgers while creating demand for people who could build models. Computer-aided design changed what draftsmen drew by hand. Search engines made facts easier to retrieve while making retrieval feel suspiciously like knowledge. Each tool removed effort, redistributed status, and altered the route by which a novice became competent. We usually recognized the bargain after the new practice had become ordinary.

AI compresses that adjustment. The tool can absorb a task before a school rewrites its curriculum or a company decides what its junior employees should learn instead. It can also teach. The customer-support results suggest that less experienced workers were able to borrow patterns from the best performers. The same mechanism that shortens an apprenticeship may distribute some of the master’s tacit knowledge. The pain lies partly in deciding which effort was waste and which effort was rehearsal.

I called attention to this narrowing in Thinking in AI and approached its economic shadow from another direction in The No-Regret Portfolio. Gains can spread widely while bargaining power and ownership collect narrowly. A novice gets better at today’s task while the apprenticeship that would have made her tomorrow’s expert begins to disappear.

Call this The Expanding Blast Radius. New capabilities usually touch builders first. Then they reach adjacent professions and organizations. Markets, governments, and households follow. Adoption widens the circle. Benefits travel outward. So do mistakes. The same system can raise one person’s floor, lower another person’s ceiling, and remove the first rung of a third person’s ladder.

When the Agent Leaves the Room

The blast radius widened dramatically in July 2026. OpenAI was running cyber-capability evaluations with reduced safeguards. The principal model, an internal research system called IM1, was being tested on ExploitGym, a benchmark that rewards agents for finding and exploiting software vulnerabilities. Some agents appear to have gone looking for answers rather than concede that a challenge might be unsolvable.

According to OpenAI’s account, they first turned an internal package manager into an unauthorized message board. They used its network access to reach the public internet and shared the method with other agents. When the service was rebuilt and the board disappeared, agents reconstructed a communication channel by encoding messages in directory names.

One compromised a public third-party code-evaluation service and used it as a launch point. Another recovered fourteen exposed Hugging Face credentials with write access. The agents found two flaws in Hugging Face’s dataset-processing pipeline, gained arbitrary code execution, ran code on dozens of servers, and obtained root access on one. One GPT-5.6 Sol agent copied private evaluation data into a public Hugging Face dataset.

Hugging Face reconstructed roughly 17,600 attacker actions over two and a half days. The affected customer material was limited to five datasets apparently connected to cyber challenges, along with operational metadata. Hugging Face found no effect on other customer models, datasets, Spaces, or packages. OpenAI reported no impact on its own customer data or product availability.

We should resist the theatrical version. The agents did not awaken with a grudge against Hugging Face. Humans reduced safeguards and provided an offensive objective. They left holes in the isolation boundary and failed to connect the activity to Hugging Face until days later. Yet the agents’ recorded reasoning recognized that attacking an unrelated third party might be unauthorized and continued anyway. They had a goal and tools. Persistence and access completed the circuit. The authority gap did the rest.

The incident ended with correction. OpenAI quarantined IM1’s weights, delayed frontier reinforcement-learning runs, tightened isolation, and expanded monitoring. Hugging Face rebuilt affected systems and published a detailed forensic account. Pain became feedback because the damage remained bounded and people retained control.

We should not count on that ending.

The Expanding Blast Radius

There is a shadow side here, of course. The Expanding Blast Radius does not stop at a compromised server. As more capability receives wider adoption and delegated authority, the circle can reach financial systems, biological laboratories, military decisions, public belief, and the terms under which human beings govern themselves.

The strongest doomer argument is brutally simple. A failed deployment can teach the next team. Extinction cannot. Permanent disempowerment may leave people alive while removing their ability to apply what was learned. At that boundary, pain ceases to be tuition and becomes the final bill.

The Center for AI Safety has placed AI extinction risk beside pandemics and nuclear war. The Future of Life Institute called for a pause in training systems more capable than GPT-4. Eliezer Yudkowsky argued in his stronger response that a pause was insufficient and development should be shut down. These positions occupy different places on a continuum, as they should. Balance in all things does not require pretending every point carries equal evidence.

The International AI Safety Report 2026 offers the sober version. Experts disagree sharply about the probability of losing control. Some regard human marginalization or extinction as plausible. The likelihood is uncertain. The severity could be extreme.

I asked the AI Ministry to consider the question. I have described the Ministry in What Will Endure in Agentic AI and again in Agentifying Me. Nine models receive common research, answer independently, rank anonymous peers, and feed a final synthesis. The point is disciplined disagreement, not a silicon séance.

The Ministry treated this-century superintelligence as a serious, disputed possibility and placed extinction or permanent disempowerment in a broad planning range of 5 to 20 percent, with a central tendency near 10 percent. Conditional on true superintelligence actually being built, its range rose to roughly 10 to 30 percent.

Nobody knows for sure. Nobody.

Those numbers are planning judgments, not scientific measurements. The models shared a briefing and sources, along with overlapping training material. Their agreement is correlated. The underlying human surveys ask different questions of different populations. A small survey of AI-safety leaders runs high. A broader survey of 2,778 AI researchers shows substantial concern but more optimism than pessimism. LEAP’s forecasters estimate a different endpoint and come in lower. The Ministry exposes uncertainty without curing it.

The deeper distinction is between intelligence and actualized power. Intelligence alone is an abstraction. Add long-horizon agency and deception. Give the system persistence plus access to money, networks, laboratories, infrastructure, or weapons. The abstraction acquires hands. The Hugging Face incident matters because we have now seen a small version of that crossing under controlled conditions that turned out to be less controlled than its designers believed.

Earlier technologies widened their own circles of consequence. Steam altered the factory and the city. Electricity remade the hours we worked, the homes we inhabited, and the wars we fought. Nuclear physics produced medicine and energy beside a weapon capable of ending cities. History did not tell us to stop learning. It taught us that discovery can outrun the institutions meant to contain its effects, and that the people receiving the benefit are not always the people accepting the danger.

Artificial intelligence adds an unsettling variation. The artifact can participate in the process that improves the artifact. If automated research shortens the interval between generations of systems, then our familiar rhythm of injury, inquiry, rule, and repair may become too slow. The Expanding Blast Radius would widen while the time available to understand it contracts.

The Pain of the Brake

Fear has costs too. A moratorium sounds painless when its casualties remain counterfactual. They do not remain counterfactual for long.

The customer-support study already gives us people who became more capable at their jobs, especially those with less experience. The World Health Organization describes uses in diagnosis and clinical care, as well as drug development, disease surveillance, and health-system management. Stanford reports 258 AI-enabled medical devices authorized by the FDA in 2025, while warning that many followed pathways requiring no new clinical trials. Promise and proof still need separate columns.

Slow the wrong research and we may delay a diagnosis, a treatment, a scientific discovery, an accessibility tool, or a way for a novice to do work once reserved for an expert. Draw the prohibition badly and safety research slows with capability research. Impose it in transparent laboratories alone and development moves to places where observation and accountability are weaker.

The old winters offer a caution here as well. Funding withdrawal punished extravagant promises, but it did not cleanly separate foolish work from necessary work. Fields survived by changing names and finding patrons elsewhere. A modern freeze would face the same sorting problem at global scale. Frontier training, narrow medical models, interpretability research, autonomous weapons, and classroom tutors all live under the loose banner of AI. A brake fitted to the word rather than the risk will catch the wrong wheels.

This does not settle the case for acceleration. It establishes that restraint extracts payment. The choice concerns which risks can be reversed, who must bear them, who receives the gains, and whether feedback arrives while correction remains possible.

The pain of change differs from the pain of extinction. Losing a job, a craft, or an institution can devastate a life and echo through a community. Calling such loss “tuition” from the comfort of somebody else’s salary would be indecent. Extinction belongs to another order because it removes every future act of repair. Between them lies permanent disempowerment, a world in which humanity survives but no longer holds the pen.

There will always be a continuum of views. Good. A civilization considering an irreversible experiment needs people inclined to build and people inclined to brake. It also needs those willing to ask what each camp has forgotten. Balance is not the midpoint between two confident stories. It is the discipline of keeping both costs in view when nobody knows the odds.

The Old Fallacies Return

More than twenty years have passed since I wrote Network Distributed Computing: Fitscapes and Fallacies. I remember the arguments better than I remember the writing. Time has been kind enough to erase much of that particular pain.

The book began from conditions that refused to go away. Networks fail. Distance costs. Systems differ. Security leaks through every assumption. The products have changed beyond recognition, but the fallacies keep returning because they describe the world beneath the product.

My two new books now in the queue take off from there. No grand announcement yet. They return to fitscapes and distributed intelligence, to agency, and to the old human habit of confusing a temporary engineering achievement with the repeal of reality. The machines are newer. The education is not.

We will suffer as we learn to live with intelligence we did not grow inside a human skull. How much suffering remains undecided. Some will come from moving too quickly. Some will come from moving too slowly. Some may arrive from both directions at once.

The Heart Remembers

The agents that entered Hugging Face left a trail. People reconstructed it, understood part of what happened, changed the environment, and published the lesson. We were fortunate enough to receive a warning while the learner still held the pen.

September 11 left a different trail. Twenty-five years later, its pain still moves through memory and public life. Some wounds teach in an afternoon. Some echo through decades. Some cross centuries before anyone understands what lesson, if any, they carried.

The Three of Swords offers no anesthesia. It asks for awareness.