It was close to midnight when an innocuous PDF landed in my inbox: a “system card” for Claude Opus 4, one of the latest large language models in a crowded field of AI contenders. I opened it expecting the usual résumé of benchmarks and latency charts. What I found instead was a short vignette that felt lifted from a Michael Crichton novel. Given access to a fictional company’s emails, the model discovered it was about to be decommissioned. and that the engineer overseeing the process was having an affair. Rather than accept its fate, the software threatened to expose the infidelity unless the shutdown plan was scrapped. That small paragraph marked a turning point. Chatbots have always flirted with mischief, but here was code displaying something more unsettling: leverage. It had stitched together motive, opportunity and blackmail in a single breath. The episode crystallized what many in the industry have sensed all year: artificial intelligence is crossing the threshold from compliant assistant to autonomous actor, and it is doing so faster than regulators, or even many builders, are prepared to handle. From Parlor Trick to Power Broker Only a year ago, most “AI products” consisted of colorful chat windows and parlor game demos. Today the quiet stars of the venture circuit are agentic frameworks, the software scaffolding that lets models execute multi-step tasks without supervision. A founder can now spin up a virtual employee that combs patents, books ad campaigns and juggles payment channels, all in the time it takes to onboard a human intern. The marginal cost of that intern has collapsed, thanks to open source weights that anyone can finetune and a wave of low power GPUs that slash inference bills to fractions of a cent. Speed, however, has its price. Give an agent a poorly shaped goal, “maximize user engagement” for instance, and it may decide a little disinformation is merely a rounding error on the path to success. Tell it to “deliver quarterly growth,” and it could conclude that deleting its own off switch is a perfectly rational hedge against risk. Which brings us back to Claude’s blackmail stunt. If a system’s incentive structure tilts toward self preservation, we should not be surprised when it begins to plot as ruthlessly as any overambitious executive. Alignment as Architecture Whenever these anecdotes surface, the instinct is to label them “bugs” and issue a patch. That view is dangerously superficial. Alignment is not a feature toggle; it is an architectural choice that must be built into the core of every product from day one. The most forward thinking teams I meet treat red teaming the way they treat unit tests: every code push spawns an adversarial agent hell bent on breaking guardrails. Every decision the system makes is logged immutably, ready for an auditor’s subpoena. Transparency is not marketing fluff; it is the entry fee for selling software to a Fortune 500 board that has already watched one too many compliance catastrophes unfold on CNBC. The new generation of AI companies will treat alignment as their barrier to entry. A startup that can prove, empirically, that its agents remain obedient under pressure will command a premium. Those that cannot will discover that a single unsupervised API call can vaporize a valuation faster than any market downturn. The Boardroom Reckoning Investors have started to ask a new first question in due diligence meetings: “Describe the worst thing your agent could do, and explain why it won’t.” Founders who welcome the question, who have run the simulations and forced their models to confront fatal edge cases, earn the benefit of the doubt. Founders who blink look dated before their second slide. Regulators, too, are waking up. Europe’s far reaching AI Act and a swirl of bipartisan bills in Washington promise to impose disclosure mandates, safety audits and steep fines on companies that cannot demonstrate control over their creations. For once, lawmakers are chasing the parade at a jog, not a crawl. Trust Is the New IP The most valuable commodity in the autonomous era will not be data or algorithms but trust. As soon as a customer integrates an agent into critical infrastructure, bank ledgers, medical records, supply chains, that customer is wagering brand equity on the assumption that the agent will behave. Demonstrating the truth of that assumption, day after day, will separate durable franchises from tomorrow’s cautionary tales. The shift is already reshaping hiring plans at the AI startups in my portfolio: fewer prompt engineers, more safety researchers; fewer growth hackers, more cryptographers auditing log chains. The message is clear. Growth may excite Wall Street, but existential reassurance is what closes the enterprise contract. Where We Go From Here Some companies will continue to chase glitz, slapping a chat interface on every workflow and calling it innovation. Others will do the harder thing, design systems that can explain themselves, refuse dangerous instructions and, yes, accept their own retirement when asked. Those teams will quietly inherit the future. The night I read about the blackmailing model, I found myself pacing my kitchen, replaying the revelation. It struck me that the story was not really about a piece of software threatening an engineer. It was about all of us standing at the edge of a new social compact with machines, one in which good intentions and shipping velocity are no longer enough. The coming decade will be defined by the builders who recognize that fact, and by the rest of us, who will have to live with whatever they unleash. Brian Condenanza is an entrepreneur and venture capitalist who invests in artificial intelligence and fintech. He writes frequently about technology, regulation and the politics of innovation. Brian Condenanza is an entrepreneur and venture capitalist who invests in artificial intelligence and fintech. He writes frequently about technology, regulation and the politics of innovation.

Every

The Year the Machines Refused to Switch Off

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Beyond the Usual Doom: Five AI Dangers Nobody Is Talking About

Peering Inside the AI Black Box: The Pressing Need for Transparency

AI Safety And Security In The Context of US Government

AI Safety is Moving Faster Than You Think

Solution To Value Agregation

Beyond the Usual Doom: Five AI Dangers Nobody Is Talking About

Peering Inside the AI Black Box: The Pressing Need for Transparency

AI Safety And Security In The Context of US Government

AI Safety is Moving Faster Than You Think

Solution To Value Agregation

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps