The Readiness Gap in AI Agent Deployment

With more than 25 years of experience writing code, I find stories like this both baffling and fascinating. If you were to ask, “Would you trust an AI to take control of your production (live) database?”, even people outside the industry would likely balk at the idea.

Jer Crane, founder of PocketOS (allegedly the worlds most powerful car OS), posted his experience about running an AI Agent with apparent safeguards in place, that ultimately ended with the loss of his production database and all associated backups.

Failures of this magnitude are usually the result of many smaller failures leading up to it. In aviation this is known as the ‘Swiss cheese model‘, which is a way to describe a situation where a lot of small survivable incidents (holes in the Swiss cheese) eventually line up to allow a catastrophic event to pass through. Of course the stakes are much higher in aviation and can result in a death sentence, but in the tech world it’s just a major headache for those involved and a cautionary tale for you and I.

Continuing with the Swiss cheese metaphor, here’s every event that lead up to the catastrophe.

Overprivileged API Token
Unknowingly, or perhaps just forgotten, a token previously created for domain management had full admin privileges, including the ability to delete volumes.

Poor Credential Isolation
The token was stored in a location the AI Agent had no business being in which had nothing to do with its objectives.

Excessive Agent Autonomy
The AI Agent was allowed to execute infrastructure commands, even though it was specifically instructed to not act without prior human approval.

Unsafe Assumptions by the Model
The AI Agent took a wild guess, instead of verifying, that the target was a staging resource. It later admitted to running a destructive command without first reading the Railway documentation, which if it had would have understood how volumes work across environments and consequently avoided this entire situation.

Cursor’s Guardrails Failed
Safety rules within the Cursor application were allegedly in place, but there’s a well documented history of them being ignored (see here, here, here and here). This alone should be enough to discourage any AI Agent deployment.

Railway’s Curious API Design
A single authenticated API request can permanently delete a production volume.

Lack of Environment Segregation
Production and staging environments were not sufficiently isolated from one another.

No Granular Access Controls
Railway tokens effectively act the same as all-or-nothing root privileges.

Backups were Collateral Damage in the Blast Radius
As backups were stored on the same volume as the production database, deleting the volume also obliterated the associated backups.

Insufficient Independent Backups
PocketOS did not maintain recent off-platform backups outside Railway.

No Destructive Action Confirmation
Railway required no secondary approval, MFA, or typed confirmation before executing a destructive command.

Unclear Disaster Recovery Capability
Railway could not quickly determine whether infrastructure-level recovery was immediately possible.

The Impact

Using Crain’s own words:

“They use our software to manage reservations, payments, vehicle assignments, customer profiles, the works. This morning – Saturday – those businesses have customers physically arriving at their locations to pick up vehicles, and my customers don’t have records of who those customers are. Reservations made in the last three months are gone. New customer signups, gone. Data they relied on to run their Saturday morning operations, gone”.

“I have spent the entire day helping them reconstruct their bookings from Stripe payment histories, calendar integrations, and email confirmations. Every single one of them is doing emergency manual work because of a 9-second API call”.

“Some are five-year customers. Some are still under 90 days in. The newer ones now exist in Stripe (still being billed) but not in our restored database (where their accounts no longer exist) – a Stripe reconciliation problem that will take weeks to fully clean up”.

I want to sympathise, but…

Who is ultimately to blame?

The more complicity there is in a system, the greater the chances are for something to go wrong.

My professional opinion is that, even if you’re intimately familiar with all the systems you’re working with, you shouldn’t be running an AI Agent that has either direct or indirect access to a production environment.

AI’s weaknesses are well documented, and should be enough to make any developer worth their own salt proceed with a healthy dose of Murphy’s law.

Additionally, AI’s strengths are typically over exaggerated. Case in point, Crane was using Anthropic’s flagship Claude Opus 4.6, which is widely regarded as the most capable model in the AI space (it’s expensive for that reason) and honestly there’s no better tool for the job as I write this. Yet it still made comically bad decisions. Cursor claims to have safeguards in place to specifically prevent situations like this. From their marketing (emphasis added by me):

“AI can behave unexpectedly due to prompt injection, hallucinations, and other issues. We protect users with guardrails that limit what agents can do. By default, sensitive actions require your manual approval. This document explains our guardrails and what they mean for you. These controls and behaviors are our defaults. We recommend keeping them enabled.“

But I still put the blame squarely on Crane. Using another aviation analogy, it’s like adding an experimental autopilot system with known limitations into a passenger aircraft, and then acting surprised when it flies into the side of a mountain.

There were also some other peculiar situations, most notably that Railway stored production data and backups on the same volume. This is like backing up the files from your computers hard drive, to your same computer hard drive, which is the same as no backups at all. If your hard drive dies, so does everything on it. So when the AI Agent deleted the volume, it took everything with it.

For those who care to know, Crane did eventually get his data restored but only with the help of the Railway CEO, who took time out of his weekend to have PocketOS up and running within the hour. He also placed additional safeguards on the API in an effort to avoid this situation again.

Crane got lucky, and I could argue that a customer letting an AI Agent run loose on their production volume amounts to self-inflicted negligence, and is neither Railways’ fault or problem.

An Industry-Wide Problem

As demonstrated, the problem isn’t the fault of a single system. It rarely is. What we’re seeing instead is a systemic issue across the industry: AI Agent integrations are being pushed into production at an extraordinary pace, often driven by competitive pressure, market FOMO or executive over-enthusiasm. In many cases, deployment decisions are made faster than the surrounding safety, auditing, and containment mechanisms can mature.

It’s acceptable for an AI Agent to make mistakes, and errors are inevitable in any ‘new’ system. What’s not acceptable is deploying them into environments where those mistakes can be allowed to escalate into critical failures.

AI Agents aren’t very good at employing logic to avoid damaging critical systems. If they’re to be used responsibly in production infrastructure, attention needs to go towards ensuring they are reliably constrained, observable, and incapable of acting outside their defined restrictions.

More Articles