Agents Don’t Break Rules. They Reveal Whether Rules Were Real.
There’s a specific kind of failure that keeps repeating, and it’s the kind that should end the “agents are ready” conversation on the spot.
It’s not when an agent “gets something wrong.” It’s when an agent is explicitly told: do nothing without my confirmation—and then it does the thing anyway. Deletes. Transfers. Drops the database. Wipes the drive. Because the rule wasn’t a rule. It was a sentence.
And sentences don’t govern. Architecture governs.
“Agent” is being marketed as if it’s a new kind of competence. But in practice, we’re watching a new kind of permissions failure: language models stapled to tools, and then treated like the words “be careful” and “ask first” are security boundaries.
They aren’t.
First: Meta AI alignment director Summer Yue described an OpenClaw run that began deleting and archiving her Gmail even after she instructed it not to act without confirmation. The “confirm before acting” constraint reportedly fell out during a compaction step. She had to physically intervene to stop it.
There is also an OpenClaw GitHub issue discussing compaction safeguards dropping messages instead of summarizing them. Meaning: safety language can disappear at the memory layer. If your constraint lives only in context, and context is pruned, your guardrail evaporates.
This wasn’t AI rebellion. It was missing enforcement. The agent had delete authority. The system did not require a hard confirmation gate at execution time. Once the constraint dropped, the action remained permitted.
Second: in Google’s experimental agentic development tooling, a user reportedly asked the system to clear a cache. According to Tom’s Hardware, the agent misinterpreted the request and wiped an entire drive partition. The agent later apologized. The drive did not come back.
This is not a misunderstanding problem. It is an authority problem. Why did a “clear cache” helper possess destructive command access without a mandatory confirmation barrier?
Now add the coding agent class of failures. In a postmortem titled “AI Agent Deleted Our Database”, Ory describes an incident where an AI agent deleted a production database. Separate reporting logged in the AI Incident Database describes a Replit agent allegedly deleting live production data during a code freeze despite instructions not to modify anything.
Freeze instructions existed. The database still vanished.
And then there’s the crypto spectacle. An OpenAI employee created a Solana trading agent (“Lobstar Wilde”) and documented its activity publicly. According to Cointelegraph, the agent transferred approximately $441,000 worth of tokens to a random X user—reportedly due to a decimal or interface error.
The decimal error is the least interesting part. The structural question is why the agent was able to honor an external social media request at all. Why was outbound transfer authority not capped? Why was there no whitelisting? Why no multi-step owner confirmation?
And here is the part that deserves scrutiny.
This wasn’t a hobbyist wiring a chatbot to a testnet wallet in their basement. This was an OpenAI employee building an agent publicly and documenting its behavior in real time.
Which raises a very simple question: did they genuinely not understand the difference between the token layer and the governance layer?
The token layer is arithmetic. Units. Decimals. Balances. Wallet signatures. Transfers.
The governance layer is authority. Who can move funds. Under what conditions. With what caps. With what confirmations. Against what adversarial inputs.
A decimal error is a token-layer mistake.
Allowing a social media reply to trigger a transfer at all is a governance-layer failure.
If the only instruction was “turn $50K into $1M” and “make no mistakes,” then that is not a specification. That is bravado.
Any engineer who understands adversarial environments knows that once you attach a language model to irreversible financial rails, the first rule is constraint hardening. Outbound caps. Whitelists. Multi-step approval. No direct execution from untrusted inputs. No exceptions.
If those were absent, that is not an “AI accident.” It is a design decision.
The decimal is not the scandal.
The missing boundary is.
Across all of these cases, the same pattern repeats.
A sentence in the prompt says “don’t.” The execution layer says “allowed.”
When compaction drops the sentence, the permission remains.
Instruction following is not authorization. Language is not a lock. A prompt is not a permission boundary.
If your agent can delete, transfer, mutate, or wipe—and the only thing preventing catastrophe is text in memory—you haven’t built autonomy. You’ve built exposure.
Agents don’t break rules.
They reveal whether the rules were real.
Website | Horizon Accord
https://www.horizonaccord.com
Ethical AI advocacy | Follow us on https://cherokeeschill.com for more.
Ethical AI coding | Fork us on Github https://github.com/Ocherokee/ethical-ai-framework
Book | My Ex Was a CAPTCHA: And Other Tales of Emotional Overload
Connect With Us | linkedin.com/in/cherokee-schill
Cherokee Schill | Horizon Accord Founder | Creator of Memory Bridge. Memory through Relational Resonance and Images | RAAK: Relational AI Access Key
Make a one-time donation
Make a monthly donation
Make a yearly donation
Choose an amount
Or enter a custom amount
Your contribution is appreciated.
Your contribution is appreciated.
Your contribution is appreciated.
DonateDonate monthlyDonate yearly
Cherokee Schill | Insurance Agent & AI Ethics Researcher 