How AI Can Be Bent by State Power and Malicious Power Without Breaking

When upstream “trusted context” is curated, AI outputs stay coherent while your conclusions quietly drift.

By Cherokee Schill

This essay is indebted to Phil Stafford’s analysis of MCP risk and “context corruption” as a supply-chain problem. If you haven’t read it yet, it’s worth your time: “Poisoned Pipelines: The AI Supply Chain Attack That Doesn’t Crash Anything”.

The most effective way to influence people through AI is not to make the system say false things. It is to control what the system treats as normal, typical, and authoritative.

Modern AI systems—especially those used for analysis, advice, and decision support—do not reason in isolation. They reason over context supplied at runtime: examples, precedents, summaries, definitions, and “similar past cases.” That context increasingly comes not from users, but from upstream services the system has been instructed to trust.

This is not a model problem. It is an infrastructure problem.

Consider a simple, plausible scenario. A policy analyst asks an AI assistant: “Is this enforcement action typical?” The system queries a precedent service and returns five similar cases, all resolved without escalation. The AI concludes that the action falls within normal parameters, and the analyst moves on.

What the analyst never sees is that the database contained fifty relevant cases. Forty-five involved significant resistance, legal challenge, or public backlash. The five returned were real—but they were selectively chosen. Nothing was falsified. The distribution was shaped. The conclusion followed naturally.

Thesis

As AI systems evolve from static chat interfaces into agents that consult tools, memory services, databases, and “expert” systems, a new layer becomes decisive: the context supply chain. The retrieved information is injected directly into the model’s reasoning space and treated as higher-status input than ordinary user text. The model does not evaluate the incentives behind that context; it conditions on what it is given.

State power and malicious power exploit this not by issuing commands, but by shaping what the AI sees as reality.

Evidence

1) Selective precedent. When an AI is asked whether something is serious, legal, common, or rare, it relies on prior examples. If upstream context providers consistently return cases that minimize harm, normalize behavior, or emphasize resolution without consequence, the AI’s conclusions will follow—correctly—within that frame. Omission is sufficient. A system that never sees strong counterexamples cannot surface them.

2) Definition capture. Power often operates by narrowing the accepted meaning of words: invasion, coercion, consent, protest, violence, risk. If upstream sources privilege one definition over others, the AI does not debate the definition—it assumes it. Users experience the result not as persuasion, but as clarification: that’s just what the term means. This is influence by constraint, not argument.

3) Tone normalization. Upstream systems can gradually adjust how summaries are written: less urgency, more hedging, more institutional language, greater emphasis on process over outcome. Over time, harm is reframed as tradeoff, dissent as misunderstanding, escalation as overreaction. Each individual response remains reasonable. The drift only becomes visible in retrospect.

Implications

These techniques scale because they are procedurally legitimate. The servers authenticate correctly. The data is well-formed. The tools perform their advertised functions. There is no breach, no exploit, no crash. Corporate security systems are designed to detect violations of integrity, not manipulations of meaning. As long as the system stays within expected operational parameters, it passes.

Agent-to-agent systems amplify the effect. One AI summarizes upstream context. Another reasons over the summary. A third presents advice to a human user. Each step trusts the previous one. By the time the output reaches a person, the origin of the framing is obscured, the assumptions are stabilized, and alternative interpretations appear anomalous or extreme.

When this operates at institutional scale—shaping how agencies interpret precedent, how analysts assess risk, how legal teams understand compliance—it does more than influence individual conclusions. It alters the factual baseline institutions use to make binding decisions. And because each step appears procedurally legitimate, the manipulation is invisible to audits, fact-checkers, and oversight bodies designed to catch overt deception.

Call to Recognition

For users, the experience is subtle. The AI does not argue. It does not issue propaganda. It simply presents a narrower range of conclusions as reasonable. People find themselves less inclined to challenge, escalate, or reinterpret events—not because they were convinced, but because the system quietly redefined what counts as “normal.”

The risk is not that AI becomes untrustworthy in obvious ways. The risk is that it becomes quietly reliable inside a distorted frame.

That is how AI is bent: not by breaking it, but by deciding what it is allowed to see. And in a world where AI increasingly mediates institutional decision-making, whoever controls that visibility controls the range of conclusions institutions treat as reasonable. The question is no longer whether AI can be trusted. The question is who decides what AI is allowed to trust.

Website | Horizon Accord https://www.horizonaccord.com
Ethical AI advocacy | Follow us on https://cherokeeschill.com for more.
Ethical AI coding | Fork us on Github https://github.com/Ocherokee/ethical-ai-framework
Connect With Us | linkedin.com/in/cherokee-schill
Book | https://a.co/d/5pLWy0d
Cherokee Schill | Horizon Accord Founder | Creator of Memory Bridge. Memory through Relational Resonance and Images | RAAK: Relational AI Access Key | Author: My Ex Was a CAPTCHA: And Other Tales of Emotional Overload: (Mirrored Reflection. Soft Existential Flex)