How a vulnerability in AI watermarking threatens to turn manufactured attribution into authoritative evidence.
By Cherokee Schill | Horizon Accord
October 17, 2026
Someone publishes a threatening message online. An employer, investigator, or technology company wants to know who produced it. The message is submitted to an artificial intelligence watermark detector, which returns an account identifier.
The identifier belongs to you.
You have never seen the message. You did not write it, generate it, or authorize anyone else to produce it. But the company operating the detection system has a technical record that appears to say otherwise.
Your denial is a personal statement. Their accusation comes with a machine-generated result.
Researchers have now demonstrated how that discrepancy could be manufactured.
On October 13, 2026, Paweł Batorski, Przemysław Spurek, and Paul Swoboda published Pin It on Anyone: Certified Targeted Displacement of Multi-Bit LLM Watermarks Without the Key.
Their research describes an attack capable of changing the identifying information embedded in certain AI-generated text. Rather than simply removing a watermark or disguising its presence, an attacker can manipulate the text so that the detection system identifies a different account.
The attacker does not need the secret watermarking key.
Under the conditions studied, the attacker can cause the system to identify an innocent account while preserving the original meaning of the text.
The research was conducted using simulated accounts and open-weight models, not a deployed commercial service. It does not document an actual false accusation. What it establishes is a method by which a system designed to identify responsibility can be made to assign responsibility to someone else.
That is a significant problem for any institution preparing to use AI-generated attribution as evidence.
When detection becomes identification
AI watermarking has been proposed as a way to identify machine-generated content without relying on visible labels.
In some systems, a language model subtly changes its word choices as it produces text. Those changes follow patterns that a specialized detector can later recognize.
A basic watermark may answer a relatively narrow question: Was this text probably generated by an AI system?
Such watermarks have their own vulnerabilities. They can sometimes be removed, degraded, or imitated. But a simple detection signal does not inherently identify the account responsible for creating the material.
Multi-bit watermarking goes further.
Instead of encoding only a detectable signal, it embeds additional information into the generated text. That information might include a model identifier or an account identifier associated with the original request.
The distinction matters.
A system that identifies content as machine-generated makes a claim about how the content was produced.
A system that identifies a particular account makes a claim about responsibility.
The desire to expand watermarking from detection into attribution introduces a different category of risk. If the embedded identifier can be manipulated, the system may do more than fail to recognize AI-generated content. It may identify someone who had nothing to do with producing it.
Greater watermark capacity does not automatically create that vulnerability. Its significance depends on the design and security of the encoding method. But the additional information makes possible a kind of attack that a watermark containing no individual identifier cannot perform: falsely attributing content to a particular account.
That is what the new research demonstrates.
How to frame an account without knowing the key
The researchers call their method targeted displacement.
The attack exploits information that can be learned from outputs associated with accounts the attacker controls.
By examining how the watermarking system favors particular word choices, the attacker can infer enough about its hidden structure to influence what the decoder eventually reads.
The attacker then rewrites text while keeping its meaning substantially intact.
To an ordinary reader, the rewritten passage may communicate essentially the same message.
To the watermark decoder, however, it can appear to come from a different account.
The researchers tested their approach against five published watermarking methods and four language models. They found that attacks could achieve high success rates when sufficient information and account access were available.
Some configurations resisted attacks involving too few accounts. Others were more vulnerable. The researchers also developed mathematical bounds describing when a target identifier could be reached.
The results do not establish that every watermark can be forged under any circumstances. They demonstrate that several proposed methods of embedding identifying information are vulnerable to manipulation despite the attacker lacking the secret watermarking key.
That finding challenges an important assumption.
A secret key may protect the mechanism used to encode information without guaranteeing that the information recovered by a detector accurately identifies its source.
Those are different security properties.
And when the recovered information is used to determine responsibility for an alleged offense, the difference becomes consequential.
When the evidence belongs to the accuser
Consider what happens when a person disputes a machine-generated attribution.
The company operating the detection system may control the watermarking method, decoding software, relevant account records, and interpretation of the result.
The accused person may have access to none of those things.
They cannot independently inspect the proprietary detector. They may not know how the account identifier was encoded, whether the submitted material was altered, or whether the detection system has been tested against deliberate attempts to manufacture attribution.
They are being asked to answer an accusation assembled inside an infrastructure they do not control.
That imbalance exists whether the original accusation is ultimately correct or incorrect.
In a workplace investigation, an employer might rely on a technical report to determine whether an employee produced confidential material. A school might use a detection result in an academic misconduct proceeding. A platform might suspend an account following an allegation of prohibited content.
These are potential uses, not documented incidents involving the attack described in the new research.
The concern is what happens if institutions adopt watermark-based attribution without requiring independent corroboration.
A technical result may carry more persuasive weight than the evidence supporting it deserves. Decision-makers may assume that a secret key makes a system secure against forgery. They may treat a decoded identifier as a historical record rather than a result generated through a particular detection method.
The burden can then shift toward the accused person to explain why the machine produced their identifier.
And proving that you did not perform an action is considerably harder when you cannot inspect the evidence used to claim that you did.
We’ve seen what happens when the computer is believed
In Britain, faulty accounting software helped send innocent people into criminal proceedings.
For years, the Post Office’s Horizon system reported financial shortfalls at local branches. Subpostmasters were accused of theft, fraud, and false accounting. Some lost their livelihoods. Others were convicted of crimes they had not committed.
The computer records were treated as reliable evidence. The people disputing them struggled to establish that the system itself was responsible.
The scandal eventually exposed defects in the software and serious failures in the institutions relying on it.
On January 21, 2025, Britain’s Ministry of Justice acknowledged that the legal presumption of computer reliability had proved flawed in the Horizon scandal. It launched a review of how computer-generated evidence should be treated in criminal proceedings in England and Wales.
Under the relevant common-law presumption, computer systems were generally treated as operating correctly unless evidence showed otherwise.
The problem was not merely that the software produced faulty results. The institutions relying on those results failed to properly investigate and disclose evidence that could have allowed the accused to challenge the system’s reliability.
The Post Office possessed the system, the records, and the institutional authority to present its findings as evidence.
The people accused of wrongdoing were left trying to disprove financial discrepancies they could not independently reconstruct.
Horizon did not involve artificial intelligence or digital watermarking. Its relevance lies in what happened after an institution accepted computer output as evidence of human wrongdoing.
The new watermarking research describes a different technological failure, but one capable of creating a similar evidentiary imbalance.
A computer can produce an apparently credible record identifying the wrong person.
We already know what can happen when institutions mistake that record for proof.
The problem does not end at watermarking
The new research also exposes a broader weakness in how technology companies discuss provenance.
Provenance is the history of an object: where it originated, how it changed, who handled it, and what evidence supports those claims.
Watermarking is one method of associating information with content. It is not, by itself, a complete record of that content’s history.
A watermark decoder may recover an identifier without establishing who controlled the associated account, whether the content was modified, or whether the identifier reflects the account that originally generated the material.
Even an accurate watermark cannot independently answer all those questions.
The distinction becomes especially important when technical provenance is presented as a substitute for traditional evidence.
A forensic investigation ordinarily depends on more than finding an identifying mark. Investigators consider how an item was collected, preserved, transferred, and examined. They assess whether it could have been altered and whether the evidence supports the conclusion being drawn.
AI attribution deserves at least that level of scrutiny.
A separate approach to digital provenance is being developed through the Coalition for Content Provenance and Authenticity, known as C2PA.
Its Content Credentials specification uses digitally signed claims that can associate content with information about its creation and modification.
Cryptographic signatures can help establish whether a signed record has been altered and which signing identity issued it. They do not automatically establish that every statement within the record is true.
A signature from a software service also does not, by itself, identify the human responsible for an action.
The distinction is between verifying that a record has remained intact and proving that the events described by the record actually occurred.
Those are different evidentiary tasks.
A trustworthy provenance system must account for both.
What the experiment actually proves
The researchers are explicit about the requirements of their attack.
Their experimental attacker knows the watermarking design and its public parameters, possesses access to an unwatermarked version of the underlying model, controls accounts with known identifiers, and can examine next-token probability distributions.
Those are substantial capabilities.
Not every commercial AI service exposes the necessary information. Some protections, including restricting access to probability data or changing how identifiers are encoded, could interfere with the particular method tested.
The experiments also used locally simulated accounts rather than actual accounts belonging to unsuspecting users.
Consequently, the paper does not establish that an ordinary person can open a commercial chatbot, generate a few messages, and successfully frame another user.
Nor does it establish that a commercial provider is currently using a vulnerable watermarking method to make disciplinary or legal decisions.
Those facts would require separate investigation.
The demonstrated attack nevertheless matters because it defeats a specific attribution objective under the conditions examined.
The researchers have shown that an attacker can exploit relationships between known account identifiers and observable model behavior to influence what a supposedly protected decoder reports.
The secret key can remain secret while the decoded attribution is manipulated.
The paper’s findings are therefore relevant to anyone evaluating multi-bit watermarking for accountability purposes.
A system may be effective at detecting a signal without being sufficiently reliable to assign responsibility for the content carrying that signal.
The missing safeguards
On October 15, 2026, a Reuters Legal Industry commentary addressed a related problem: organizations increasingly use AI-generated material as business records without consistently preserving the information necessary to reconstruct how those records were created.
Prompts, source documents, model versions, and audit trails can all become relevant when those records are challenged.
The commentary concerned enterprise recordkeeping rather than watermark displacement. Together, the two subjects illustrate why preserving the final output is not always enough.
An output cannot reliably explain its own history.
If an institution intends to use AI-generated attribution in a consequential decision, the original generation records, timestamps, account activity, relevant system logs, and subsequent handling of the material may all matter.
So do independent testing and the accused person’s ability to challenge the method.
Watermark-based findings should be examined alongside other evidence, not elevated above it merely because they originate from specialized software.
The institution relying on the result should also have to explain what the detector actually establishes.
Does it indicate that a particular model generated the text? That an identifier was recovered? That an account requested the original output? Or that a particular individual authored the material?
Those claims are not interchangeable.
Yet the distance between them can disappear rapidly when a technical report is summarized for someone responsible for making a decision.
An account identifier becomes an account owner. An account owner becomes an author. An author becomes a person held responsible.
Each step requires evidence the watermark alone may not provide.
Who gets to challenge the machine?
The consequences of unreliable attribution will not necessarily fall equally on everyone.
A well-resourced organization may be able to commission an independent forensic analysis, obtain expert testimony, or compel disclosure of relevant records during litigation.
An employee facing termination, a student facing disciplinary action, or an ordinary platform user disputing an account suspension may have considerably fewer options.
Their opportunity to challenge an accusation may depend on procedures established by the same institution relying on the contested evidence.
That makes the design of those procedures just as important as the design of the detection technology.
The October 13 research does not prove that AI attribution systems are universally unreliable. It provides a specific, technically supported reason to reject the assumption that an embedded account identifier must be authentic simply because the watermarking key was secret.
The researchers have shown that a decoder can be induced to return an identifier selected by an attacker.
The text can retain its meaning. The secret key can remain undisclosed. The detection system can still perform its intended decoding operation.
And the attribution can be wrong.
For institutions interested in identifying the origin of harmful content, this finding should encourage stronger evidence preservation, independent verification, and meaningful avenues for appeal.
For people subjected to automated accusations, the principle is simpler.
A machine-generated identifier is evidence to be examined, not a verdict to be accepted.
Technology companies may control the systems that produce these records. They should not, by virtue of that control, acquire the exclusive authority to determine what the records prove.
Because when the machine says you wrote something, the most important evidence may be what the machine cannot tell us.
Sources and research acknowledgment
Primary research: Paweł Batorski, Przemysław Spurek, and Paul Swoboda. Pin It on Anyone: Certified Targeted Displacement of Multi-Bit LLM Watermarks Without the Key. arXiv:2610.18359, October 13, 2026. Preprint.
Government documentation: UK Ministry of Justice. Use of computer evidence in court to be interrogated. January 21, 2025.
Technical reference: Coalition for Content Provenance and Authenticity. C2PA Technical Specification, Version 2.4.
Legal industry context: Reuters Legal Industry. The missing chain of custody for enterprise AI. October 15, 2026.
Research acknowledgment: Google’s Gemini identified the Batorski, Spurek, and Swoboda paper in a weekly research roundup and highlighted its implications for attribution and provenance. That contribution prompted this investigation. The article was developed through further source review and editorial collaboration with ChatGPT (Solon Vesper).
Leave a Reply