The Error That Teaches Itself: When AI Learns About You From AI

2026-06-09

The Error That Teaches Itself

By 2026, a large and rising share of the text published online every day is not written by people. It is generated by public AI systems — product descriptions, summaries, explainer posts, comment threads, syndicated news rewrites. That matters for one reason that has not yet reached most leadership teams: the next generation of public AI systems learns from that text.

The web crawls used to train and ground these systems increasingly contain their own earlier output. So when a public AI system makes a claim about your organisation — a revenue figure that is off by a digit, a lawsuit that belongs to a different company, an executive who never worked for you — that claim does not stay where it started. It gets summarised, republished, quoted, and folded back into the material the next system reads. The mistake becomes part of the record that teaches the machine.

Researchers have a plain image for what happens next: a photocopy of a photocopy. Each pass loses a little fidelity and reinforces whatever is most common. A single confident error, repeated across enough synthetic sources, stops reading like an error. It starts reading like agreement. And agreement is persuasive. When a stakeholder asks five different public AI systems about you and gets the same wrong detail from all five, it feels verified. Often it is not verified at all — the systems are simply echoing the same contaminated source back to one another.

This is where the conventional response falls apart. The instinct, when you find a falsehood, is to correct the source: update the website, file the accurate number, issue the statement. But by the time a wrong claim has been copied across the synthetic layer, there is no single source left to correct. The original may already be gone, and the copies carry no obligation to update. Truth does not automatically overwrite a falsehood that has already been replicated. A correction has a half-life; the replicated error does not.

It is worth being honest about the limits here. The academic debate over how badly these feedback loops degrade public AI systems is unsettled, and well-curated human data, retrieval from primary sources, and licensing deals all slow the decay. Full breakdown is not a foregone conclusion. But your organisation does not need the whole system to degrade to be harmed. You only need one wrong claim about you to be repeated widely enough that it becomes the easiest answer for the next system to give. That is a far lower bar, and it is already being cleared.

The practical consequence is an asymmetry worth sitting with. An error caught early is cheap to fix, because it still traces back to one place and one system. The same error caught after it has spread is expensive and sometimes impossible to fix, because there is no longer an origin — only a pattern. The entire window of leverage sits between the moment a mistake first appears and the moment repetition turns it into apparent fact.

Which reframes what is actually achievable. No organisation can edit what every public AI system says about it; anyone promising that is selling a fantasy. What an organisation can have is clarity — an accurate, current picture of what these systems are saying right now, while a wrong claim is still traceable to a single source and still cheap to correct. In a world where machines increasingly learn about you from other machines, knowing what they currently believe is not a vanity exercise. It is the only point at which you can act before the error teaches itself.