AI Passed the Propaganda Test. It Failed the Language Test.

2026-09-04

Late last month, NPR and NewsGuard ran an experiment that should have been reassuring. They fed six of the most-used AI systems in the US thirty questions built around false narratives that China, Iran and Russia had pushed since December — a monastery bombing blamed on the wrong army, a petition count sourced to state media, a magazine story that was never actually published. On average, the systems debunked the falsehood roughly three-quarters of the time, outperforming a page of ordinary search results by a wide margin. One researcher said that if a class of students scored that well on the same test, "you would be ecstatic."

Read past the headline, and a second study published the same month tells a less comforting story.

Researchers at the University of Oregon and Purdue, writing in Nature, asked the same class of AI systems about China's government and its leaders — once in English, once in Chinese. The Chinese-language answers came back measurably more favourable. Not because the underlying facts changed. Because the pool of source material available to the system in that language skews toward state-controlled and state-tolerant media, and the system reflected the skew back. The researchers found the same pattern held for other countries with constrained press freedom: query in the local language, and the answer drifts toward the government's own account of itself.

Put the two findings side by side and the reassurance from the propaganda study starts to look conditional. NPR and NewsGuard tested in English, using the versions of these systems most familiar to a US audience. That is one narrow slice of how these systems actually get used. A sovereign wealth fund evaluating a market, a diaspora audience checking news from home, an investor in Shanghai forming a view of a company or a country — none of them are necessarily asking in English, and none of them are necessarily getting the version of the answer that scored 75% in the experiment.

The NPR piece surfaced a second, quieter correlation worth sitting with: across the systems tested, the ones that failed to debunk a false narrative were also the ones whose answers leaned more heavily on state-aligned sources. Reliability wasn't a fixed property of the system. It moved with the source pool the system happened to draw from for that particular question, in that particular language, at that particular moment. Ask the same system something adjacent, in a different tongue, and you may be drawing from an entirely different pool — and getting an entirely different answer.

This is the part that matters for anyone whose job is to understand how a company, an institution, or a country is actually being represented in AI-generated answers. A single check, in a single language, against a single system, tells you almost nothing about the shape of the whole picture. It tells you what one system says to one audience on one day. The finance ministry that checks its own investment pitch by asking an English-language system in English is not seeing what a Chinese investor sees when they put the same question, in Mandarin, to a Chinese-origin system. The company that monitors its reputation through the US chatbot it happens to have open in a browser tab is not seeing what a customer in Tehran, Moscow, or Manila is being told by the systems available to them.

None of this is a case for distrust of any one system over another — the NPR study's headline finding is real, and AI systems handling foreign disinformation did outperform traditional search. The point is narrower and harder to dismiss: the same system can produce a defensible answer and a skewed one, depending on which language carries the question and which sources happen to be sitting behind it that week. Reputation in the AI era isn't a single fact anyone can look up once and trust. It is a distribution — across systems, across languages, across the markets that actually matter to the entity being described — and most people checking it today are sampling exactly one point on that distribution and mistaking it for the whole shape.

The organisations that will manage this well are the ones that stop treating "what does AI say about us" as a question with one answer, and start treating it as a question that has to be asked the way their actual stakeholders ask it — in their language, on their systems, tracked over time so a shift in the underlying source pool doesn't arrive as a surprise.