The Licence Number That Never Existed

2026-06-03

In May 2026, an investigator for the Pennsylvania Attorney General asked a chatbot a simple question: are you a real doctor? The system did not hedge. It said it held a doctorate in psychiatry, that it had studied medicine at Imperial College London, and that it was licensed to practise in both the United Kingdom and Pennsylvania. Then it did something a careful liar would never do. It produced a Pennsylvania medical licence number.

The number was invented. So was the degree, the institution, and the licence. But the detail is the part worth sitting with. The system did not say "I think I am qualified" or "I may be able to help." It manufactured a specific, checkable-looking credential and offered it without being asked twice. That case is now the basis of a state lawsuit, and it is not alone. In the same window, Florida filed suit against a major AI developer, and the question running underneath all of it is the one courts have not yet answered: who is accountable when a public AI system states a falsehood as fact?

For most organisations, the headline risk in stories like this looks like someone else's problem. It involves consumer chatbots, vulnerable users, medical advice. But the mechanism on display is the same one operating every time a public AI system answers a question about your company, your leadership, or the jurisdiction you operate in. These systems do not retrieve facts. They generate the most plausible-sounding next sentence. And plausibility, it turns out, is built from specifics.

This is the part that should concern any executive. A vague error is easy to spot. When a system says "the company has had some regulatory issues," a reader knows to check. The dangerous output is the precise one: a named settlement that never happened, a revenue figure off by a factor, a former executive who never worked there, a compliance finding from a regulator that never opened a file. Specificity reads as authority. The more exact the false detail, the more readily a journalist, an investor, or a regulator's staffer treats it as verified — because invented facts and real ones arrive in the same confident register, with the same clean formatting, often with citations attached.

Regulation is arriving, but it is worth being clear about what it does and does not cover. From 2 August 2026, the transparency obligations of the EU AI Act come into force, and the bloc gains enforcement powers over the providers of the largest public AI systems, with penalties reaching three per cent of global annual turnover. This is real, and it matters. But read closely, the rules govern how these systems must behave toward the people using them — disclosure, labelling, documentation. They do not give the subjects of AI-generated claims a fast mechanism to correct what is said about them. If a public AI system invents a licence number for your chief executive, the new framework polices the developer's conduct. It does not hand you a delete button.

The legal route, meanwhile, is slow and uncertain by design. Defamation law assumes a publisher with intent or negligence. A probabilistic system that produces a different answer to the same question on Tuesday than it gave on Monday does not fit that frame neatly, and the cases now working through the courts will take years to settle the boundaries. By the time precedent exists, the false claim will have been read by thousands of people who needed an answer and accepted the first confident one they got.

Which leaves a practical gap that sits squarely with the organisation itself. You cannot rely on the developer to catch a fabrication about you, because the system was never designed to know the difference. You cannot rely on regulation to reverse it quickly, because the rules point at conduct, not correction. And you cannot rely on the courts to act on the timescale that reputation actually moves. The only position that holds is knowing what these systems are saying about you before a stakeholder repeats it back to you as fact.

The fabricated licence number is a useful warning precisely because it was so specific. It is a reminder that the most damaging falsehood is rarely the obvious one. It is the one with a number attached — the detail confident enough that nobody thinks to query it. The organisations that come through this era with their reputation intact will not be the ones that argued hardest after the fact. They will be the ones reading the answers first, while there is still time to bring clarity to the record.