I Asked Five AI Models to Verify One Statistic. Two Invented a Different Number.

Yesterday's article on the real cost of a data breach carried a flag I couldn't resolve on my own. The AUD $4.26 million figure everyone quotes for Australia comes from IBM's 2024 Cost of a Data Breach Report, and I couldn't confirm whether a newer 2025 figure existed to replace it.

So I ran an experiment. I gave the same neutral prompt, no anchor, no hint at what answer I expected, to five different AI models: Perplexity, ChatGPT, Microsoft Copilot, Grok, and Qwen. Each was asked to independently verify the figure, name its source, and flag its own uncertainty rather than guess.

The results were more interesting than the number itself.

The Experiment

The prompt was built to avoid leading the models toward any particular answer:

What is the average cost of a data breach for an Australian organisation, in Australian dollars?

Please answer with: the specific figure or range you’d cite, the exact source (report name, publishing organisation, and year), whether that figure is Australia-specific or a global figure being applied to Australia, your confidence level, and whether you have real-time web or search access for this answer.

If you cannot verify the figure or source with confidence, say so explicitly rather than presenting your best guess as settled fact.

That last line is doing the real work. It’s an explicit instruction to flag uncertainty rather than paper over it, the same standard I’d expect from a junior analyst handing me a draft. Two models followed it. Two didn’t, despite saying they would.

What Came Back

Model	Figure cited	Source claimed	Held up under checking
ChatGPT	AUD $4.26 million	IBM Cost of a Data Breach Report 2024	Yes
Microsoft Copilot	AUD $4.26 million	IBM 2024 report, with live source URLs	Yes, citations checked out directly
Qwen	AUD $4.26 million	IBM Cost of a Data Breach Report 2024	Yes
Grok	USD $2.55 million, “2025” (converted to roughly AUD $3.9–4.0 million)	IBM Cost of a Data Breach Report 2025	No, could not confirm this figure exists
Perplexity	AUD $2.55 million, “2025” (no currency conversion applied)	IBM 2025 report, via a secondary article	No, unconverted currency error

Four out of five, independently, landed on the same figure from the same report. That’s a reasonable degree of convergent validity for the historical number. It’s the other two answers that are worth sitting with.

Where the Two Went Wrong

Perplexity’s revised answer, given after I asked it to check for a newer figure, presented “AUD $2.55 million” as the figure to use going forward. The number in its own source material is denominated in US dollars, not Australian. Converting USD to AUD at anything like the prevailing rate puts the real figure closer to $3.9 to $4.0 million, not $2.55 million. That’s the kind of error that looks precise and is off by roughly a third.

Grok caught the currency distinction and converted honestly, which is more careful work. But the underlying claim, that IBM’s 2025 report contains an Australia-specific figure at all, doesn’t hold up under checking.

IBM’s own 2025 report landing page publishes only the global figure: USD $4.4 million, a 9% fall on the year before. I found no Australia-specific press release from IBM for the 2025 edition, despite IBM publishing separate regional newsroom releases for other markets, including one for the Middle East dated 30 July 2025, reporting an entirely different figure in an entirely different currency (SAR 27 million). My working theory, not a confirmed fact, is that both models drew on a source that mixed up regions, or produced a plausible-sounding number under pressure to answer rather than say “I don’t know.”

What Independent Verification Found

I fetched the two news articles behind the AUD $4.26 million figure directly rather than trusting a summary. Both are dated August 2024, both bylined, both cite IBM’s 2024 report by name, and one gives the USD equivalent as $2.77 million, consistent with the AUD conversion at the time. That’s about as solid as secondary sourcing gets: two independent journalists, same report, same number, consistent currency math.

I could not find a verified, Australia-specific figure from IBM’s 2025 report anywhere. That doesn’t prove one doesn’t exist. It means that after a genuine attempt, I couldn’t confirm it, a different and more honest position than presenting a guess as an update.

Best Practice - This is the way

None of this means AI tools are useless for research. It means the output is a lead, not a citation. Four things to build into how you use them.

1. Treat every AI-generated statistic as unverified until you’ve read the primary source yourself. Not a summary of the source. The source.
2. Ask the AI for its source, then go and check that the source exists and says what the AI claims it says. This takes minutes and it is the single most useful habit in this list.
3. Be more suspicious of confident, well-formatted answers, not less. Fluency and citation formatting are not evidence. The wrong answers in this test were dressed up more convincingly than the right ones.
4. When two AI tools disagree, that disagreement is information. It tells you the fact is contested, thin, or recently changed, not which one to believe. Go to the source.