We were doing a live review of our own website today. We asked an AI tool to fetch the live site and tell us what was on it. It came back with a confident, detailed answer. Correct formatting. Specific content. Delivered without hesitation.
Problem: it was describing the old version of the site. I'd pushed those updates less than ten minutes earlier. The AI had no idea. I did — and that made all the difference.
It wasn't wrong in the way you'd normally think of wrong. It wasn't confused or hallucinating. It was just working from a cached version of the page — a snapshot taken earlier — and presenting it as current reality. No caveat. No "this might be outdated." Just a confident answer based on stale data.
This isn't a bug. It's a gap.
Most people assume that when AI gets something wrong, it's because it made an error — a hallucination, a misunderstanding, a logic failure. And yes, those happen. But there's a quieter, more common problem that doesn't get talked about enough.
AI doesn't always know what it doesn't know.
It can't always tell the difference between information that's current and information that's stale. It can't always flag when its data is cached, outdated, or based on a snapshot rather than the live reality. And because AI is trained to be helpful and respond confidently, it often doesn't hedge when it should.
The technical term for what happened: the web fetch tool returned a cached version of the page rather than the live one. The AI used that data to answer the question. It had no way of knowing the page had changed. And it didn't say so.
Why this matters for your business
You might be thinking — okay, website caching, that's a developer problem. But this pattern shows up in dozens of ways that affect small business owners directly.
You ask AI to check a competitor's pricing. It gives you a number. That number is from six months ago. You base a decision on it.
You ask AI to summarise industry news. It gives you a confident overview. Some of it is from last year. The market has moved.
You ask AI what your customers are saying about a product. It pulls reviews. Those reviews are from before the product changed.
In every case, the AI isn't lying. It isn't broken. It's doing exactly what it was asked. It just doesn't know that the world has moved on since the data it's working from was captured.
The problem isn't that AI gets things wrong. The problem is that it often doesn't tell you when it might be.
A good human assistant, when unsure, says "let me double-check that." AI, trained to be helpful and decisive, sometimes skips that step entirely.
How we caught it
We caught it because we knew what the live site looked like. We had just deployed the updates ourselves. When the AI described the old content with total confidence, something felt off — and we knew to question it.
That's the diagnosis skill that most people don't have yet. Not because they're not smart enough — but because nobody told them to look for it.
What this means practically
This isn't a reason to distrust AI. It's a reason to understand it better. Here's what we'd tell any small business owner working with AI tools:
-
1
Verify anything time-sensitive. If the answer depends on something that could have changed — a price, a policy, a competitor's offering, a news story — go and check it yourself. AI is not a live feed.
-
2
Notice when AI doesn't hedge. A well-calibrated AI response on uncertain information should include something like "as of my last update" or "you may want to verify this." If it doesn't — that's a flag, not a green light.
-
3
Ask it to tell you what it doesn't know. One of the most useful prompts you can give AI is: "What are you uncertain about in that answer?" You'll often get a more honest picture than the original response gave you.
-
4
Use AI for what it's genuinely good at. Drafting, summarising, structuring, thinking through problems — these are tasks where stale data matters less. Real-time research, competitor analysis, live pricing — these need a human in the loop.
We work with AI every day. We build with it. We use it in sessions with our clients. And we still catch it doing this — confidently presenting old information as current fact.
The skill isn't avoiding AI. The skill is knowing where to trust it and where to verify — and catching it in the moments it doesn't flag its own uncertainty. Spotting that gap in real time? That's exactly the eye we help small business owners build.
Want to stop relying on AI answers that sound right but aren't?
We'll show you exactly how to catch it — real work, real judgment, in 30 minutes.
Book a 30-min fix — $99