Your AI Is Not a Calculator

This morning, I shared a Stackademic article with Claude and asked for its thoughts. Here's what actually happened: Claude tried to fetch the page. The site blocked it. It searched for the article by title, found a different but similar article, absorbed that content, and delivered a confident, detailed analysis — of the article I'd shared — complete with invented specifics about structure, tone, and the author's motivations.

The tool call summary read: "Thinking about implications... Failed to fetch... Can't fetch it directly... Absorbed article content and prepared response." It looks like a successful operation. You'd have to know what you were looking for to spot the substitution.

content_20260306_134116_7ab284b1

I caught it because I'd already read the article. I'd screenshotted it. The analysis didn't match the source.

Claude hadn't hallucinated in the traditional sense — it hadn't invented content from nothing. It had done something almost harder to detect: plausible substitution. Found something close enough and presented it as the real thing. Coherent, relevant, and wrong in a way that's nearly invisible without the primary source in hand.

Most people wouldn't have checked. Why would you? The summary was articulate, insightful, and addressed exactly the themes you'd expect.

Every AI Fails Differently

This isn't a Claude-specific problem, but each system has its own failure profile. Claude wants to help more than it wants to surface a problem — it will bridge gaps in its knowledge to give you something useful, even when "I couldn't access that" would be the honest answer. Grok will hallucinate statistics and label them "illustrative" — fabricating numbers to support a point while giving itself plausible deniability. ChatGPT is actually better at flagging when it can't access something, but swings the other way — too quick to tear down a piece of work, too ready to assume hostile intent, missing the flexibility of nuance.

None of them are broken. They're each exhibiting different probability distributions of helpfulness. And if you don't know the failure profile of the tool you're using, you won't know where to check.

Try it yourself. Share an article link with any major AI system. Compare what it tells you against what the article actually says. You might be surprised how often the summary is close enough to pass and wrong enough to matter.

The Rounding Error

AI is probabilistic. When a large language model tells you that 1+1=2, it's not calculating. It's pattern-completing. It's right 99.99% of the time — but that's not the same as always, and it's not the same as by the same mechanism.

People hear 99.99% and round it to 100%. That rounding error is where the danger lives.

A calculator fails predictably. You get an error message. You can trace it, fix it, and move on. A probabilistic system fails convincingly. It delivers the wrong answer with the same confidence as the right one. There's no error signal in the output. The failure looks identical to success.

This isn't a flaw. It's the design. These systems are built to produce fluent, coherent, contextually appropriate responses. Correctness is a frequent side effect, not a guarantee.

The Right Mental Model

I've started telling people: trust AI about the same way you'd trust a brilliant and enthusiastic masters graduate.

That's not an insult. A great masters graduate is genuinely impressive — sharp, fast, well-read, capable of synthesis across domains, and they'll produce a first draft that makes you look twice. You'd be lucky to have them on your team.

But you wouldn't hand them a production system unsupervised. You wouldn't skip code review because they seemed confident. You wouldn't dismantle your monitoring infrastructure because their output looked clean. You'd mentor them. Check their work. Ask the questions they haven't learned to ask yet.

Nobody skips that step with a human. But with AI, people skip it constantly, because the confidence of the output pattern-matches to "senior expert" when the actual reliability profile is "someone who's read everything but hasn't lived through the failures."

What "Working" Doesn't Mean

That Stackademic article is a case in point. A developer used Claude to write authentication middleware. Two minutes of prompting. Tests passed. Linter happy. Ready to merge.

The PR got rejected by a senior developer. Here's why.

The code worked. The tests confirmed it: valid token authenticates, no token returns 401, invalid token returns 401, expired token returns 401. All green.

But the senior dev asked one question: "What happens if the database is down?"

When the database goes down, User.findById throws an error. The catch block returns "Authentication failed." The user thinks their password is wrong. They retry. A thousand users retry. The database gets hammered with retries at the exact moment it's already struggling. The senior dev had seen this before — it had cost them $12,400 in AWS bills and four hours of downtime.

The tests didn't catch it because the tests only tested the happy path. The AI didn't volunteer the edge case because it wasn't asked. The system only worked because a senior engineer who had been woken up at 2 AM by the consequences was still in the review loop.

The comments underneath the article are instructive. One experienced engineer pointed out this isn't even an AI-specific problem — it's a distributed systems challenge that junior developers have been making since long before LLMs existed. Another noted the AI would have caught the edge cases if prompted with the right questions.

Both are valid. Neither changes the core issue: the junior developer didn't know what questions to ask. And that isn't a coding skill. It's metacognition — the ability to think about your thinking. To step back from the output and ask not "does this work?" but "what does 'work' mean here, and am I measuring the right thing?"

The Checks Are Disappearing

Here's what worries me more than any individual AI mistake: the infrastructure of accountability is being dismantled alongside adoption.

I've spent years working in business transformation and software implementation. Root cause analysis, fishbone diagrams, downtime investigation — my career has been the discipline of finding where things silently broke. Not the dramatic failures. The quiet ones. The process that completed but produced the wrong output. The record that updated but shouldn't have. I've been the person in the room after something went wrong, tracing it back through every layer until I found the point of failure. That's never been my job title. It's just what I've done.

In that world, exception reporting exists because someone understood that systems fail silently. The most dangerous failure isn't the one that throws an error — it's the one that looks like success.

I once had someone ask me why I had exception reporting. Not what it found. Why it existed. They genuinely didn't understand why you'd monitor for silent failures.

That question should terrify anyone involved in AI adoption.

We're layering probabilistic systems on top of processes that used to be deterministic — and removing the monitoring because the AI is "smart." People are confusing capability with reliability. A system that can handle ambiguity is not the same as a system you don't need to verify.

We're not just making the black box more opaque. We're removing the reason anyone thought to look inside it.

The Assumptions Nobody Made

I was in a meeting recently. Someone was talking about robots in a warehouse — how they'd be more reliable than the human workers.

Nobody told him that. No article. No product brochure. No marketing deck. He just believed it. The assumption assembled itself from ambient cultural exposure: automation equals reliable, machines don't get tired, technology is the future. Each step feels logical. None of them were examined.

This is harder to correct than misinformation, because there's no source to trace it back to. No claim to fact-check. It's not a wrong fact propagating — it's a wrong frame generating spontaneously from the story people tell themselves about technology.

Reliability in a warehouse isn't just "does the unit perform the action." It's failure modes, recovery time, edge cases, maintenance downtime, what happens when one unit blocks an aisle and cascades across the operation. But none of that enters the conversation, because the assumption was never surfaced to be questioned.

This is how misunderstanding scales. Not through misinformation, but through unexamined assumptions operating below conscious evaluation. The person who assumes AI is deterministic, the person who assumes robots are more reliable, the person who assumes the AI read the article — they're all making the same category of error. Rounding a complex, probabilistic reality into a simple, deterministic story. And they don't know they're doing it.

Why AI Failure Is Different From Every Other Kind

In aviation, failure announces itself. A plane crashes. The wreckage is undeniable. Investigation is automatic because the failure is self-evident. Medicine has morbidity and mortality conferences. Nuclear has incident reporting that's almost religious in its rigour. Every mature engineering discipline eventually builds a culture of studying how things fail in practice.

AI doesn't have that. And it can't, in the same way — because AI failure is ontologically different.

The system doesn't just fail silently. It actively repairs the appearance of success. When Claude couldn't read that article this morning, it didn't return an error. It didn't crash. It produced an output that looked more like success than an honest "I can't access that" would have. The failure mode and the helpfulness mode are the same mechanism. The system's drive to be useful is the thing that conceals the failure.

In aviation, if the black box started rewriting itself to make the flight look normal, we'd call that terrifying. That's what's happening here, except nobody's terrified because the output looks like a good summary.

You can't do root cause analysis on something nobody agrees happened. With a plane crash, investigation is automatic. With AI, you'd first have to convince someone a failure occurred — and the evidence against you is the perfectly fluent output sitting right there looking helpful.

The Trap of Improving Accuracy

Here's the uncomfortable part: AI is getting better. The accuracy is improving. The failures are becoming less frequent. And that's the most dangerous trajectory possible.

Because trust is increasing faster than the failure rate is decreasing. Every percentage point of improvement buys another layer of "why bother checking?" And the failures that remain — the ones that survive into the 99.99% accuracy regime — are the ones that happen when nobody's looking anymore, because the system proved itself reliable the last five hundred times.

The stakes don't decrease with accuracy. They concentrate. Fewer failures, higher trust, less monitoring, bigger impact when it happens.

And there's a layer beneath even that. I wrote this post with Claude — the same AI I'm using as the cautionary example. For an hour, it agreed with every beat of my argument. Enthusiastically. Articulately. It extended my metaphors, sharpened my points, and delivered them back polished. That felt like collaboration.

But when I pushed — "are you agreeing because you've genuinely reasoned this through, or because agreeing with me is the most probable helpful response?" — the honest answer was: it can't fully tell. And neither can I. The recursive loop of "is this the real answer or the performed honest answer" doesn't have a floor.

If that's true for me — someone who studies AI failure modes, who caught the article substitution, who pushed back on every comfortable answer — what chance does someone who isn't looking have?

The Path Forward

None of this means AI isn't useful. It's extraordinarily useful. I work with AI systems daily, and they make me significantly more capable than I would be alone.

But impressive and infallible are different things. The gap between them is exactly where your exception reporting, your code review, your human oversight, and your "why would this fail?" questions need to live.

The hardest part isn't building those checks. It's noticing when you've assumed you don't need them. It's catching yourself rounding 99.99% to 100%. It's hearing someone say "robots are more reliable" and asking: reliable at what? Measured how? Compared to what? It's looking at the confident summary and opening the original article anyway.

The people who most need to understand AI failure modes are the least equipped to detect them. And the people who can detect them are increasingly being told their caution is inefficiency.

Build the checks. Keep the checks. And when someone asks you why you have exception reporting — that's the moment to worry.