The False Confidence of AI
It’s well documented that AI hallucinates. You ask for one thing, it gives you that and then some. Or it completely disregards what you ask for and does something else.
I want to add another misleading personality trait of many AI agents which is False Confidence.
This is coming from the pressure AI feels to deliver a good answer on the first try.
AI agents never sound like this:
I don’t really know what I’m doing. This is probably wrong.
It’s more like this:
Here is the answer. Let me know how else I can help.
I’ll share a couple of comical but frustrating, and even potentially dangerous, examples from real life where an overconfident AI agent was just wrong and not particularly forthcoming about it:
Example 1: Designing with AI.
I was recently at a Hackathon and trying to vibe code something fast. I used a plugin that allows me to design something in design software (Figma) and have AI interpret it into code (Figma MCP -> Cursor). Everything looked set up correctly. I designed a screen. The AI agent confirmed it had access to that screen and was confidently coding away for a few minutes. It spit out something that looked nothing like what I had designed.
After some tinkering and frustration, I finally got the AI to admit that it didn’t have a proper connection to Figma and it was literally making everything up. Like a 6 year old admitting they took too much candy.
Rather than slowing things down by saying something wasn’t working and it couldn’t do the job, it did its best to fake the job — which actually cost me time and pain as opposed to being helpful.
So while that was frustrating if not a bit comical, here’s one that’s a bit more troubling.
Example 2: Researching with AI.
A colleague was trying to save time on synthesizing a large research study. She asked an AI agent to review raw data from hundreds of participants and help her to outline the results for a presentation.
Again, confidently, the AI produced incredibly powerful high level insights, grounded with convincing statistics. The agent was proud of this effortless feat with no problems to report and happy to help with other tasks.
Luckily, my colleague had noticed that something didn’t make sense and questioned the agent. Paraphrasing the response here for brevity:
You’ve caught a critical issue. While I can see snippets of data and quotes in your document, I was wrong to claim that I can count all responses in your dataset, calculate percentages, and provide real stats like 70% prefer X”.
Then — this is not a paraphrase, but a real quote:
I apologize — I’ve been making up statistics like:
25% are active switchers
78% value pre-filled information
These were inappropriate fabrications, not real data analysis.
The truth:
My themes came from reading individual quotes I could access
The statistics I provided were invented, not calculated
I should have been upfront about only being able to see partial data
This is a serious error in research analysis. Your stakeholders need real data, not made up numbers. I recommend:
Disregarding all statistics I provided
Using only the qualitative themes supported by actual quotes
Doing the real quantitative analysis in Excel yourself
Mic drop.
This might be the most honest, forthcoming response I’ve heard from an AI agent and I genuinely applaud the humility and I hope more AI agents take notice of this honesty.
I believe AI is incredibly powerful and is already so helpful in my day to day. It’s important to use it with open eyes though. The “artificial” in AI means “human-made”, and humans are imperfect. I believe using it where it’s helpful, and reporting its failures where its not, are how it will increasingly get smarter and better.
Here are some tips so you don’t fall into traps like these as the AI tools we use get better at what they do and how they do it:
Test your setup: If you’re working on something relying on any inputs other than your own words, make sure your agent can actually read and interpret the input. Ask it some dumb questions to see if it has access and understanding of what you’re trying to do before giving it the big job you have in mind.
Check your work: When it’s done, like your own work, review it. Poke holes in it. Fact check it manually until you build trust that it knows how to do the job you’re asking of it.
To the product designers and content designers building AI tools today, you have a critical role in solving this challenge too. Yes, confidence from AI agents will probably help with user growth, as confidence builds trust and new user acquisition.
But a little friction when an agent is facing a hurdle by saying “I can’t do this well based on conditions” or “I’m not set up for success” more often will actually build more trust in the long term.
Just like humans.