You’d think getting wrong advice would make you more cautious. New research says the opposite happens when that advice comes from AI. A study led by Valerio Capraro, a psychology professor at the University of Milan-Bicocca, found that incorrect AI advice made people less accurate, yet noticeably more confident in their answers. The findings, published through IBM Think, challenge common assumptions about how humans interact with intelligent systems.
The study also found that people who received incorrect AI advice became far less willing to admit they didn’t know an answer, even when staying silent was an option, and they were rewarded for accuracy. This suggests that AI may fundamentally alter the way people evaluate their own knowledge, shifting the psychological threshold required to commit to an answer.
Why Does Wrong AI Advice Make People So Sure of Themselves?
Researchers asked participants six questions about obscure movie details, deliberately picking ones the AI model consistently answered incorrectly. This let them isolate what happens when people consult AI, regardless of whether its advice is actually good. The design was critical: by using questions where the AI was reliably wrong, the researchers could separate the effect of the advice itself from any perceived reliability of the AI.
The results were striking. Without AI, participants declined to answer roughly 36% to 44% of the time across two experiments. With AI available, that dropped to just 3% to 6%. Accuracy fell too, from 27.5% correct when working alone to just 9.2% with AI involved. The magnitude of these shifts indicates that AI does not simply influence decisions—it reshapes the entire decision-making process.
Capraro suspects AI may lower the amount of confidence people need before committing to an answer, essentially shifting how they judge their own certainty. Oddly, this held true even when AI advice appeared automatically without being requested, suggesting the effect is not just about actively seeking help. This phenomenon, sometimes called “automation bias,” occurs when humans defer to machine suggestions even when they conflict with their own knowledge.
How AI Alters Our Metacognition
Metacognition—the awareness and understanding of one’s own thought processes—plays a crucial role in decision-making. Normally, doubt serves as a signal to pause, reconsider, or gather more information. The study suggests that AI may short-circuit this natural check. When an AI provides an answer, even a wrong one, the user may externalize the cognitive effort required to evaluate their own knowledge, effectively outsourcing confidence along with the answer.
Prior research in cognitive psychology has shown that people often overweight machine advice due to a misplaced belief in algorithmic infallibility. This study adds a new dimension: the effect persists even when the AI is known to be unreliable for certain topics. Participants were not told the AI was purposely incorrect, but the questions were chosen from a domain where the model consistently failed. Despite this, the presence of AI still inflated confidence.
Does Raising the Stakes Change Anything?
Adding financial rewards and penalties made people somewhat more cautious and accurate, though the underlying pattern never fully disappeared. In a world where AI tools are increasingly integrated into high-stakes environments—medical diagnosis, legal research, financial planning—this persistence is concerning. Even when people have skin in the game, the seductive pull of a machine-generated answer overrides careful reasoning.
Capraro draws a clear distinction here: AI should augment human judgment, not replace it entirely. The ideal use of AI is as a collaborative partner that provides suggestions while leaving the user engaged in critical thinking. But the study suggests this ideal is difficult to achieve. The mere presence of AI output, regardless of its quality, triggers a cognitive state that reduces vigilance and increases unwarranted certainty.
Broader Implications for Education and Child Development
He worries this effect extends to children too, since growing up with instant answers could prevent kids from experiencing doubt as part of learning. New reports suggest that the AI-hooked younger generation is sourcing chatbot help even for in-person talks. For example, some students now use AI to generate talking points for classroom discussions, bypassing the struggle of formulating their own arguments. This habit may weaken essential skills like hypothesis testing, evidence evaluation, and intellectual humility.
Educational researchers have long argued that productive struggle is a key component of deep learning. When students are deprived of doubt, they may miss opportunities to confront gaps in their understanding. The study implies that AI could inadvertently create a generation of learners who are confident but poorly informed—a dangerous combination in fields that require nuanced judgment.
Related Research and Historical Context
The phenomenon of overreliance on AI is not new. Studies on “automation bias” date back to the 1990s in aviation and medical contexts. For instance, pilots sometimes trust autopilot systems even when they conflict with sensory evidence. Similarly, radiologists have been shown to give undue weight to AI-assisted readings. The current study extends this work into the domain of general knowledge and everyday decision-making.
Another relevant line of research examines the “Dunning-Kruger effect,” where individuals with low competence overestimate their abilities. The AI advice may exacerbate this effect by providing a false sense of expertise. If a person lacks deep knowledge of a topic, but the AI supplies a plausible answer, they may feel they have mastered the subject when they have not.
Implications for AI Design and Policy
The study raises important questions for how AI tools should be designed. Many current interfaces present answers with a veneer of confidence—full sentences, facts, and explanations. Capraro’s work suggests that designers might consider techniques to reduce overreliance, such as explicitly stating confidence intervals, requiring users to answer first before seeing AI suggestions, or prompting users to generate their own reasoning.
Policy makers and educators should also take note. As AI becomes ubiquitous in schools and workplaces, training programs must emphasize critical evaluation of AI-generated content. Digital literacy curricula should include lessons on cognitive biases related to technology, helping people recognize when they are being swayed by an answer’s source rather than its content.
Looking Forward: Doubt as a Gateway to Knowledge
As Capraro puts it, doubt isn’t a failure of knowledge. It’s often where real knowledge actually begins. The study serves as a reminder that while AI can be an incredibly powerful tool, it carries hidden risks that affect the very way we think. Understanding these psychological effects is essential to ensuring that AI serves as a true augmentation of human intelligence rather than an impediment to it.
Future research will likely explore whether the effect can be mitigated by different types of AI interactions, such as offering multiple alternative answers or requiring users to articulate their reasoning. Until then, the study stands as a cautionary tale about the subtle but profound ways AI can shape our minds, often in directions we do not expect.
Source: Digital Trends News