We present a paper that explores how critical thinking influences the outcomes of collaboration between humans and generative artificial intelligence (Gen AI), focusing on what happens when AI provides incorrect or partially correct answers. The use of Gen AI, such as ChatGPT, is becoming increasingly widespread in professional settings, as well as in educational institutions.
The research was conducted on a sample of students and young professionals, groups for whom Gen AI has been widely adopted.
What did we learn?
Lack of critical thinking and overreliance on AI
When participants were offered a solution by the ChatGPT 4o model that was partially correct (containing the correct reasoning steps, but an incorrect final answer due to a small interpretation error), the correct answer rate dropped to only 3%, which is significantly lower than the 15% correct answers in the control group that did not have the help of AI. This result suggests that individuals tend to overrely on AI. Participants accepted the incorrect final solution, rather than critically evaluating the reasoning steps provided by the AI model. Such reliance manifests itself through increased trust in AI and reduced cognitive effort, as participants who used AI spent significantly less time on the task. This “cognitive outsourcing”outsourcingposes serious risks for the education sector, as repeated reliance on AI without critical evaluation can weaken judgment skills over time and reduce learning. Previous research has already shown that students who rely on AI for exam preparation learn less and perform worse on self-tests.
The effectiveness of critical evaluation reminders
The good news is that response accuracy can be improved with a simple intervention. In another experiment, when participants were given explicit instructions to use ChatGPT “deliberately, taking time to analyze before responding,” the correct response rate increased significantly, reaching 24%. This shows that a reminder can significantly improve the quality of outcomes when AI makes mistakes, by encouraging participants to re-evaluate the AI’s response rather than automatically accepting it.
Conclusion
Given that "hallucinations" (the generation of incorrect information) by AI models are becoming more common, it is crucial for organizations and educational institutions to actively cultivate critical thinking skills in users. This includes training that highlights both the capabilities and limitations of AI systems, thereby reducing over-reliance and maintaining the long-term cognitive capacities of individuals.
