Beyond Preference: Assessing  Accuracy and Bias in Student Choices  between AI and Human Answers

This study delves into the intricate dynamics of how students interact with ChatGPT, a new chatbot using machine learning algorithms, specifically examining whether they tend to place excessive trust in the information it provides. The study, involving 169 student participants, uses an experimental design to explore biases, trust perceptions and decision-making processes related to both AI-generated and human expert answers. Two different groups are examined in this study - the labelled group, with knowledge about the answer source, and the unlabelled group, which operates without information about the answer source. The study reveals a preference for answers from human experts, both in the overall sample and in the labelled group. Interestingly, this preference is absent in the unlabelled group, where students do not show a preference when unaware of the answer source. Furthermore, participants in all groups consistently show a tendency to choose correct answers over incorrect ones...Show moreThis study delves into the intricate dynamics of how students interact with ChatGPT, a new chatbot using machine learning algorithms, specifically examining whether they tend to place excessive trust in the information it provides. The study, involving 169 student participants, uses an experimental design to explore biases, trust perceptions and decision-making processes related to both AI-generated and human expert answers. Two different groups are examined in this study - the labelled group, with knowledge about the answer source, and the unlabelled group, which operates without information about the answer source. The study reveals a preference for answers from human experts, both in the overall sample and in the labelled group. Interestingly, this preference is absent in the unlabelled group, where students do not show a preference when unaware of the answer source. Furthermore, participants in all groups consistently show a tendency to choose correct answers over incorrect ones, indicating an inherent ability to distinguish accurate, high-quality information regardless of their awareness of the source. In conclusion, this study offers valuable insights into how students navigate and interact with ChatGPT. The notable preference for answers from human experts in the labelled group indicates a possible source bias. However, the consistent selection of correct answers in both groups underscores students' skill in identifying high-quality information. As such, this study highlights the trust dynamics in an AI-influenced society and emphasises the need for educating the new generation about the transformative role of AI.Show less

Leiden University Student Repository

Documents

In Collections

Beyond Preference: Assessing Accuracy and Bias in Student Choices between AI and Human Answers