The “Label” Effect: How Knowing the Source Influence Students Trust in AI

The emergence of Generative AI, particularly Large Language Models, has shown promise in supporting learning. While existing research has focused on LLMs’ technical capabilities, little is known about students’ trust in these systems despite its importance in learning and education, which limits our understanding of their adoption and effective use. This study aims to cover this gap by investigating whether students undertrust ChatGPT as a learning tool. In the study, participants had to read two stories, where specific keywords were highlighted and paired with definitions. The research included a pre-test (N = 66) to discriminate between less versus more informative human-generated definitions, and a main test (N = 55) where the least informative human-generated definitions were compared with more informative AI-generated ones. The main test employed a two-by-two within-subjects design manipulating source type (AI vs. Human) and labeling presence. When investigating labeling...Show moreThe emergence of Generative AI, particularly Large Language Models, has shown promise in supporting learning. While existing research has focused on LLMs’ technical capabilities, little is known about students’ trust in these systems despite its importance in learning and education, which limits our understanding of their adoption and effective use. This study aims to cover this gap by investigating whether students undertrust ChatGPT as a learning tool. In the study, participants had to read two stories, where specific keywords were highlighted and paired with definitions. The research included a pre-test (N = 66) to discriminate between less versus more informative human-generated definitions, and a main test (N = 55) where the least informative human-generated definitions were compared with more informative AI-generated ones. The main test employed a two-by-two within-subjects design manipulating source type (AI vs. Human) and labeling presence. When investigating labeling presence, definitions were explicitly labeled as “ChatGPT” or “Human” in the labeled condition; in the unlabeled condition, no source was provided. Participants selected the most informative definition for each keyword and then answered comprehension questions about the stories and reported their trust in ChatGPT. Findings showed that when no source was provided, participants preferred the AI-generated definitions. However, explicit labeling reduced this preference and led to more selection of lower-quality human definitions, indicating that students do undertrust. Contrary to the initial expectation, higher reliance on AI definitions did not improve comprehension. These findings highlight the importance of fostering well-calibrated trust in AI tools with their actual capabilities, ensuring users make informed and effective decisions.Show less

Leiden University Student Repository

Documents

In Collections

The “Label” Effect: How Knowing the Source Influence Students Trust in AI