Do They Still Prefer Humans?: Exploring the Influence of Trust, Familiarity, and User Types.

Generative AI (GenAI), such as ChatGPT, introduces both innovation and challenges to higher education. While research points to GenAI’s learning benefits, skepticism and mistrust persist. This study examines whether students undertrust ChatGPT, focusing on how content source preferences (human vs. ChatGPT) impact comprehension and whether familiarity influences trust. Additionally, we explore how interactions with ChatGPT affect trust by categorizing participants into three user types based on their motives for use, interaction depth, skills, and frequency. A two-phase study design was employed, consisting of a between-subjects pre-experiment and a within-subjects main experiment. In the pre-experiment, we obtained the human-generated content for the main experiment, which was objectively of worse quality than the ChatGPT-generated content. During the main experiment, 55 participants read two texts, chose their preferred definitions in both labeled and unlabeled conditions,...Show moreGenerative AI (GenAI), such as ChatGPT, introduces both innovation and challenges to higher education. While research points to GenAI’s learning benefits, skepticism and mistrust persist. This study examines whether students undertrust ChatGPT, focusing on how content source preferences (human vs. ChatGPT) impact comprehension and whether familiarity influences trust. Additionally, we explore how interactions with ChatGPT affect trust by categorizing participants into three user types based on their motives for use, interaction depth, skills, and frequency. A two-phase study design was employed, consisting of a between-subjects pre-experiment and a within-subjects main experiment. In the pre-experiment, we obtained the human-generated content for the main experiment, which was objectively of worse quality than the ChatGPT-generated content. During the main experiment, 55 participants read two texts, chose their preferred definitions in both labeled and unlabeled conditions, completed comprehension quizzes, and responded to surveys about familiarity and ChatGPT use. In the labeled condition, participants could see the origin of the definitions, while they could not in the unlabeled condition. Trust was measured by the proportion of times participants chose ChatGPT definitions over human definitions. Results revealed undertrust as ChatGPT-generated definitions were preferred overall, but less in the labeled condition than in the unlabeled. Reliance on AI didn't improve comprehension, and familiarity revealed a positive but non-significant link with trust. Lastly, trust differences appeared between user types, with skilled users understanding the system better and trusting it more. However, system performance trust remained consistent. These findings suggest that undertrust is still present among familiar users and emphasize the need for educational initiatives teaching ethical and transparent AI use.Show less

Leiden University Student Repository

Documents

In Collections

Do They Still Prefer Humans? Exploring the Influence of Trust, Familiarity, and User Types.