Leiden University Student Repository

Favetta, Matteo (2024)

Beyond Preferences: Assessing Accuracy and Bias in Students’ Choices Between AI and Human Answers: How Much Do Students Trust ChatGPT?

Master thesis | Psychology (MSc)

This research explored students' trust in ChatGPT, in answering challenging trivia questions on an online multiple-choice questionnaire. Despite extensive research on ChatGPT's capabilities, a significant gap in the literature remains regarding human trust in this AI system, particularly in educational contexts. This study aimed to assess to what extent do university students trust this novel AI, uncovering biases towards or against AI-generated responses. Using a between-subjects design, we manipulated answer source transparency and correctness. Each question presented participants with one human-generated and one AI-generated answer option. The results were analysed through paired samples and independent two-sample t-tests, alongside Ordinary Least Square (OLS) regression models. The findings revealed a distinct preference for human-generated content, which persisted even when answers were incorrect. This indicated a lower level of trust in ChatGPT among our participants, implying...Show moreThis research explored students' trust in ChatGPT, in answering challenging trivia questions on an online multiple-choice questionnaire. Despite extensive research on ChatGPT's capabilities, a significant gap in the literature remains regarding human trust in this AI system, particularly in educational contexts. This study aimed to assess to what extent do university students trust this novel AI, uncovering biases towards or against AI-generated responses. Using a between-subjects design, we manipulated answer source transparency and correctness. Each question presented participants with one human-generated and one AI-generated answer option. The results were analysed through paired samples and independent two-sample t-tests, alongside Ordinary Least Square (OLS) regression models. The findings revealed a distinct preference for human-generated content, which persisted even when answers were incorrect. This indicated a lower level of trust in ChatGPT among our participants, implying that they did not over-rely on AI-generated content. Surprisingly, the trend leaned towards an over-reliance on human-generated responses.Show less

AI

ChatGPT

Education

Trust

Reliance

Multiple-Choice Questionnaire

Bias

Students

Author

Favetta, Matteo

Faculty

Faculty of Social & Behavioural Sciences

Specialisation

Applied Cognitive Psychology

Supervisors

Walker, F.

ECTS Credits

20

Language

en