To what extent do readability and cohesion of AI-generated summaries differ from the readability and cohesion of human-written scientific articles: an exploratory study

Reading and comprehending scientific articles are crucial learning tasks for university students. Text comprehension is tied to text readability (a measure of how easy the text is written) and text cohesion (a measure of connectedness of concepts in that text that indirectly indicates the reader’s effort to comprehend that text). In recent years, university students started to apply AI systems such as ChatGPT to facilitate their reading process, generating summaries of scientific articles with AI. In this study, we measured readability and cohesion of scientific articles (N = 96) with Coh-Metrix software and compared the obtained data to readability and cohesion of two types of those articles’ AI-generated summaries, running a series of repeated-measures ANOVAs on four measures of readability and on seven measures of cohesion. We used the GPT 4.1 model to create two types of summaries: to explain those articles to the general public (“simple summaries”) and to help a student to...Show moreReading and comprehending scientific articles are crucial learning tasks for university students. Text comprehension is tied to text readability (a measure of how easy the text is written) and text cohesion (a measure of connectedness of concepts in that text that indirectly indicates the reader’s effort to comprehend that text). In recent years, university students started to apply AI systems such as ChatGPT to facilitate their reading process, generating summaries of scientific articles with AI. In this study, we measured readability and cohesion of scientific articles (N = 96) with Coh-Metrix software and compared the obtained data to readability and cohesion of two types of those articles’ AI-generated summaries, running a series of repeated-measures ANOVAs on four measures of readability and on seven measures of cohesion. We used the GPT 4.1 model to create two types of summaries: to explain those articles to the general public (“simple summaries”) and to help a student to prepare for exam (“exam summaries”). Simple summaries had the highest readability scores, while exam summaries had the readability scores lower than those of original scientific articles. On measures of cohesion, we did not observe conclusive evidence that AI-powered summarization results in increased text cohesion. Exam summaries were less cohesive than original articles on three measures of cohesion and were less cohesive than simple summaries on five measurers of cohesion. We concluded that summarization is not always beneficial for text readability and text cohesion, as a differential choice of a prompt may make the generated summary less readable and less cohesive than the original article. We further concluded that it is premature to claim that AI-generated summaries help readers comprehend scientific articles better. Our study may lead to a better understanding of how the use of generative AI changes the landscape of reading in academia.Show less

Leiden University Student Repository

Documents

In Collections

To what extent do readability and cohesion of AI-generated summaries differ from the readability and cohesion of human-written scientific articles: an exploratory study