AI’s Go-to Gobbledygook: The Overuse Epidemic

In an intriguing turn of events, a newly released study explores the measurable impact of large language models (LLMs) on scientific literature. Spotted by Ars Technica, this yet-to-be-peer-reviewed study draws parallels between epidemiology and linguistic analysis, providing a novel perspective on how AI is reshaping academic writing. By employing a method reminiscent of “excess deaths” used during the COVID-19 pandemic, researchers measured “excess word usage” in biomedical papers to detect the influence of LLMs.

Using papers published before 2023 as a baseline, researchers compared them with those released during the commercialization of LLMs like ChatGPT. The findings were astonishing: words that were once considered rare, such as “delves,” experienced a 25-fold increase in usage. Other terms like “showcasing” and “underscores” were used nine times more frequently than before. Even some common words like “potential,” “findings,” and “crucial” saw frequency boosts of up to 4 percent. The study suggests that as much as 10 percent of abstracts in 2024 were processed using LLMs, a revelation that underscores the profound impact of AI on scientific writing.

Interestingly, the study reveals a shift from “content” words to “style” words in the lexicon of academic papers. While the excess word usage between 2013 and 2023 featured terms like “Ebola,” “coronavirus,” and “lockdown”—all nouns tied to significant real-world events—the excess words in 2024 were predominantly verbs and adjectives. Of the 280 excess style words identified, two-thirds were verbs and about a fifth were adjectives. This subtle yet significant change points to the evolving nature of scientific discourse, heavily influenced by AI-driven language models.

The researchers used these excess style words as markers to estimate the prevalence of AI processing in academic papers globally. They found that around 15 percent of papers from non-English speaking countries like China, South Korea, and Taiwan were likely AI-processed, compared to just 3 percent in native English-speaking countries like the United Kingdom. This disparity might be attributed to the proficiency of native speakers in subtly integrating LLM outputs, making AI usage less detectable.

However, the study does caution that the presence of these words alone does not definitively indicate AI-generated text. It remains a marker rather than conclusive proof. Nonetheless, the study provides a fascinating glimpse into the evolving landscape of scientific writing, highlighting the transformative role of AI. As large language models continue to refine and expand their capabilities, their influence on academia is likely to grow, raising important questions about authenticity, originality, and the future of scientific communication.

In summary, the study opens a Pandora’s box of considerations for the academic community. While LLMs offer unprecedented capabilities, their integration into scientific writing necessitates a careful examination of both the benefits and potential pitfalls. As the debate continues, one thing is clear: the world of academia will never be the same again.