Can artificial intelligence assess the quality of literature? Yes, say researchers at Aarhus University

This post is also available in: Danish

Since September 2021, Professor of Literature Mads Rosendahl Thomsen and five research colleagues have been working on the Fabula-NET project, an interdisciplinary collaboration between literary studies, linguistics and informatics. After three years of effort, the team has developed a technology that can assess whether an unknown work has a special literary quality.

Read more: Aarhus becomes center of new international pioneer center: Medical data research for DKK 250 million – TechSavvy

“It would have been hopeless for us as humans to read and evaluate the 9,000 works that we have included in the project. We’ve used machine learning – or artificial intelligence – to increase our understanding of different forms of literary quality. The many types of data and the ability to train a system to predict what will be successful means that we can now see the literary landscape more clearly,” says Mads Rosendahl Thomsen, Professor of Literary History at the Department of Communication and Culture and Head of the Center for Language Generation and AI (CLAI) at Aarhus University in a press release.

Creating a deeper understanding of literature and quality

The project aims to create a deeper understanding of literary quality and develop a more qualified system for assessing literary works that can benefit publishers, authors, librarians and researchers.

Read also: Every faculty at Aarhus University gets its own innovator – TechSavvy

“We can see a connection between people’s preferences for literature and the intrinsic properties that literature has. We see it as basic humanities research. We can look backwards and better understand how literary history is connected, and we can see if a work has been unfairly overlooked. We can also look forward and understand why a work has a particular literary quality. This is important to understand,” says Mads Rosendahl Thomsen.

The researchers have ambitions to make the technology accessible to everyone and plan to develop a dashboard where users can upload their texts.

“We’ve found that readers react negatively if a work is too random or too predictable. By focusing on the emotional perspectives in the individual words, a so-called sentiment analysis, we can go further and calculate how predictable a narrative is,” says Mads Rosendahl Thomsen.

This allows editors to identify manuscripts with potential, while authors can evaluate and improve their texts. At the same time, researchers can analyze large text collections from world literature at a more advanced level.

The research group plans to complete the Fabula-NET project in June 2025. The project is funded with DKK 5.3 million from VELUX FONDEN’s core group program.

How Fabula-NET works: Feeding the technology with success parameters

The researchers used the so-called Chicago corpus, which consists of over 9,000 works published in the US between 1880 and 2000.

They have fed the technology with high-quality fiction by award-winning authors, including Nobel Prize in Literature winner Toni Morrison and National Book Award winner Don DeLillo, as well as mainstream literature by Agatha Christie and important works of so-called genre literature by J.R.R. Tolkien, among others.

The researchers then worked with three success parameters for a work.

The intrinsic success parameter looks at the internal structure of a work, such as readability or stylistic characteristics. It can reveal how a work that is either randomly or predictably structured affects the reader.

The extrinsic success parameter evaluates how often libraries lend a work, how many copies stores sell, or how positive reviewers view a work.

The preferential success parameter examines whether a work appeals more to, for example, men or women or young or old, and whether a work is more popular with target audiences depending on whether they are rural or urban.