Amy Merrick (Professional Lecturer, College of Communication) presented “AI as a Bias Detector” at the AI in Teaching Symposium, October 18, 2024.

Merrick, a senior professional lecturer in the journalism program in DePaul’s College of Communication, explores how AI can be used to help students identify and address bias in their own writing.

The Problem of Bias

There is a problem of bias in writing, both from the perspective of the writer and the audience. Merrick points out that audiences are often aware of bias in news reporting. A 2020 study by the Pew Research Center found that about 80% of survey respondents believe that news organizations favor one side in their stories. This awareness of bias can erode trust in news sources and make it more difficult for audiences to engage with important information.

AI as a Bias Detector

Merrick’s classroom experiment was inspired by Trusting News, a non-profit group that works with newsrooms on building trust and transparency. Trusting News raised the question of whether ChatGPT could be used by journalists to analyze their own writing for biases. Merrick and her students put this question to the test by asking an LLM chatbot to evaluate a published news story.

The Experiment

The students began by reading and discussing the Society of Professional Journalists Code of Ethics. They then read a published news story and worked in pairs to identify examples of bias and potential ways to improve it. Finally, they asked an LLM chatbot to evaluate the same news story.

Findings

The experiment found that both the students and the chatbot identified similar categories of bias in the news story. Both groups suggested adding context and balance to the story. However, students were more likely to focus on the language used in the story, while the chatbot was more likely to suggest adding data and graphics.

Student Feedback

Students found that both their human partners and the chatbot provided useful suggestions. They found the chatbot’s explanations to be more detailed and comprehensive, but also potentially overwhelming. Students felt that their human partners were more focused and easier to have a back-and-forth discussion with.

Merrick also asked students to rate the usefulness and trustworthiness of the suggestions from their partners and the chatbot. They found the chatbot’s suggestions slightly more useful, but also less trustworthy than the suggestions from their partners.

The Medium and the Messenger

Merrick draws a parallel between her classroom experiment and the work of communication scholar Marshall McLuhan. McLuhan’s famous phrase, “the medium is the message,” suggests that the way information is conveyed is as important as the content itself. Merrick extends this idea to argue that “the messenger is also the message” – the same information coming from different sources may be perceived differently.

Suggestions for Integrating AI

Merrick concludes her presentation with suggestions for how faculty and staff can integrate AI into their teaching. She suggests incorporating a step reflecting on bias in the drafting or revision process, sharing research on both human and AI biases, and experimenting with different methods for delivering feedback on bias.

Merrick’s presentation highlights the potential of AI to be a valuable tool for teaching students about bias in writing. However, it also raises important questions about the trustworthiness of AI and the importance of human interaction in the learning process. By engaging with these questions, faculty and staff can help students develop the critical thinking skills they need to navigate the increasingly complex world of information and communication.