Have you ever tried to convince a conspiracy theorist that the moon landing wasn’t staged? You likely didn’t succeed, but ChatGPT might have better luck, according to research by MIT Sloan School of Management professor David Rand and American University professor of psychology Thomas Costello, who conducted the research during his postdoctoral position at MIT Sloan.
In a new paper “Durably reducing conspiracy beliefs through dialogues with AI” published in Science, the researchers show that large language models can effectively reduce individuals’ beliefs in conspiracy theories — and that these reductions last for at least 2 months — a finding that offers new insights into the psychological mechanisms behind the phenomenon as well as potential tools to fight the spread of conspiracies.
To test their theory, the authors harnessed the power of GPT-4 Turbo, OpenAI’s most advanced large language model, to engage over 2,000 conspiracy believers in personalized, evidence-based dialogues. The study employed a unique methodology that allowed for deep engagement with participants’ individual beliefs. Participants were first asked to identify and describe a conspiracy theory they believed in using their own words, along with the evidence supporting their belief.
GPT-4 Turbo then used this information to generate a personalized summary of the participant’s belief and initiate a dialogue. The AI was instructed to persuade users that their beliefs were untrue, adapting its strategy based on each participant’s unique arguments and evidence.
These conversations, lasting an average of 8.4 minutes, allowed the AI to directly address and refute the specific evidence supporting each individual’s conspiratorial beliefs, an approach that was impossible to test at scale prior to the technology’s development.
The results of the intervention were striking. On average, the AI conversations reduced the average participant’s belief in their chosen conspiracy theory by about 20%, and about 1 in 4 participants — all of whom believed the conspiracy beforehand — disavowed the conspiracy after the conversation. This impact proved durable, with the effect remaining undiminished even two months post-conversation.
The AI conversation’s effectiveness was not limited to specific types of conspiracy theories. It successfully challenged beliefs across a wide spectrum, including conspiracies that potentially hold strong political and social salience, like those involving COVID-19 and fraud during the 2020 U.S. presidential election.
While the intervention was less successful among participants who reported that the conspiracy was central to their worldview, it did still have an impact, with little variance across demographic groups.
Notably, the impact of the AI dialogues extended beyond mere changes in belief. Participants also demonstrated shifts in their behavioral intentions related to conspiracy theories. They reported being more likely to unfollow people espousing conspiracy theories online, and more willing to engage in conversations challenging those conspiratorial beliefs.
“This research indicates that evidence matters much more than we thought it did — so long as it is actually related to people’s beliefs,” co-author professor Gordon Pennycook, from Cornell University, said. “This has implications far beyond just conspiracy theories: Any number of beliefs based on poor evidence could, in theory, be undermined using this approach.”
—
Source: MIT Sloan