Beyond Fact or Fiction: Evaluating the Advanced Fact-Checking Capabilities of Large Language Models like GPT-4

Researchers at the University of Zurich focus on the role of large language models (LLMs) like GPT-4 in autonomous fact-checking, evaluating their ability to formulate queries, retrieve contextual data, and make decisions while providing explanations and quotes. The results indicate that LLMs, particularly GPT-4, perform well with contextual information, but accuracy varies depending on the query language and the veracity of the statements. While showing promise in fact-checking, inconsistencies in accuracy highlight the need for more research to better understand its capabilities and limitations.

Automated fact-checking research has developed with diverse approaches and shared tasks over the past decade. Researchers have proposed components such as assertion detection and evidence extraction, often based on large language models and sources such as Wikipedia. However, ensuring explainability remains a challenge, as clear explanations of fact-checking verdicts are crucial for journalistic use.

The importance of fact-checking has increased with the rise of online misinformation. Hoaxes triggered this surge during major events such as the 2016 US presidential election and the Brexit referendum. Manual data verification needs to be improved for the vast amount of information online, requiring automated solutions. Large language models like GPT-4 have become vital for verifying information. Greater explainability in these models is a challenge in journalistic applications.

The current study evaluates the use of LLM in fact-checking, focusing on GPT-3.5 and GPT-4. The models are evaluated under two conditions: one without external information and another with access to the context. The researchers introduce an original methodology using the ReAct framework to create an iterative agent for automated fact checking. The agent autonomously decides whether to conclude a search or continue with more queries, with the aim of balancing precision and efficiency, and justifies its verdict with the aforementioned reasoning.

The proposed method evaluates LLMs for autonomous fact checking, and GPT-4 generally outperforms GPT-3.5 on the PolitiFact dataset. Contextual information significantly improves LLM performance. However, caution is advised due to variable accuracy, especially in nuanced categories such as half true and mostly false. The study calls for further research to improve understanding of when LLMs excel or fail at fact-checking tasks.

GPT-4 outperforms GPT-3.5 in fact checking, especially when contextual information is incorporated. However, accuracy varies depending on factors such as query language and completeness of claims, especially in nuanced categories. It also emphasizes the importance of informed human oversight when implementing LLM, as even a 10% error rate can have serious consequences on the current information landscape, highlighting the irreplaceable role of human fact-checkers.

Further research is essential to comprehensively understand the conditions under which LLM agents excel or fail at fact-checking. It is a priority to investigate the inconsistent accuracy of LLMs and identify methods to improve their performance. Future studies can examine LLM performance across query languages ​​and its relationship to the truthfulness of claims. Exploring various strategies to equip LLMs with relevant contextual information has the potential to improve fact-checking. Analyzing the factors that influence how models improve the detection of false claims compared to true ones can provide valuable information to improve accuracy.

Review the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to join. our 32k+ ML SubReddit, Facebook community of more than 40,000 people, Discord channel, and Electronic newsletterwhere we share the latest news on ai research, interesting ai projects and more.

If you like our work, you’ll love our newsletter.

we are also in Telegram