Amidst a surge in misinformation during a brief conflict between India and Pakistan, social media users turned to AI chatbots for verification. However, these chatbots often provided more false information, highlighting their unreliability as fact-checking tools. As major tech platforms continue to reduce the number of human fact-checkers, users are increasingly relying on AI-driven chatbots such as xAI’s Grok, OpenAI’s ChatGPT, and Google’s Gemini for reliable information.
The Rise of AI Chatbots for Fact-Checking
On Elon Musk’s X platform (formerly Twitter), asking "@Grok, is this real?" has become a common practice among users. Grok, the AI assistant integrated into X, reflects the growing trend of seeking instant debunking on social media. However, responses from these AI chatbots are often riddled with misinformation.
Grok recently came under scrutiny for inserting the far-right "white genocide" conspiracy theory into unrelated queries. Erroneously identifying old video footage from Khartoum Airport in Sudan as a missile strike on Pakistan’s Nur Khan Airbase during the Indo-Pak conflict, Grok further misidentified a burning building in Nepal as "possibly" depicting a military response from Pakistan to an Indian attack.
Limitations of AI Fact-Checking
McKenzie Sadeghi, a researcher at NewsGuard, an organization that monitors information, told Agence France-Presse (AFP) that "As X and other large tech companies have cut back on investments in human fact-checkers, there’s been increased reliance on Grok as a fact-checker." She cautioned that "Our research has repeatedly found that AI chatbots are not reliable sources of news and information, especially when it comes to breaking news."
NewsGuard’s research has found that 10 leading chatbots, including Grok, readily repeat false narratives, including Russian disinformation narratives and false or misleading claims related to recent Australian elections. A recent study by the Tow Center for Digital Journalism at Columbia University examined eight AI search tools and found that chatbots "often do not do well at refusing to answer questions that they cannot answer accurately, instead offering incorrect or speculative answers."
AFP’s fact-checkers in Uruguay asked Gemini about an AI-generated image of a woman. Gemini not only confirmed the image was real but also fabricated details about the woman’s identity and where the image might have been taken.
Grok recently labeled a video purporting to show a giant anaconda swimming in the Amazon River as "real," even citing plausible-sounding scientific expeditions to back up its false claim. In reality, the video was AI-generated, and AFP’s fact-checkers in Latin America reported that numerous users had cited Grok’s assessment as proof that the video was genuine.
Concerns About AI Fact-Checking
These findings raise concerns as surveys indicate a growing number of online users are turning to AI chatbots instead of traditional search engines to gather and verify information. Meanwhile, Meta announced earlier this year that it would end its third-party fact-checking program in the United States, shifting debunking duties to ordinary users using a model called "Community Notes," which has been promoted by the X platform. However, researchers have repeatedly questioned the effectiveness of "Community Notes" in combating disinformation.
Human fact-checking has long been a flashpoint in a polarized political climate, particularly in the United States, where conservatives claim it suppresses free speech and censors right-wing content, an assertion that professional fact-checkers strongly deny.
AFP currently partners with Facebook’s fact-checking program in 26 languages, including in Asia, Latin America, and the European Union.
The quality and accuracy of AI chatbots can vary depending on their training and programming, raising concerns that their output could be subject to political influence or control.
Musk’s xAI recently blamed an "unapproved modification" for Grok generating unsolicited posts referencing "white genocide" in South Africa. When AI expert David Caswell asked Grok who might have modified its system prompts, the chatbot listed Musk as the "most likely" culprit.
Musk, who is a South African-born billionaire and a supporter of former U.S. President Donald Trump, has previously spread baseless claims that South African leaders were "openly pushing for genocide of white people."
Angie Holan, director of the International Fact-Checking Network, told AFP that "We have seen AI assistants fabricate results or give biased answers after human coders specifically changed their instructions." "I am particularly concerned about how Grok handles requests on very sensitive subjects after being instructed to provide pre-authorized answers."
Risks of AI Chatbots in the Information Landscape
The increasing prevalence of AI chatbots in providing information raises several serious concerns involving their potential for misleading and manipulative influence. While these technologies offer advantages in rapidly accessing information and streamlining research, the inherent limitations and biases can inadvertently or deliberately spread misinformation or propaganda.
Inherent Limitations of AI Chatbots
AI Chatbots learn by analyzing vast amounts of textual data but lack the critical thinking capabilities of humans. This means that they may extract patterns and generate seemingly plausible responses from the data but not truly understand the underlying meaning or context. This lack of understanding can lead to the production of inaccurate, incomplete, or misleading information.
Furthermore, the training data used for AI chatbots may contain biases, which are then reflected in the generated responses. For example, if the training data is primarily from a specific source or viewpoint, the chatbot may exhibit a preference toward that source or viewpoint, thereby propagating biased information.
Risks Posed by AI Chatbots
With a growing number of people relying on AI chatbots for information, the risk of spreading misinformation increases. If a chatbot generates inaccurate or misleading information, users may unknowingly accept and spread it, leading to the proliferation of false beliefs.
Additionally, AI chatbots may be used for malicious purposes, such as disseminating propaganda or manipulating public opinion. By carefully designing the training data or tweaking the algorithms, malicious actors can manipulate the chatbot to generate specific types of responses that influence users’ beliefs and behaviors.
Mitigating the Risks
To mitigate the risks associated with AI chatbots, several measures are required. First, developers should strive to ensure that the training data for AI chatbots is as comprehensive and unbiased as possible. This may involve including a diverse range of data samples from different sources and viewpoints to reduce the impact of biases.
Second, AI chatbots need to be rigorously tested and evaluated to identify and correct any tendencies toward inaccuracy or bias. This may involve human verification of the chatbot’s generated responses and the use of various metrics to assess its accuracy and impartiality.
Third, users should maintain a critical mindset and not blindly trust the information provided by AI chatbots. Users should always cross-reference information from different sources and be aware of the potential for bias in the chatbot’s responses.
Role of Tech Platforms and Communities
Technology platforms and communities play a crucial role in regulating and overseeing the content generated by AI chatbots. Platforms can implement policies to identify and remove misinformation or propaganda, and they can collaborate with fact-checking organizations to verify the accuracy of the chatbot’s generated responses. In addition, communities can help raise user awareness by fostering discussions about the limitations and biases of AI chatbots and encouraging users to maintain a critical mindset.
Conclusion
AI chatbots are useful tools, but they also come with inherent limitations and risks. To ensure that these technologies are used responsibly and to reduce the spread of misinformation and manipulation, vigilance is required from developers, regulators, and users alike. By taking the necessary measures, we can maximize the benefits of AI chatbots while minimizing their potential harms.