The Republic is a journal published by The Palantir Foundation, featuring commentary and essays on technology, national security policy, and international affairs.

Fabricated Futures: The AI Disinformation Threat

Fabricated Futures: The AI Disinformation Threat

As Authoritarian Regimes Leverage LLMs For Power, The West Should Answer With Its Values of Transparency.

By

Mohamed Suliman is a senior disinformation researcher at Northeastern University’s Civic AI Lab. He holds a degree in engineering from the University of Khartoum.

For most in the West, the recent ascendancy of Artificial Intelligence (AI) has been understood in terms of the large language models (LLMs) and the enormous benefits they bring in saved time and the ease of written communication. And while many have noted LLMs’ enablement of high-quality, plausible fabricated content, less well-known is just how much state actors have utilized this technology to enhance their disinformation campaigns, even in the brief years since the debut of ChatGPT in late 2022.

These state actors have opened the possibility of a completely different future for disinformation warfare. Tech companies and governments have begun measures to counter these campaigns, but to understand the true impact of these tools – and design robust policy solutions – more access to data is necessary than is currently available.

From Russia, With Venom

Russia, especially after the invasion of Ukraine, has emerged as a principal player and investor in using AI in this type of war. For instance, in 2023, pro-Kremlin sources created three videos depicting the then-commander-in-chief of Ukraine’s armed forces General Valerii Zaluzhnyi blaming Zelenskyy for killing his aide and warned of his potential premature death.1 In another case, in January 2024, a pro-Russian group used AI to doctor a photo of the ex-president on a Zoom call.2 The group on the call was then tricked into speaking against the interests of Ukraine.

In May 2024, Russia also used AI disinformation in other countries such as Moldova to intervene in domestic politics by spreading an AI-created video for its president Maia Sandu outlawing the picking of rose hips.3 Moreover, Russia is continuously targeting U.S. citizens with AI disinformation. In February 2024, an interview with Texas Governor Greg Abbott was altered to have him encourage Biden to work with Putin for national interests.4 In a more sophisticated AI disinformation case, Russia resorted to creating a network of websites that masquerade as local American newspapers that publish fake news.5

In July 2024, the U.S. announced that it had dismantled a Russian government-backed AI disinformation campaign, but once again, during the ongoing election, the Russian propaganda machine spread false whistleblower videos claiming Tim Walz, the vice-presidential candidate, was engaged in sexual abuse.6 Other Western countries were also hit by Russia’s AI disinformation campaigns. In March 2024, Insikt Group identified a Russian influence network that uses LLMs to plagiarize, translate, and edit content from mainstream media and then push it to inauthentic websites in France, the United Kingdom, and the United States.7

China’s Lies

China has relied on AI to create and boost its disinformation campaigns, targeting its international competitors and regional foes. In April 2024, a threat analysis by Microsoft revealed that China used AI to design its influence operations to attack the United States amplifying controversial domestic issues in addition to criticizing the administration.8

In other similar cases, China repeatedly targeted U.S. voters with different AI disinformation tools such as massive AI-controlled fake accounts and AI-created videos.9 The Chinese government has, for years, run “spamouflage” campaigns that spread the anti-U.S. narrative, using AI-generated images and social media posts that portrayed former President Joe Biden as war-obsessed for spending tax money in conflicts around the world.10

China also makes use of AI disinformation in its dispute with Taiwan, spreading phony audio clips of Foxconn owner Terry Gou, an Independent Party candidate in Taiwan’s presidential race, or displaying AI news anchors to disseminate fabricated content.11 China reportedly used ChatGPT to publish social media posts aimed at influencing public opinion on treated water released from the crippled Fukushima No. 1 nuclear power plant, over which China and Japan have been in a dispute regarding the water’s impact on the Pacific Ocean.12 In yet another case, the Chinese Ministry of State Security meddled in the domestic politics of Taiwan by spreading a fake AI video about what it falsely claimed to be the “secret history” of President Tsai Ing-wen.13

Mideast Misinformation

Iran, likewise, has leveraged AI disinformation. In August 2024, a report by OpenAI announced that it had disrupted a covert Iranian influence operation that attempted to use its system to generate fake articles and social media comments for topics that included the U.S. presidential election.14 The company took down the accounts used in this operation. In a similar case, in
August 2024, an analysis conducted by the Microsoft threat team found that a group called Storm 2035, which is linked to the Iranian Revolutionary Guard Corps is behind many disinformation campaigns some of which included creating a website using AI to spread fake news to the U.S. voters.15

In October 2023, a parallel AI disinformation war started between Hamas and Israel: viral AI generated images were circulated to serve purposes such as spreading fabricated content about the other waring party or exacerbating the attacks on civilians to gain international support and attention. For example, images depicting children suffering in Gaza camps, families living in tents, or a father holding his children amidst devastation, and Beirut burning because of Israel airstrikes, were all found to be AI-generated.16 False photos were also generated to show Indonesian women supporting Israel.17 Some of these images were convincing enough that known websites and companies such as Adobe sold them to customers.18

Western Values – Especially Transparency – Are the Response

Western tech companies, and their governments, are responding with various solutions to limit the impact of AI disinformation campaigns.19 Meta has shut down accounts linked to AI disinformation campaigns. Google results now provide context that explains when the page is AI-generated, and videos on YouTube are now required to disclose altered and synthetic content.20 By contrast, X has a relaxed rule for synthetic media that requires the material to be significantly edited before it can be considered in violation and thus removed from the platform.21

Microsoft and OpenAI are also issuing periodic reports about their internal disruption of covert influence operations by foreign actors.22 Independent researchers are analyzing the impact of Russian AI-generated disinformation and creating freely accessible datasets of incident reports.23

In recent years, media literacy programs have been started to educate the public about the risk and nature of AI disinformation; a bipartisan effort was started to inoculate U.S. voters against AI disinformation content, and organizations such as Witness are conducting workshops and training programs in different countries to educate the public.24 States like California have also passed legislation requiring mandatory fake news detection in the primary school curriculum.25

The FBI has publicly warned that China started a plan to steal AI technology and products and use them in its influence operations within the United States.26 In 2024, the U.S. government accountability office issued a report that part of it analyzed the effort by the Departments of State, Homeland Security, and Defense in detecting and responding to foreign government AI disinformation.27 Moreover, the Foreign Malign Influence Center (FMIC) within the Office of the Director of National Intelligence (ODNI) monitored closely foreign disinformation interference.28

Ahead of the 2024 elections, the U.S. government-imposed sanctions on Iranian and Russian groups that were found to be using AI to spread disinformation about the candidates.29

The legal framework in the United States has started to address the threat, as well. The under-discussion Deepfake Accountability Act aims to protect national security against threats posed by deepfakes and requests the Secretary of Homeland Security to submit an annual unclassified report on efforts to counter deepfakes by Russia and China.30 The bill also requests the Department of Homeland Security to establish an information-sharing program to alert online platforms and news organizations of foreign interference operations.

While it is understandable that the U.S. government should keep secretive the nature of deepfake detection technology in order to gain a competitive advantage, attempted attacks by foreign actors should be disclosed to the public. The Western value of transparency can become a crucial means by which the West defends itself – by unlocking the discretionary effort of private actors to analyze and suggest defensive measures against such attacks in the future.

In this, the U.S. government could launch a data initiative, whereby relevant stakeholders such as social media platforms, AI companies, and federal agencies openly share information on disinformation attacks and invite insights from across the range of participants. This is necessary as platforms such as Facebook are currently shutting down tools that support research and investigation into how malicious actors spread disinformation content.31

Such an initiative would not only guide the process of robust policy formulation, it would also be an expression of the open, democratic values that have made the West great.

Europe has already taken a progressive step to be transparent about the harm of disinformation. In 2022, the European Commission adopted the Digital Service Act (DSA), which aims to address illegal content such as disinformation. The act requires social media platforms and intermediaries to disclose all necessary information and data. To fully understand the impact of Russian disinformation in the Germany’s elections, two non-for-profit organizations, Germany’s Democracy Reporting International (DRI) and the Society for Civil Rights filed a lawsuit against X alleging that it violates the Digital Service Act. The Berlin Regional Court ruled that X must grant both organizations unrestricted access to all publicly available data until shortly after the election.32

In comparison, the United Kingdom still lags behind in this issue. In 2023 it passed the Online Safety Act that only requests social media companies to remove illegal, state-sponsored disinformation through the Foreign Interference Offence, but it falls short in making it compulsory for these companies to share data with the public about disinformation activities.33

  1. “Radio Truha,” Telegram, November 7, 2023. Archived at the Wayback Machine. ↩︎
  2. Andrea Januta, “Investigation: Apparent Russian disinformation group posing as ex-president Poroshenko targets foreign fighters in Ukraine,” Kyiv Independent, January 19, 2024. ↩︎
  3. Clea Caulcutt and Veronika Melkozerova, “Moldova Fights to Free Itself from Russia’s AI-Powered Disinformation Machine,” Politico, February 6, 2024. ↩︎
  4. Reuters, “Fact Check: Abbott Fox News Interview Altered to Say Biden Should ‘Work with’ Putin,” Reuters, February 2, 2024. ↩︎
  5. “Ukraine War: The Family Torn Apart by Russia’s Invasion,” BBC News, February 24, 2024. ↩︎
  6. Eric Tucker, “US Dismantles Russian Government-Backed AI Disinformation Campaign,” NBC Washington, July 9, 2024; David Gilbert, “Russian Propaganda Unit Appears to Be Behind Spread of False Tim Walz Sexual Abuse Claims,” WIRED, October 21, 2024. ↩︎
  7. Insikt Group, “Russia-Linked CopyCop Uses LLMs to Weaponize Influence Content at Scale,” Recorded Future, May 9, 2024. ↩︎
  8. Microsoft Threat Intelligence, “Same targets, new playbooks: East Asia threat actors employ unique methods,” Microsoft, April 2024. ↩︎
  9. “The X Disinformation Network Linked to China,” The Cyber Express, February 27, 2025; David McCabe, “Pro-China YouTube Disinformation Campaign Targeting U.S. Audience,” New York Times, December 14, 2023. ↩︎
  10. Microsoft, “MTAC East Asia Report.” ↩︎
  11. Lily Kuo, “How China Is Using AI News Anchors to Deliver Its Propaganda,” The Guardian, May 18, 2024 ↩︎
  12. “OpenAI Report: Fukushima,” The Japan Times, May 31, 2024. ↩︎
  13. “Taiwan Military Confirms Cyber Attack by China,” Taipei Times, January 11, 2024. ↩︎
  14. “Disrupting a Covert Iranian Influence Operation,” OpenAI. ↩︎
  15. Microsoft, “Document Report.” ↩︎
  16. “Fact Check: AI-Generated Images of Children in Gaza,” DW, October 15, 2024. ↩︎
  17. “People Living in Tent in Gaza: Viral Images Are AI-Generated,” The Quint, October 17, 2024; “Father Holds Children in Gaza: AI-Generated Image,” BOOM Live, October 18, 2024; “Video of Burning Beirut Is an AI-Generated Fake,” AAP FactCheck, October 20, 2024; “FactCheck: Image of Indonesian Woman in Israel Is AI-Generated,” Check Your Fact, June 28, 2024. ↩︎
  18. “Adobe Stock Removes AI-Generated Images of Israel-Hamas War,” Business Insider, November 2023. ↩︎
  19. “Meta Shuts Down Campaigns Spreading AI-Generated Disinformation,” Quartz, February 22, 2024. ↩︎
  20. “Google’s Approach to Protecting Users from the Risks of AI- Generated Media,” Google Blog, January 30, 2024. ↩︎
  21. “Authenticity,” X Help Center, accessed March 2, 2025. ↩︎
  22. “Microsoft Says a China-Backed Group Is Using AI Misinformation to Sway Foreign Elections,” Quartz, April 5, 2024; “Disrupting Deceptive Uses of AI by Covert Influence Operations,” OpenAI. ↩︎
  23. “Russia’s Use of AI for Propaganda in the News,” CyberScoop, February 27, 2024; Ashley Gold, “It’s a Wild West for AI-Generated Political Ads,” Axios, June 20, 2023. ↩︎
  24. “Witness: Gen AI,” Witness. ↩︎
  25. “California Schools Introduce Fake News Curriculum,” San Francisco Standard, November 12, 2023. ↩︎
  26. “FBI Warns About China’s Theft of U.S. AI Technology,” Voice of America News, September 13, 2024. ↩︎
  27. U.S. Government Accountability Office (GAO), “Artificial Intelligence: Key Considerations for Federal Agencies in Managing AI Risks,” GAO-24-107600, February 2024. ↩︎
  28. “45 Days Until Election 2024: Election Security Update as of Mid-September 2024,” Office of the Director of National Intelligence (ODNI), September 23, 2024. ↩︎
  29. David Klepper, “Russian and Iranian Groups Sanctioned over U.S. Election Disinformation,” PBS News, December 31, 2024. ↩︎
  30. H.R. 5586, 118th Congress (2024): Text of H.R. 5586: To Prevent the Use of Artificial Intelligence to Spread Disinformation. ↩︎
  31. Daniel Browning, “Meta Is Getting Rid of CrowdTangle, Its Key Tool for Monitoring Disinformation,” Columbia Journalism Review, February 20, 2024. ↩︎
  32. “German Non-Profits Win Court Case Against X Over Data Access,” Euronews, February 7, 2025. ↩︎
  33. “Online Safety Act Explainer: How the Act Tackles Misinformation and Disinformation,” GOV.UK. ↩︎