Attackers are poisoning popular AI chatbots with misinformation and phishing links so they serve up this content to users as facts or trusted websites. The content-manipulation campaign is targeting popular brands across various industries, creating reputational damage and other ramifications for the organizations.

Researchers from Vigilance Security identified the campaign, dubbed “Dark Sourcery,” which essentially uses social engineering on OpenAI’s ChatGPT, Google Gemini, and Google AI Overview, according to research published today.

“Attackers are flooding the web with carefully optimized posts, PDFs, reviews, and fake support pages, to trick AI into presenting fraudulent phone numbers, email addresses, and login pages,” Ariel Simon, vice president of research at Vigilance, wrote in the post.

Vigilance researchers identified tens of thousands of malicious pages containing fake support numbers, email addresses, login pages, software-update information, and other fraudulent details as part of the attackers’ aim to manipulate the content delivered by AI chatbots, he said.

Related:Malicious Linux Implants Mimic Asian Mail Security Products

Manipulating ChatBot Results

To get the AI chatbots to deliver the malicious content and links, attackers use search engine optimization (SEO) and other content-distribution techniques to improve the chances that the chatbots will retrieve them and use them as part of their “conversations” with users.

“We suspect that high authority domains (universities, government) combined with public opinion sources (social media, forums, user reviews) achieved high success in poisoning AI answers,” Simon tells Dark Reading. “Depending on the model, the AI looks at the source and trusts the content more if it’s coming from an authoritative entity.”

Attackers have so far swept up at least 374 companies — including Fortune 100 organizations, major airlines, banks, travel companies, and software providers — in the campaign. Popular brands affected include Delta, Lufthansa, Qatar Airways, Chase, Bank of America, Airbnb, and TripAdvisor.

A Dangerous AI Attack

The attack is similar to SEO poisoning, as it aims to trick AI into serving up results much like how that style of attacks ranks search results to appear higher than others. But it’s different in a fundamental way and takes social engineering to a new level, according to Simon.

“Here, the malicious information becomes part of the AI’s answer itself,” he says. “A fraudulent support number is presented as helpful guidance, potentially without the user ever looking at the page that supplied it.”

Related:Russia’s Star Blizzard Ditches ClickFix to Widen Phishing Net

The attack also differs from prompt injection attacks that commonly exploit AI chatbots in that it doesn’t involve the attacker giving AI explicit instructions or prompts, which might be thwarted by AI defenses. “Here, the disinformation displayed as facts is making the AI answer the user with the false information, without ever following any instruction/prompt by the attacker, essentially bypassing all defenses,” Simon tells Dark Reading.

While most Internet users have been warned about clicking on suspicious links or falling for phishing lures, they tend to have a different view of AI as a “know-it-all” entity that most trust blindly. This is why the attack “presents a new and dangerous risk,” he says.

Indeed, a study by Exploding Topics cited by Simon and published in August found that 91% of people using AI chatbots don’t verify the answers provided. “That means that if attackers control AI, and we blindly trust it, they essentially control us,” he wrote in the post. Indeed, many experts have cited blind trust of AI as one of the technology’s main threats to overall user and organizational security.

Ramifications for Stakeholders

The campaign remains ongoing and has ramifications not only for users, but also for the brands being attacked and organizations using AI chatbots and agents within their networks. “We believe the real impact of these scams is happening now, and we are still unable to fully comprehend or measure the full impact yet,” Simon adds.

Related:‘NeedyMantis’ Provides Long-Term Access to Compromised Networks

For users, the most obvious best practice is to verify results delivered by AI and not trust chatbots as if they are infallible, he says. For the brands whose content is being manipulated to deliver malicious links or misinformation, he recommends that security teams take customer complaints about scams seriously and monitor the AI answers that customers receive, along with the sources those answers cite.

“Prioritize support, account recovery, refunds, payments, and software downloads,” Simon says. “Compare returned phone numbers, email addresses, and URLs against verified company records, then investigate unfamiliar details and repeated indicators across unrelated websites.”

Indeed, he says that Vigilance already has seen “a few occurrences” of people complaining about being scammed into giving up payment details or payment card data by fraudulent phone numbers that were provided in chatbot answers. The researchers even called some of the fake numbers and were met with a representative who “wished to help us ‘move our flight,’ or ‘unlock our bank account'” and asked for credit card details to do so, Simon tells us.

Finally, for organizations using AI chatbots and agents in their workflows, Simon recommends they monitor these agents at runtime to verify every source and piece of content they rely on and that enters their context to protect employees from any erroneous answers. They also should analyze the AI agent’s delivered content and behavior, and verify any critical details, including phone numbers, links, software packages, command lines, and other information.





Source link

#

Comments are closed