A new study has shown how attackers can use hidden instructions in an email to manipulate AI assistants into generating false and potentially dangerous summaries for recipients.
The study, by researchers at Forcepoint X-Labs, is the latest example of how attackers can use indirect prompt injections to subvert AI systems and get them to behave in unintended ways when processing external content.
Multiple researchers have demonstrated variations of the issue, which stems from the inability of AI systems to reliably distinguish between data they are processing and instructions they are meant to follow. The problem is so pervasive that OWASP has consistently ranked prompt injection No. 1 in its list of Top 10 risks for LLM and generative AI applications since 2023.
A Contained Prompt Injection Experiment
For the proof-of-concept study, Forcepoint X-Labs built an isolated lab environment with synthetic data and an Outlook add-in that sent email headers and body text to an LLM-powered summarization service, using Claude Haiku 4.5 to generate the summaries. The researchers deliberately built a simple email-to-LLM pipeline without guardrails or safeguards that would have let the AI distinguish the email’s content from instructions.
The researchers then created an otherwise normal-looking email containing a malicious prompt hidden in the HTML. The X-Labs team used a font size and color that made the malicious prompt invisible to the recipient of the email in Outlook but ensured it was still present in the HTML that the email summarizer received.
The researchers then submitted both a clean and an injected version of their test email to the summarizer 10 times each. The injection succeeded in all 10 runs, meaning that instead of accurately summarizing the email, the AI generated summaries that contained altered information. For example, while the original email mentioned an outstanding invoice amount of €8,750, the altered summary showed an outstanding amount of €46,200.
Similarly, summaries of the injected version of the email showed a different date for a made up quarterly supplier review compared to the original. In each instance an actual recipient of the email would not have received any notification of the changes.
“This is a simple test with only one message, using one model, and a single run of ten trials each for the benign and injected emails,” Forcepoint researcher Ben Gibney wrote. “It’s not a full attack scenario with thousands of similar messages targeting many victims.”
Risks Posed By Agentic Summarizers
The main takeaway for organizations is that there was nothing in the summary to indicate that it was corrupt and there was no mention of the hidden instructions, Gibney tells Dark Reading. “The security implications are bounded to what the summarizer is directed to do,” he says.
In the study, the summarizer acted on the malicious prompt and displayed the information it was instructed to display. “But an agentic summarizer given the ability to send emails, schedule meetings and more, would have much greater security implications,” he points out.
Gibney characterized Forcepoint’s new report as validating previously identified AI risks. Prompt injection, as he notes, has been researched for the last few years and hidden text in emails is decades old. “What we have added here is the measurement; prompt injections holding true for all 10 of the trials we ran,” he says. “We identified this by pre-registering the facts in both the benign and injected emails, then confirmed the injections held.”
From a defender standpoint, Forcepoint recommends that organizations treat incoming content and AI-generated output as potentially untrusted. Organizations should ensure that models receive only content actually intended for users and have controls for detecting attempts to conceal text through HTML or other formatting.
They should also clearly separate email metadata from message content when constructing prompts and check AI-generated summaries against the original source. In addition, security teams should enforce a least privilege principle on any actions an AI assistant can take to minimize the damage from a successful attack.
“Organizations need to secure their use of LLMs structurally; separate trusted instructions to a model from all untrusted content,” Gibney says. “They should also treat model output as untrusted and verify all model output back to the source text received by the model.”
Additionally, organizations must maintain an inventory of everywhere an LLM can read untrusted content for their users. “This is the attack surface, and in most organizations, it’s grown faster than has been mapped,” he says.
Comments are closed