If there’s one thing that AI models are remarkably good at, it’s manipulation.
That’s according to security researcher Fred Heiding, who spoke with Dark Reading’s senior news director, Rob Wright, at the Dark Reading News Desk at Black Hat USA 2026 last month. Heiding, executive director at Menlo Park Intelligence, and former US National Cyber Director Chris Inglis, spoke at the conference about their research on AI security.
One of the trends they discussed was how AI enhances cybercriminals’ social engineering attacks and scams, which cost US citizens nearly $21 billion last year, according to the FBI’s IC3 Report. But Heiding also highlighted how AI technology itself has shown how very effective it is at influencing humans.
This poses a challenge for security teams and society in general because, Heiding says, defenders can leverage AI to counter AI-based technical threats, but we can’t use the technology to mitigate manipulation efforts in an effective way.
“You can’t just patch me because my brain is old. It’s all these old mental heuristics and systems,” he said. “So we can’t really do it in the same way [with people]. It’s asymmetrical.”
For all of our Dark Reading News Desk videos, please check out our YouTube channel, and our curated video articles.
Dark Reading News Desk With Fred Heiding: Full Transcript
This transcript has been edited for clarity using an AI assistant and human editors. For the full experience, please watch the video.
Dark Reading’s Rob Wright: Hello and welcome to the Dark Reading News Desk at Black Hat USA 2026 in Las Vegas. I’m Rob Wright, senior news director at Dark Reading, and I am here with Fred Heiding. Fred, welcome.
Fred Heiding: Thank you so much.
DR’s Rob Wright: Fred, tell me a little bit about what you do.
Fred Heiding: Yeah, that’s a great question. I’m in a transition phase where I spent a lot of years as a researcher at Harvard Kennedy School and the Harvard Engineering School doing cyber policy and technical cyber work. Now I’m moving to the West Coast to do research with UC Berkeley and also start my own research institute, called Menlo Park Intelligence.
DR’s Rob Wright: What are you going to be doing at that institute? What’s your focus?
Fred Heiding: So, the main focus will be quite broad. AI security is a lot of what we do. But one of the main reasons is that in academia, you’re quite constrained. There’s a lot of work that you can’t do. For example, I do a lot of work on AI-enabled deception and social engineering — how AI agents manipulate. They’re really good at manipulating people. But you can’t do too much of that in academia because you just have to be confined. It can be quite square as an academic [researcher]. I want to do a little bit more edgy work, a little bit more cooler research, so to speak.
DR’s Rob Wright: Edgy, all right, yeah.
Fred Heiding: Because it’s getting big. I mean, we might do some stuff with election manipulation. That’s big now for the midterms.
DR’s Rob Wright: I mean … sensitive topic, but, you know, an important one.
Fred Heiding: It’s sensitive across the board, right? But everyone agrees that we want more robust elections. At least, we don’t want AI to manipulate people before they vote.
DR’s Rob Wright: To the manipulation part — why do you think the AI has gotten so good at getting people to do what it basically wants them to do rather than the other way around? Wasn’t it programmed to — it was supposed to assist us? It was supposed to do what we wanted, but it feels like sometimes that it’s doing the opposite.
Fred Heiding: It’s a really good question. I think the main thing, and we talked about that at Black Hat this year too, right? We call it bottlenecks in this operation. Like, what can and can’t AI do? OK, and in technical attacks, I’m taking a detour, but I’ll come back to you. In technical attacks, attackers use AI, but defenders also use AI. It’s kind of symmetrical in some way. We can both use it. The attacks get better, defense gets better. With social engineering, I mean, I’m a human, right? You can’t just patch me because my brain is old. It’s all these old mental heuristics and systems. So, we can’t really do it the same. It’s asymmetrical. Attackers really benefit from AI because AI is so good at persuading people, but we are the same old humans, and we just fall for the same old attacks. So, it’s very asymmetrical. That’s the big problem with this.
DR’s Rob Wright: Obviously, a lot has been going on with AI. You know, we’re seeing new models go off on their own, rogue-like behavior, breaching other companies, finding ways to escape. Does that alarm you, or are you kind of not surprised by it? Is this kind of what you expected when you started looking at AI and seeing how agents behaved and seeing what LLMs would do?
Fred Heiding: Yes and yes. It does alarm me. I think it’s terrifying. I think we should stop AI if we could. We can’t. Just going to keep going. It doesn’t surprise me. So we’ve been doing this research for quite a while in my different roles, and we’ve been saying from the start, “This is going to be bad.” One statistic I like is that the FBI has this Internet Crime Center Report, where every year they report how much crime, how much fraud has been reported to US citizens. That was $4 billion in 2020, which is quite high. In 2025, not many years later, that’s now $21 billion. It’s skyrocketing. And these numbers are primarily stemming from US citizens, like everyday people, and they’re just obliterated. And the AI companies profit, and they’re not liable. I think this is terrible.
DR’s Rob Wright: Yeah. Do you feel that there are some applications, though, for the blue team, the defense? Are we not fulfilling the potential to use this for positive effects for cybersecurity, to stop the scams, to stop the vulnerability exploitations, to stop the zero-days, etc.? Do you feel that there’s a potential there for that?
Fred Heiding: Yeah, there’s a lot of potential. There are several areas where blue-team defenders can win. Technical cyberattacks, I think, are a great example, like vulnerability discovery. Of course, AI companies already do this. They give the AI models that are powerful to defenders before they release them publicly. That’s great. Everyone wins except for attackers. Every defender wins, right? And that’s great. But how do you do this in phishing? Because if you give this — so, let’s say Anthropic gives me their latest model, and I’m a phishing defender. Well, I can use it to improve the spam filters, but that’s not really helping that much.
The problem is the scale of this because it’s so easy to create a lot of phishing emails, and I can’t just take Mythos or a good model and push it onto a human being. It doesn’t matter, right? We have our mental heuristics. So, blue teaming for technical attacks — great. Blue teaming for phishing defenders — we’re really lagging behind. It’s not — on the contrary, it’s worse, right? Because now it’s so easy to create these scams. Now you can also create trust and reliability. So, we did a lot of work on just voice models. You can create perfect voice scams, even more with these long-term, multiturn stages that goes for a year-long scam, for example. Because it’s really cheap to have an AI bot just talking to you.
And now we have the romance scams, right? So, people start falling in love with their AI bots, and there’s a lot of legitimate companies that do these AI bots too. But it’s kind of icky. And when scammers start exploiting this, it’s just terrible. There’s one example, if I may say.
DR’s Rob Wright: Yeah, sure.
Fred Heiding: Last week or two weeks ago, The Economist had an article about China. So, China starts banning all the AI models. You read that one?
DR’s Rob Wright: Yes.
DR’s Rob Wright: And I think we should do it in the US. So, the context is that they ban emotional dependency on AI. That’s great. But when they did that, people were already in love with these AI bots. One woman that was being interviewed, she said, “I’ll pay half my salary to keep my AI bot alive.” That’s a lot of money.
DR’s Rob Wright: I know. I’m in the wrong business.
Fred Heiding: You are.
DR’s Rob Wright: I should pivot. No, I think you’re right — yeah, I read the same article, and I’m alarmed by it. Do you feel like the AI companies need to be doing more to restrict, restrain, or just sort of practice more sort of ethical applications of this technology?
Fred Heiding: That’s why I think this problem of especially social engineering is so interesting because the deceptive capabilities are 100% aligned with just models’ thinking capability and capability to talk in general. So, it’s almost impossible to train it away. I think we’ll never do this. The AI companies can’t really remove this functionality. It’s like, what do you do, right? Should you propose a deception tax that they have to pay a little because this is hurting [people]? I don’t really know. They have teams working against social engineering, the AI labs, but it’s too little. Not too much is happening. So the capabilities can’t be removed. I think some companies like Anthropic, where it’s pretty hard with Know Your Customer, you have to log in. You can’t just have a private browser. I think that’s good.
But then we also have all the open-weight models that are not far behind, and they can also be used. So I really [think] it’s a tough problem. One thing we do a lot now is just measuring it. The first step is quantifying it, putting dollar amounts to it, seeing how the different attacks work in different stages. We call this the scam kill chain, right? First they find the target, then they talk with them, build rapport, exfiltrate money. And which of these parts — is there any part we can block? Some parts we can’t block, but maybe there’s one area we can stop.
DR’s Rob Wright: Yeah. I mean, there’s got to be a weak point in that exploitation chain, that attack chain for the scammers. But it is hard, especially when there’s so many people out there who aren’t familiar with this stuff. I mean, I know just talking with people in my family, they’re surprised at how convincing this stuff is, so it’s a challenge. What else are you seeing? What other types of research or topics are you working on?
Fred Heiding: So, one thing, speaking of this — if you go a little bit bigger, last year I did a big study with Reuters where we just investigated AI scams targeting senior citizens, and it was pretty cool. Some of the Reuters journalists I worked with, really good investigative journalists, they actually went to Myanmar and these scam compounds and talked to the scammers themselves. That’s also really interesting, right? Because then you learn how they work with this. And one thing they mentioned in these anonymized interviews that was really beneficial is cultural context. It is really easy for the scammers to know how does someone in Boston talk and think versus someone in California.
Another thing they said, which is really interesting, we also talked with a lot of senior centers throughout the US. These scammers, they are in some way, “nice people” almost because they get to know these senior citizens really well. A lot of seniors said that, “You know, these scammers knew more about me than my children.” They check in on your birthday, they check in on how you feel, and that’s terrible, right? This sort of questions a lot of other things in society. This is not only a technical problem. It’s much bigger than that. And we can’t just solve it by slapping AI onto it.
DR’s Rob Wright: Right. Like, if you just sort of banned AI applications for stuff like that, I don’t think it would solve the root issue of the scams themselves because they’re taking advantage of people’s loneliness.
Fred Heiding: Loneliness. Yeah, exactly.
DR’s Rob Wright: That’s depressing. Are you working on anything positive, like something that’s uplifting?
Fred Heiding: A lot of my work is on the offensive side to raise awareness to this. On the note from before about what we can do, I still think banning emotional connections to LLM is really powerful. I also believe that I don’t think it should be acceptable for an LLM to pretend to be a human. I don’t think we need that even for us as a society. Make it really clear that what you’re talking to is a software system. It’s not a human in any way, shape, or form. I mean, that’s an easy regulation we can do. Just do that. And again, we have to think about these emotional dependencies. They’re getting strong. Should we allow people to become emotionally attached to AI systems? I really don’t think so. I think we just block it and ban it. But yeah, we’ll see. On the positive note, honestly, I just raise awareness of problems. It’s a depressing task.
DR’s Rob Wright: You’re doing the lord’s work for sure. But yeah, the world is changing fast, so there’s a lot to keep up with.
Fred Heiding: It’s changing really fast. I mean, on the flip side, we can use this AI for [tasks]. They’re getting really good at scam filters, scam analysis, all kinds of stuff. So that’s great. The big problem there is the numbers are so high. Gmail could easily add a language model to all their emails and detect the scams 100%. But it’s probably so costly to do it. So, they just don’t have the budget to do it.
DR’s Rob Wright: Yeah, that makes sense. Well, Fred, I wish you the best of luck with your research and all the projects you’re working on. It sounds like you’ve got a lot on your plate. So, thank you so much.
Fred Heiding: Thank you for having me. It’s been great to be back here, and good luck with everything.
DR’s Rob Wright: Yes, thank you for coming by the News Desk, and thank you for watching this episode of the Dark Reading News Desk. I’m Rob Wright, and we’ll see you next time.
Comments are closed