🤖 AI-Generated Phishing Emails: How to Detect Deepfake Messages

AI-Generated Phishing Emails: How to Detect Deepfake Messages

Traditional phishing detection relied on broken English, spelling errors, and generic greetings — red flags that even casual users could spot. AI-generated phishing emails eliminate all of these telltale signs. A May 2026 study by the Anti-Phishing Working Group found that AI-written phishing emails bypass traditional spam filters 87% of the time, compared to just 52% for human-written phishing emails. The implications are stark: every organisation that still relies on "spot the typo" training is actively vulnerable.

The problem is compounded by the sheer volume of AI-generated messages. With large language models, attackers can generate thousands of unique, personalised phishing emails in minutes — each one tailored to the recipient's role, company, and recent activity. This is not a theoretical future threat. It is happening now, and the data shows it is working better than traditional phishing ever did.

How Attackers Use AI to Craft Phishing Emails

Attackers leverage LLMs (Large Language Models) like ChatGPT, Claude, Gemini, and open-source models such as Llama and Mistral to automate every stage of phishing campaign creation. The workflow typically follows three steps:

Step 1 — Reconnaissance: The attacker feeds the LLM scraped data about the target organisation — employee names, recent projects, annual reports, social media posts, and press releases. This context enables the AI to generate messages that reference real internal initiatives and use authentic company terminology.

Step 2 — Drafting: The LLM generates the email body, subject line, and sender name. Attackers can instruct the model to mimic a specific person's writing style by providing sample emails or LinkedIn messages from the target. The result is an email that reads exactly like the person being impersonated.

Step 3 — Refinement and A/B Testing: Some advanced campaigns use AI to automatically A/B test subject lines and opening sentences, measuring which variants generate the most clicks. The AI then iterates on the highest-performing versions, producing increasingly effective phishing lures with each cycle.

This automation means that a single attacker — or small criminal group — can now operate at a scale previously only possible for nation-state actors. The cost is minimal: GPT-4o or Claude can generate hundreds of convincing phishing emails for under $5 in API credits.

What AI-Generated Phishing Looks Like in Practice

AI-crafted phishing emails share several common characteristics that distinguish them from older, clumsier attacks:

• Flawless grammar and spelling — no more "Dear Costumer" or urgent subject lines with random capitalisation. AI-written emails are grammatically perfect in multiple languages.
• Contextual personalisation — the email references your actual job title, department, recent projects, or even a specific vendor relationship, all sourced from public data.
• Natural conversational tone — the writing flows naturally, uses appropriate transitions, and avoids the robotic or stilted language that marked earlier automated phishing.
• Thread continuation attacks — attackers can hijack an existing email thread and use AI to generate a reply that perfectly matches the conversation's tone and context, making detection nearly impossible without sender verification.
• Multi-language capability — AI generates native-quality phishing emails in dozens of languages, dramatically expanding the geographic reach of any campaign.

In one documented 2026 case, attackers used Claude to impersonate a CFO and carry on a 12-email conversation with the accounts payable department, ultimately authorising a fraudulent $47,000 wire transfer. The conversation was flagged only because a sharp-eyed employee noticed the CFO's signature font was slightly different — not because of anything the AI wrote.

Detection Methods That Still Work Against AI Phishing

While content-based detection is increasingly unreliable, several technical and behavioural methods remain effective:

• Sender domain inspection — AI can't fake the actual SMTP envelope. Always check the return-path and received headers, not just the display name or From address. Look for lookalike domains (rnicrosoft.com instead of microsoft.com).
• DMARC/DKIM/SPF authentication — properly configured email authentication blocks most domain spoofing. Organisations should enforce a strict DMARC policy (p=reject) and monitor alignment failures.
• Contextual anomaly detection — does the request match the sender's normal behaviour? Would your CEO really ask you to buy gift cards or transfer funds outside of normal procurement processes?
• Secondary verification protocol — verify any unexpected request involving payments, credentials, or sensitive data through a completely different channel. A phone call to a known number, a direct message on Slack or Teams, or an in-person conversation.
• AI-powered email security tools — ironically, the best defence against AI-generated phishing is AI-based detection. Modern security solutions analyse sender behaviour patterns, communication graphs, and metadata anomalies rather than just scanning content for malicious keywords.

Deepfake Voice (Vishing) Adds Another Layer of Risk

AI doesn't just write better phishing emails — it makes convincing phone calls too. Deepfake voice cloning technology has advanced to the point where 30-60 seconds of source audio is sufficient to create a convincing voice clone. Attackers harvest this audio from:

• Voicemail greetings on corporate phone systems
• Conference call recordings available through compromised accounts
• Social media videos — LinkedIn, YouTube, TikTok, Instagram
• Podcast appearances and public speaking engagements
• Previous legitimate phone calls that were recorded

In the most sophisticated vishing attacks, the attacker calls the victim posing as their CEO or a trusted vendor contact. Because the voice is convincing and the attacker has contextual information (scraped from internal communications or LinkedIn), the victim is far more likely to comply with urgent requests. The 2025 UK case where fraudsters cloned a CEO's voice to authorise a £220,000 transfer is now part of a growing pattern, not an isolated incident.

Some advanced attackers combine AI-generated phishing emails with follow-up voice calls — a technique known as "multi-channel phishing." The email establishes urgency, and the voice call provides the final nudge. Because the two channels appear to confirm each other, victims perceive the request as doubly legitimate.

Building AI-Aware Defences for Your Organisation

Security awareness programs must be redesigned for the AI era. Grammar-spotting is obsolete. Here is what should replace it:

• Behaviour-based verification training — teach employees to question the request, not the writing. Every request for money, credentials, or sensitive data must be verified through a separate channel, regardless of how legitimate the email looks.
• Tabletop exercises with AI-generated scenarios — run simulated phishing campaigns that use AI-crafted emails so employees learn to recognise the new threat landscape.
• Technical controls — deploy DMARC enforcement, AI-powered email security, and browser isolation for high-risk transactions. Implement strict vendor payment verification processes that require multiple approvals outside of email.
• Password manager integration — encourage employees to use password managers that auto-fill credentials only on recognised websites. This prevents credential harvesting even if a phishing link is clicked.

What to Do If You Receive a Suspicious AI-Generated Email

If you suspect an email was generated by AI for phishing purposes, follow these steps:

1. Do not reply, click links, or open attachments. Engaging with the attacker confirms your email address is active.
2. Inspect the full email headers. Look for mismatches between the display name and the actual sending domain.
3. Verify through a separate channel. Call the person allegedly sending the email using a known phone number — not one from the suspicious email.
4. Report to your IT/security team if at work, or forward to reportphishing@apwg.org if at home.
5. Enable multi-factor authentication on all accounts. Even if credentials are stolen, MFA can block account takeover if the attacker doesn't have access to your second factor.

The threat landscape has fundamentally shifted. AI-generated phishing is no longer a novelty — it is the new baseline. The organisations and individuals who adapt their detection strategies from language-based to behaviour-based will be the ones who avoid becoming statistics.