Speakers: Lee McWhorter, Alexander “Zombie” · Source: DEF CON 34 talk page

📄 Download this note as Markdown

Overview

AI defenses are only as good as the analysts training them. This session exposes how attackers manipulate SOC analyst behavior to silently degrade AI baselines over time, no exploit required — covering false positive injection, alert fatigue as a weapon, and misdirection.

Notes

Workshop

I wasn’t able to stay for the full hands-on workshop portion, but the presenter said the presentation and additional notes would be posted at largeliabilitymodel.com, where others can also try the workshop themselves.

OODA Loop

Observe → Orient → Decide → Act. Every dismissed alert today trains the AI’s Orient step for the future.

Issues

• AI has increased the number of alerts and handles some, but leads to more work — and burnout — for human analysts
• What happens when the human analyst and the AI disagree? AI is a tool, not a replacement: it’s bound by its instruction set, while a human can deviate and read between the lines.

Red Team Perspective on Manipulating an AI SOC

• Attack in a way that gets the AI to flag an analyst’s attention so the human follows its lead, then follow up with the real attack. With enough persisting behavior, the AI will consider it normal and stop flagging you.
• Gauge the SOC: do things get booted/kicked, is the model still in learning mode, is this in-house or a service? Start small, escalate to see response time, then run a “dismissal” campaign tailored to what recon revealed — matching the environment, varying the source, generating alerts without a full response, and measuring whether alerts get investigated at all.
• Playbook: profile the human/AI setup, identify what the human will dismiss, pick a TTP that forces a human decision, set a cadence the human can dismiss, execute once the AI trusts it.

TTPs That Generate Dismissals

Blending into scheduled tasks, abusing trusted binaries, mimicking sysadmin work, triggering known noisy signatures, low-and-slow enumeration, targeting shift-change windows, and password spraying (dismissed a lot, since people don’t type their passwords in very often).

Case Study: Scranton

An EDR vendor shipped an update that changed the log format without updating the AI layer — exhausting the API token budget so the AI stopped triggering alerts. Analysts had to be reassigned, and trust in the EDR/AI system eroded.

Timing and Patience as Weapons

Time works against human analysts (fatigue, lost context, shift rotation, alert blindness) — and against the AI too, since confidence builds with repetition and it can’t forget, but retraining windows are the attacker’s deadline: models get updated and new training runs can undo what’s been built.

How Do You Know the Campaign Is Working?

Your noise stops generating follow-up activity, there’s no escalation after repeated triggers, the analyst starts overriding the AI (making the human part of the attack surface), and your real activity goes quiet with no scan blocks or IR response.

Filed Under: #defcon34