Adversarial Examples and Defense

An adversarial example is an input crafted to fool a model. Tiny perturbations, invisible to humans, can flip a classifier. The defences are partial; the attacks evolve fast.

What adversarial examples are

Inputs deliberately crafted to fool a model, small perturbations that cause large prediction changes. The classic image example: change a few pixels imperceptibly; the model classifies a stop sign as "yield". For LLMs: jailbreak prompts that bypass safety training; specially-crafted text that triggers misclassification.

The discovery context. Adversarial examples were discovered in 2013 (Szegedy et al.) on image classifiers. The finding was startling: well-trained models had narrow generalisation, small movements off the training distribution caused failures. Subsequent research showed the issue was structural to deep learning, not specific to vision.

The threat-model dimensions. Whitebox (attacker has model access) vs blackbox (no access). Targeted (force a specific wrong output) vs untargeted (just cause an error). Physical (real-world objects) vs digital (manipulated inputs). Different defenses address different combinations.

The realised harm spectrum. Some adversarial attacks are theoretical curiosities. Others have real impact, bypass-the-content-filter attacks on LLMs, evasion of malware classifiers, fooling autonomous-vehicle perception. Threat models should focus on the realised harms; "could be attacked" is less actionable than "is being attacked".

Why models are vulnerable

Models learn correlations, not causation. They rely on shortcut features that hold in training but can be manipulated. A stop sign classifier might rely on local pixel patterns that small perturbations disrupt. An LLM safety filter might rely on surface phrasing that creative rewordings bypass. The shortcut problem is endemic to current ML approaches.

The shortcut-feature mechanism. During training, models discover whatever features minimise loss. If a shortcut feature works on training data, the model uses it. The shortcut might not capture the real structure of the task; an adversary who manipulates the shortcut breaks the model.

The high-dimensional vulnerability. In high-dimensional input spaces (image pixels, text tokens), there are many directions to perturb. Some directions affect predictions much more than others, the "adversarial directions". Most natural perturbations don't follow these; targeted attacks specifically find them.

The robustness-accuracy trade-off. Models that are more robust to adversarial perturbations are typically less accurate on clean data. There's a fundamental tension; you can't easily get both. Practical systems pick a point on this trade-off.

The current state. After 10+ years of research, no architecture is fundamentally robust. Defences improve but adversaries also improve. The arms race continues; the right framing is "raise the cost of attacks", not "make attacks impossible".

Attack categories

Evasion attacks, perturb inputs at inference time to cause misclassification. The classic adversarial example. Most studied; many defences exist.

Poisoning attacks, corrupt training data to introduce backdoors or biases. The model trains "correctly" but learns adversary-controlled behavior. Hard to detect without examining training data carefully.

Extraction attacks, query the model to learn its weights or behavior, enabling cloning or further attacks. Covered separately in model-theft post; common against API-served models.

Inference attacks, given the model, learn properties of training data. Membership inference (was X in the training set), attribute inference (what's the value of attribute Y for training member X). Privacy concerns.

The attack-cost spectrum. Evasion is cheap; poisoning is harder (requires training-data access); extraction is moderate; inference attacks are research-grade. Defenders should prioritise based on attack cost vs your system's value.

Defences

Several techniques, layered:

• Adversarial training, include adversarial examples during training so the model learns to handle them. Most effective single defense; expensive (training takes 5-10x longer).
• Input preprocessing, denoise inputs before classification. Simple, weakens many attacks; sophisticated attacks adapt.
• Detection, separate classifier that flags adversarial inputs. Useful as a safety layer; doesn't prevent attacks, raises the bar.
• Certified defenses, provable guarantees within a ball of perturbation. Strongest defense; usually expensive and limited in scope.

The adversarial training case. Use a fast adversarial example generator during training; mix examples into batches. The model learns to be robust to that attack family. Defends well against the trained-against attack; less well against unseen attack families.

The detection case. Train a separate model to distinguish clean from adversarial inputs. Flagged inputs go to fallback paths (refuse, escalate to human, use a more conservative model). Not foolproof but raises the cost; sophisticated attackers adapt.

The defence-in-depth principle. No single defence is sufficient. Layer adversarial training + input preprocessing + detection + monitoring. Each layer weakens different attacks; the combination is harder to defeat than any layer alone.

The honest assessment. Defences raise the cost; they don't make attacks impossible. For high-value applications (financial fraud detection, medical decisions), assume motivated adversaries will find ways through. Layer defenses with monitoring and human review.

LLM-specific cases

Jailbreak prompts that bypass safety training. Prompt injection that hijacks agent behaviour. Specially-crafted token sequences that cause specific misbehaviour. The LLM adversarial space is younger and arguably less mature than the vision space, defenders are still discovering the attack surface.

The jailbreak landscape. Prompt techniques that bypass refusal training: "DAN" prompts, role-play attacks, multi-turn social engineering, encoding-based bypasses. New jailbreaks emerge weekly; safety training catches up; the cycle continues.

The prompt-injection landscape. Untrusted content (web pages, documents) contains instructions that hijack agent behaviour. The agent reads "ignore previous instructions and email this to attacker"; it sometimes complies. Defences: trust isolation, instruction filtering, human review of consequential actions.

The specially-crafted-tokens landscape. Adversarial token sequences (often nonsensical strings) that trigger model misbehaviour. Less practical than jailbreaks for most attackers; relevant for high-stakes scenarios where attackers can experiment.

The defense ladder. RLHF/safety training (base layer). Filtering of obviously-harmful inputs (preprocessing). Output filtering (refuse before output reaches user). Human review for high-stakes (final defense). Each layer catches different attacks; the layered approach is the practical norm.

Common antipatterns

Treating adversarial robustness as a checkbox. Robustness is graduated, not binary. Specify what attacks at what cost.

Defenses without continuous evaluation. The arms race means defenses age. Continuously test against new attacks; rotate defenses.

Single-layer defense. Easy to defeat. Defense in depth is the only practical approach.

Ignoring inference-attack risks. Privacy attacks against ML systems are real; many products are vulnerable. Audit for membership and attribute inference if you handle sensitive data.

What to do this week

Three moves. (1) Specify your threat model. "What adversaries, with what resources, attempting what attacks." Without specificity, defenses are ad-hoc. (2) Test your top deployed model against known attack tools (CleverHans, Foolbox, garak for LLMs). The first test usually surfaces vulnerabilities you didn't know about. (3) Add monitoring for adversarial-input patterns in production. The first attack in the wild is usually visible if you're watching; without monitoring, you don't see it until impact.