Evasion attack: when someone games your AI without breaking it

An insurance broker found a pattern: shift the applicant's age by 2 years and income by 5%, and the AI always approves — even when the real numbers get rejected. He did it for dozens of clients, systematically.

What an evasion attack is

The AI didn't "break." The model wasn't poisoned in training. The broker simply probed the live system until he found which input combinations crossed the approval threshold — then exploited it at scale. It's an attack where the AI system itself is the target.

The four mitigations

• Data validation — cross-check against authoritative sources (tax, social security). Mismatch → flag.
• Liability allocation — the provider–deployer contract must define who's responsible when the system is gamed. A governance gap, not a technical one.
• Anomaly detection — one borderline application is coincidence. Forty from the same broker in a week is a pattern.
• Human-in-the-loop for edge cases — near the decision boundary, a human reviews. The model doesn't decide alone in the grey zone.

The audit question isn't "is the model accurate?" It's "what happens when someone tries to find where it breaks?"