Imagine, if you would, that a fraud team spots a new attack on Monday. The rule that stops it reaches production a week later, sometimes two, and the attacker has spent that whole stretch onboarding accounts or draining them. The standard diagnosis for those lost days is the engineering handoff: the analyst who designs the fix hands it to a separate team that owns production, the fix sits in a queue behind other work, and every handoff in the chain adds time.
We believed that diagnosis, and it's partly right, so we spent years building the handoff out of our platform. Analysts write and deploy rules straight from the dashboard without filing a ticket. Model deployments run through an automated pipeline, and testing is built into both paths, so a change that used to wait days for a deploy window can go live in minutes.
That work paid off, and I'd make the same investment again. Deploys that took days now take minutes, and it's the part of the stack customers feel first. What the work also did was expose the constraint behind it: with the handoff gone, the time from spotting an attack to running a fix in production was still measured in days, which meant most of those days were never in the handoff to begin with. We could only see that once the known bottleneck was out of the way.
Where the time goes during the fraud reaction cycle
It took us a while to see it clearly, partly because the slow part sits between two functions rather than inside either one. An investigation produces a theory about an attack. Testing turns that theory into a control you can ship. The seam between those two, where a theory has to be checked against your actual data before anyone will deploy it, was eating most of the calendar time.
A few distinct problems live in that seam, and they're worth walking through separately, because each one resists a different kind of automation.
Was the fraud signal data even captured?
The first chunk of time goes to confirming the data exists. An analyst wants to test whether a certain device signal separates the attack from normal traffic, and before any test can run, someone has to explain why that field is empty for a third of the events. Was it ever collected for this traffic segment? Did a vendor integration stop firing at some point without anyone noticing? Or the mundane version: the field only populates for clients on a newer SDK release, so half the affected population is invisible to the rule.
Those questions sound basic. Answering them requires knowing how each field gets produced, which in most organizations means finding one of a handful of people who carry that knowledge in their heads. None of the deployment automation we built touches this work, because it happens well before there's anything to deploy.
The fraud label problem
The second chunk goes to labels, and labels in fraud are a mess in ways people outside the field tend to underestimate.
A chargeback can take weeks to arrive. A scam report may never show up at all, since it depends on a merchant, a partner bank, or the victim noticing something and bothering to file. And your own controls censor the record: the transactions you blocked, your most confident calls, generate no feedback whatsoever, because a blocked payment produces no outcome to learn from.
So before an analyst can evaluate a proposed rule, they have to spend real time establishing what ground truth even is for this attack. The backtest takes an hour. Deciding whether to believe the backtest takes days.
Fraud backtest leakage
The failure mode I worry about most is leakage: a rule or model that performs well in the backtest because information from the outcome leaked into the evaluation, then falls apart in production. Sometimes a feature only existed in hindsight. Sometimes the label window overlaps the evaluation window in a way nobody caught.
The timing is what makes it dangerous. The rule passes review, ships, and starts making automated decisions immediately, so a flaw that a manual process would have surfaced slowly compounds within hours instead. We have seen cases where reversing the damage took longer than working the original attack.
Catching this before deployment takes diagnostic infrastructure and a fair amount of paranoia. Both costs land in that same seam between investigation and testing.
What this means if you're buying AI agents in fraud operations
Most agentic fraud tooling is pitched on speed, and the speed on offer usually sits at the edges of the cycle: faster rule writing, faster deployment. If your rule engine is decent, those parts were already fast, which is exactly the situation we were in.
The gains we have actually seen come from pointing agents at the middle, and that only works when the underlying infrastructure exists:
- Entity linking, which in practice means a graph capability, so an agent scoping a fraud ring can traverse the connections instead of guessing at them
- Organized data models underneath it, because an agent can't join fragmented data any more reliably than an analyst can
- Labels with known provenance, so a rule evaluation rests on ground truth you'd defend
- Context attached to every signal: what produces the field and when it stops being trustworthy
Without that groundwork, an agent produces conclusions at machine speed with the same blind spots your analysts had.
One principle has saved us repeatedly here: anything we build for agents has to work for a human through the same interface. We spent years building investigation tools for people before pointing agents at them, and the lessons from that period, where the data lies and which entity connections mislead, are baked into what the agents now use.
Check your own fraud rule deployment cycle
Chen has written on this blog about the reaction cycle, the time from spotting a gap to running a fix, as the number a fraud team should watch. From the engineering side I'd add one thing to that: when you trace where the time goes across your last few incidents, the answer is probably a stage nobody has been optimizing. For us it was the seam between investigation and testing, and clearing everything downstream of it is what made that seam visible.
What is fraud rule deployment and why does it matter for fraud teams?
Fraud rule deployment is the process of taking a fraud control, whether a rule, model, or threshold adjustment, from investigation through testing and into production. It matters because the time between spotting an attack and having a live fix determines how much damage a fraud ring can do in the window between detection and response. A faster, more reliable fraud rule deployment process directly reduces that exposure.
What is fraud backtest leakage and why is it dangerous?
Fraud backtest leakage happens when information from the outcome of a fraud event leaks into the evaluation of a proposed rule or model, making it appear to perform better than it will in production. It is particularly dangerous because a rule that passes review due to leakage ships and starts making automated decisions immediately, meaning a flaw that a manual process would surface slowly can compound within hours. Catching leakage requires diagnostic infrastructure before deployment, not after.
What is the fraud label problem?
Fraud labels are the ground truth signals that tell a model or rule whether a past event was fraudulent. The problem is that labels in fraud are often incomplete, delayed, or censored. Chargebacks take weeks to arrive. Scam reports may never appear at all. And the transactions your fraud rules blocked generate no outcome feedback because a blocked payment produces nothing to learn from. Before a backtest can be trusted, an analyst has to spend real time establishing what ground truth actually looks like for that specific attack, which can take days even when the backtest itself takes an hour.
What does fraud signal data capture mean in practice?
Fraud signal data capture refers to whether the device, behavioral, or network signals an analyst wants to use in a rule were actually collected for the population being evaluated. A field that appears in the data model may be empty for a subset of events because a vendor integration stopped firing, an SDK version difference caused inconsistent collection, or the signal was never collected for a particular traffic segment. Confirming this requires knowledge of how each field is produced, which in most organizations lives with a small number of people rather than in documentation.
Where do AI agents for fraud operations actually help?
AI agents for fraud operations tend to be pitched on speed at the edges of the fraud rule deployment cycle, specifically faster rule writing and faster deployment. If the rule engine is already capable, those edges were already fast. The gains that matter come from pointing agents at the middle of the cycle: entity linking for fraud ring traversal, organized data models that agents can query reliably, fraud labels with known provenance, and context attached to every signal explaining what produces it and when it stops being trustworthy. Without that groundwork, an agent produces conclusions at machine speed with the same blind spots a human analyst would have.
How should fraud teams measure their fraud rule deployment performance?
Track the full fraud reaction cycle from the moment an attack is spotted to the moment a fix is running in production. Then trace where the time goes across your last several incidents. The answer is usually a stage nobody has been actively optimizing. Common culprits include fraud signal data capture gaps, fraud label quality issues, and fraud backtest leakage checks, all of which sit upstream of deployment itself. Clearing the known bottleneck, whether that is the engineering handoff or the deployment pipeline, is often what makes the real fraud detection bottleneck visible for the first time.
What is the fraud detection bottleneck most teams miss?
Most fraud teams assume the fraud detection bottleneck is the engineering handoff, where an analyst hands a proposed fix to a separate team that owns production. Removing that handoff through analyst-driven rule deployment is valuable, but it often reveals a deeper bottleneck: the seam between investigation and testing. That seam includes verifying fraud signal data capture, working through the fraud label problem, and diagnosing fraud backtest leakage before any automated fraud rules ship. These problems sit between two functions rather than inside either one, which is why they tend to go unoptimized the longest.
