The Saturday Fraud Strategist
False Positives Masterclass, part 1: How to measure FPs in systems that hide them
Honestly, most fraud teams have no idea how many good users they are actually blocking.
Ask someone for their chargeback data and you’ll usually get a very precise answer. Ask how many legitimate customers were declined by mistake and suddenly things get a lot less scientific.
Usually somewhere between a shrug and “probably not many.”
Not a great sign.
False positive fraud detection is fundamentally difficult, not because fraud teams do not care, but because fraud systems are often designed in ways that make false positives invisible by default.
If you approve a transaction, the system gets feedback. Fraud turns into chargebacks. Legitimate users come back and transact again.
But when you block someone, the signal disappears.
The complaint gets buried in a support queue. The customer never retries. The event never becomes a label. And suddenly your fraud analytics pipeline has no idea the mistake even happened.
That is really the core problem this episode explores.
More specifically, how fraud teams can start measuring false positive rates using imperfect but practical approaches like fraud rules simulation, manual review, entity resolution, control groups, transaction monitoring, and user feedback.
Before you can reduce false positives, you first need to prove they exist.
What you’ll hear in this episode:
- Why false positive fraud detection is difficult in systems built around incomplete feedback loops
- How declined transactions disappear from fraud analytics and model training data
- Why chargeback data is easier to measure than blocked legitimate users
- A breakdown of fraud rules simulation and where simulation fails operationally
- How manual review helps identify hidden false positives inside payment fraud detection systems
- Why entity resolution becomes one of the strongest tools for linking blocked users to later legitimate behavior
- How control groups expose hidden weaknesses in fraud decisioning systems
- Where user feedback loops can help, and where they become dangerous
- Why fraud prevention strategy depends on understanding false positive reduction at the operational level
- How fraud risk management changes once teams understand where false positives actually come from
A conversation about fraud systems, hidden mistakes, operational blind spots, and why measuring false positives is mostly an exercise in triangulation rather than certainty.
Who should listen:
- Fraud leaders and fraud analysts
- Risk and compliance teams
- Fraud operations managers
- FinTech fraud prevention teams
- Payment fraud detection professionals
- Teams managing fraud decisioning systems
- Data science and fraud analytics teams
- Anyone responsible for transaction monitoring, fraud prevention tools, or false positive reduction
Basically, if you have ever looked at your fraud system and wondered whether you are blocking more good users than you realize, this episode is for you.
Episode notes
Visibility
Fraud systems are very good at measuring confirmed fraud. They are much worse at measuring legitimate customers who disappear after getting blocked.
And that creates a strange problem operationally: fraud teams often optimize around what they can see while ignoring the losses hidden inside declined transactions.
So I walk through several practical methods fraud teams use to estimate false positive rate, including fraud rules simulation, manual review, control groups, user feedback, and entity resolution.
None of them are perfect.
That is kind of the point.
Triangulation
You combine several incomplete signals until the picture becomes good enough for decision-making.
The conversation also gets into fraud decisioning layers, model thresholds, transaction monitoring, operational friction, and why fraud prevention tools often create blind spots the teams themselves cannot fully measure.
And honestly, once you realize how many false positives never become structured labels, you start understanding why so many fraud teams underestimate the problem in the first place.
Key takeaway
Zero false positives is a myth.
The real goal is understanding where false positives come from, how much of the problem is actually under your control, and which mistakes are worth fixing operationally.
Once fraud teams can finally measure false positive fraud detection properly, they can stop guessing and start making tradeoffs intentionally.
Less magical.
Probably much more useful.
Not ready to stop the conversation about my, and hopefully your, favorite subject? Subscribe to The Saturday Fraud Strategist newsletter.
Episode transcript
Chen Zamir
00:11
Everyone agrees false positives are bad, but most teams don't know how to quantify them. Ask a fraud leader for last month's chargeback count, and you will get a precise number. Ask how many good customers were blocked by mistake, and you'll get a shrug, or worse, none. Why? Because the way we build fraud systems makes false positives invisible by design. Here's what I mean. If you approve a transaction, you get feedback. If it was fraud, you'll hear about it. If it was good, you will see the customer come back, spend more, log in again, do normal customer things. But if you block something, you get nothing. Best case, a customer complains, but even then that complaint will rarely make its way back into your data. It usually dies in a silo, buried in a support ticket, stuck in an inbox, or fading into the memory of a support rep. It almost never becomes a label attached to a specific event that fits into your model training pipelines or powers your dashboards. So, before we talk about reducing false positives, we have to talk about labeling them. So, how do you measure an error that the system is designed to hide? Let me start with a disappointing truth. There is no silver bullet. Measuring false positives is always an exercise in triangulation. You need to combine several imperfect methods to get a good enough picture for decision making. Let's get down to it. Teams naturally gravitate toward simulation because it's easy. You simply backtest your logic against historical data. Take a rule, or a rule set, or a workflow, and run it against the last six months of traffic. Look at all the events it would have declined, then cross-reference them against their final status, legit or fraud. Don't forget to account for fraud maturation, since chargebacks take time to materialize. Recent data is effectively unlabeled. A 30-day buffer is the minimum, and a 90-day window ensures your clean users are actually clean. Any transaction labeled clean that your rule would have blocked is a theoretical false positive. The appeal here is accessibility. You can run this in SQL or your fraud platform without deploying any new infrastructure. But simulation comes with important limitations. You are testing one rule in isolation, but in production that rule lives inside a complex decision stack with different rules, model thresholds, AI agents, manual reviews, third-party decisions, and so on. You also cannot backtest operational friction. Simulation assumes perfect code execution, but misses bugs, integration issues, or the behavior of upstream actors like issuers or processors. All of those can create false positives, and none of them are captured in a backtest. So, yes, simulate. It's a good start. Just be aware of its limitations. The second method is older, slower, and much more powerful. You manually review decline traffic. Instead of looking at the logic, you look directly at the impact. You take a sample of blocked events, either across the board or for a specific solution, and you ask an experienced fraud investigator to label them manually. This is essentially the same process you use to design your rules or hunt for new fraud patterns, just applied in reverse. Now, the limitations are obvious: it doesn't scale, it's time consuming, and it's expensive operationally. You also need experienced analysts, and even then, a percentage of events will remain as inconclusive gray cases. On the flip side, because this is an offline audit rather than a live decision, case-specific accuracy matters less. You also don't need to label everything. All you need is a sense of where you stand and where the worst offenders are. After all, it doesn't really matter if you conclude that the rule has an accuracy of 45% or 48%.
Chen Zamir
04:14
The third method is the one I consider the most underrated.
Chen Zamir
04:16
Linking. The idea is simple: in many cases, good users show up multiple times on your platform before or even after they get blocked. They may try again with a different card. They may reattempt onboarding with a different email address. They may transact again from the same device with an IP address you don't find risky. If you can link these events together and see the full sequence, you suddenly get an extremely strong signal that the original decline was a false positive. This linking can be straightforward: same user ID, different card. Same device, different email. Same IP and name, different device. Or it can be fuzzier, like a blocked user followed by a successful registration of their spouse. Once you have the basic infrastructure in place, the method scales beautifully. You simply run over your block population periodically and try to associate them with good events that happen before or after. The catch is that you need that infrastructure. You need to research a heuristic that would perform well. You need some kind of entity resolution solution. You need to be able to pull sequences of events over time. That is non-trivial work, but the ROI is massive. This really enables you to generate a continuous stream of high-precision false positive labels. It won't catch every false positive, but the ones it does catch will be very reliable. The fourth method is one that some might feel uncomfortable with: control groups. You take a randomized sample of traffic, typically around 1%, and whitelist it completely. No rules. No model-based declines. Normal reviews. Nothing. The population sails straight through to completion. Then you simply watch what happens. If your overall system performs well, you will see a meaningful amount of fraud in that control group. If, on the other hand, you notice that the control group's false positive rate is much higher than your expectations, then you know that you need to re-examine your setup. Control groups are powerful because they answer a very direct question: What would have happened if we removed the system entirely? The answer usually isn't pretty to look at, but it is an honest one. This approach, however, is only practical at scale. You need enough traffic that 1% or 2% still gives you statistically meaningful numbers in a timeframe that makes sense. You also need the budget to absorb the loss. It may not seem much, but if your incoming fraud pressure is 5% and you approve 1% of that, you just created five bps of loss before even considering chargeback fees. So this is a great tool to have, but it's likely only practical in your situation if your organization is mature and profitable enough. The fifth method, requesting direct user feedback, is a more opportunistic tactic, only available in select contexts. If you're looking at declined transactions on accounts you already know and trust, you can trigger a notification: We blocked a transaction ending in 1234. Was this you? But this isn't something you can do across the board. It's unusable at onboarding when you've never seen the user before. Asking a stranger if they are a fraudster doesn't really make much sense. Additionally, and I cannot stress this enough, do not give fraudsters a way to mark their own attempts as good. If a “yes, it was me” button automatically lifts the block, you haven't built a feedback loop. You've built a back door for fraudsters to find and exploit. But as an additional source of ground truth in very specific situations, such as failed login attempts or contact detail changes, this can be surprisingly helpful. If your conclusion is none of these methods are perfect, you are correct. That is the nature of the beast. But remember, you're not looking for a single magic metric that tells you the exact number of false positives in your system. You're looking to combine several independent indicators to build both accuracy and scalability. In practice, most teams end up with some combination of simulations for quick rule-by-rule sense checks, manual reviews for high-value flows and top offending solutions, linking as the main automation workflow once the engineering work is done, control groups for large populations where you can tolerate some control losses, and finally, user feedback in narrow, carefully chosen contexts. And if you mix these properly, you will end up with a workable estimate of your false positives broken down by at least a few important dimensions: product flow, traffic type, decision layer, and so on. And once you can measure the problem, you can finally start asking the right questions.
Chen Zamir
09:10
It's one thing to know that you have a false positive problem. It's another thing to know where it comes from. Is it mostly from rules? Your model threshold? Your human reviewers? A specific integration that corrupts IP addresses on mobile? A third-party payment processor that's hyper-aggressive on a specific region? In the second part of this series, we'll take the false positives you've managed to detect and bucket them by root cause. Wrapping up, let me just say that zero false positives is a myth. But you can still get to a point where every remaining false positive is either consciously accepted or outside of your sphere of influence. And that clarity alone is worth a lot.
View full transcript
