The Saturday Fraud Strategist
False Positives Masterclass Part 3: How to reduce FPs inside your system
Okay, so here’s the thing about reducing false positives. Most teams want to jump straight into tactics. Tune the rule. Adjust the threshold. Add an exemption. Move the weird edge cases into manual review. Fine. All of that might be useful. But honestly, if that is where you start, you are probably guessing.
And guessing in fraud prevention is not exactly my favorite operating model. Not because it never works. Sometimes it does. Which is almost worse, because then everyone gets confident. Not a good look.
In this episode, I continue the False Positives Masterclass by moving from measurement and bucketing into the part everyone actually wants to get to: fixing the parts of the system that are misbehaving. But the point is not just to reduce false positives. The point is to reduce false positives without creating a new fraud problem you only discover three weeks later when the losses mature and everyone starts quietly looking at the dashboard like it personally betrayed them.
This episode is about discipline. It is about manual review, fraud rules, fraud model precision, fraud model recall, shadow mode testing, data quality issues, and the uncomfortable but necessary question every fraud team eventually has to ask: is this rule actually helping, or have we just been emotionally attached to it since that one fraud spike in 2022?
What you’ll hear in this episode:
- Why reducing false positives should start with manual review, not instinct
- How to decide whether a fraud rule should be removed, downgraded, or improved
- Why fraud model precision and fraud model recall matter when rules catch fraud but hurt good users
- How to build exclusions without accidentally creating a back door for fraudsters
- Why shadow mode testing and challenger rules are essential before release
- How data quality issues can make otherwise reasonable fraud prevention logic misbehave
- Why fraud operations teams need to be pragmatic, not elegant, when the data is broken
You should listen to this episode if you:
- Own fraud rules, fraud detection rules, models, AI agents, or review flows
- Are trying to reduce false positives without increasing fraud losses
- Have a high false positive rate but are not sure which part of the system is causing it
- Need a more structured way to review manual review samples and top offenders
- Are dealing with corrupted data, noisy signals, or flows where fraud prevention logic keeps misfiring
Episode notes & key takeaways
(Accordion dropdown)
Reducing false positives starts with looking, not guessing
The first uncomfortable point in this episode is also the simplest one: you cannot reason your way through a false positive problem from a dashboard alone. You have to look. Manually. At real events. I know, very glamorous. Everyone got into fraud operations so they could inspect 100 blocked cases and ask themselves why a rule exists. Honestly, though, this is where the useful work starts.
If a rule, model, or AI agent is producing a high number of false positives, the first step is to manually inspect a sample of what it blocked. In most cases, 50 to 100 events is enough to get directional clarity. You are not trying to create a dissertation. You are trying to answer practical questions:
- Is this rule actually as inaccurate as we think?
- Does this rule still deserve to exist?
- Should it still make automated decisions?
- Could it move into manual review instead?
- What would reduce false positives without materially increasing fraud losses?
The key is that every answer depends on your business. There is no universal “good enough” threshold. Some teams have manual review capacity. Some do not have manual review at all. Some rules are noisy but still necessary because the volume is too large to send to analysts. Annoying? Yes. But also reality.
Three paths for every misbehaving fraud rule
Once you review the evidence, most misbehaving fraud rules or models fall into one of three categories. This is where the work gets cleaner.
The first category is rules that should not exist at all. These are the low-hanging fruit. Usually they were created during a crisis, caught something once, and then stayed in production forever because nobody wanted to touch them. Very human. Very common. Also, not great. If the fraud coverage is negligible and the false positives are significant, removing the rule may be the safest and cleanest fix.
The second category is rules that should exist, but should not make automated decisions anymore. These are mid-accuracy rules. They catch meaningful fraud, but they do not do it reliably enough to justify auto-decline. If your fraud operations team has capacity, moving that logic into case management can preserve fraud coverage while reducing false positives. You put a human in the loop, and that human can spot pattern changes in real time.
The tradeoff is cost. Manual review is useful, but it is not free. You reduce false positives, but you increase operational workload. So you’ve got to ask yourself whether the compromise is actually worth it.
The third category is the most common and most interesting: rules that should exist, but need to be improved. These rules often have high recall and low precision. In plain English, they catch fraud, but they also hit too many good users. So the answer is not to delete them. The answer is to refine them.
False positive reduction is fraud detection in reverse
One of the more useful ways to think about reducing false positives is that it is basically fraud detection in reverse.
When you build a fraud rule, you usually start with confirmed fraud. You review the fraud cases, identify the pattern they share, compare that pattern against the good population, and then build logic that catches fraud without catching too many legitimate events.
To reduce false positives, you do the same thing with the opposite goal. You start with confirmed false positives, identify the patterns they share, compare those patterns against the fraud cases, and then build exclusions that separate good users from fraud without releasing too much risk.
This is where teams sometimes get into trouble. It is not enough to say, “A lot of false positives look like X.” Okay, useful. But do fraud cases also look like X? If yes, your exclusion is not an exclusion. It is a fraud invitation with better formatting.
That last step matters. A lot.
How to build exclusions without creating fraud back doors
The example in this episode is intentionally simple: decline if IP country does not match account country. It is a basic mismatch rule, and yes, real-world rules are often more complex. But not always as much as we like to pretend.
Suppose you manually review 100 events blocked by this rule. Thirty are fraud. Seventy are legitimate. Of those 70 legitimate events, 20 involve U.S. users whose IP addresses appear in Canada. That is a meaningful false positive pattern.
Now the next question is the important one: do your fraud cases also show that U.S.-account, Canadian-IP pattern? If not, you may have a clean separation. Maybe you have commuters. Maybe VPNs route through Canadian endpoints. Maybe your user base lives near the border. Whatever the cause, the false positive pattern is real, and the fraud overlap is low.
That gives you the basis for a safer exclusion:Decline if IP country does not match account country, unless IP country is Canada and account country is the U.S.
Or, depending on your business footprint and available features, maybe you broaden that into neighboring-country logic. The point is not the exact rule. The point is the method. The exclusion comes from the false positive population and is validated against the fraud population.
That is how you reduce false positives without accidentally creating a new entry point for fraudsters. Am I being too optimistic that everyone does this step? Probably.
Test every fraud logic change before release
Any time you change fraud prevention logic, even if it is “just” an exclusion, you are changing your risk posture. Treat it like a new rule release.
Shadow mode testing and challenger rules are your best friends here. Keep the current rule active, but run the refined version in parallel. Then measure the difference.
You want to know:
- How many additional events would the new version allow?
- How many of those events eventually mature into fraud?
- How many false positives would the change prevent?
- Does the new logic behave consistently over time?
If the results hold, you can deploy with more confidence. If they do not, adjust. It is not glamorous. It is not a big dramatic launch moment. It is just better fraud operations.
The monitoring window should depend on how quickly fraud matures in your environment. If you have time, wait long enough to compare fraud outcomes in the challenger version against the original. If you are under pressure, manually review random samples. Not perfect, but better than flying blind.
Data quality issues can make good logic look bad
Sometimes the problem is not the rule. It is the data feeding the rule.
This is frustrating because everyone wants the fix to be in the fraud logic. Change the rule, lower the score, add the exception, ship it. But if the underlying field is corrupted, missing, inconsistent, or broken in a specific flow, the rule may be behaving exactly as designed. The input is just bad.
Okay. Annoying, but useful to know.
If the data issue is under your control, the best long-term answer is to fix the data quality issue itself. That might mean repairing an API payload, updating an SDK version, fixing an internal integration, or working with product and engineering to close the gap. It may take days. It may take weeks. But once the data is fixed, the fraud rule may start behaving correctly again.
Also, you may fix other problems at the same time. Data issues rarely ruin just one thing. They usually wander around the system quietly making several things worse. Very efficient, in the worst possible way.
When the data cannot be fixed quickly, adjust the logic
Not every data quality issue is fixable in the near term. Sometimes it comes from an external partner. Sometimes it comes from a platform you do not control. Sometimes it comes from a system you technically do control, but the fix is sitting somewhere behind 14 roadmap priorities and one team that keeps saying “next quarter.” Not a good look, but familiar.
If you cannot fix the data soon, adjust the fraud logic for that flow. That might mean excluding the problematic flow from the rule, lowering the rule’s weight, changing the rule’s impact, or moving those cases into manual review.
This is where fraud prevention needs to be pragmatic rather than elegant. A clean system is nice. A working system is better. Sometimes the right answer is not the most theoretically satisfying one. Sometimes it is the one that keeps good users moving while still containing fraud risk.
Final takeaway:
Reducing false positives is not about making your system softer. It is about making it more precise.
That means reviewing the actual events, separating rules that should be removed from rules that should be downgraded or improved, building exclusions based on evidence, validating those exclusions against fraud cases, testing changes in shadow mode, and fixing data quality issues when the logic is not really the problem.
Anyway, the slightly uncomfortable takeaway is this: if you are reducing false positives without looking at the underlying cases, you are not tuning. You are guessing.
And maybe you get lucky. Maybe.
But in fraud prevention, luck is not really a control. It is more of a temporary condition.
Connect with Chen Zamir | LinkedIn
Host of The Saturday Fraud Strategist
Helping fintechs build smarter fraud defenses
Co-author of “The Fraud Fighter’s AI Playbook”
