The Saturday Fraud Strategist
Masterclass sobre Falsos Positivos, parte 1: Como medir FPs em sistemas que os ocultam
Sinceramente, a maioria das equipes de fraude não faz ideia de quantos bons usuários elas estão realmente bloqueando.
Peça a alguém os dados de chargeback e você geralmente receberá uma resposta bem precisa. Pergunte quantos clientes legítimos foram recusados por engano e, de repente, as coisas ficam muito menos científicas.
Geralmente, algo entre um encolher de ombros e um “provavelmente não muitos.”
Não é um bom sinal.
A detecção de falsos positivos em fraudes é fundamentalmente difícil, não porque as equipes de fraude não se importem, mas porque os sistemas de fraude muitas vezes são projetados de forma que os falsos positivos se tornem invisíveis por padrão.
Se você aprovar uma transação, o sistema recebe feedback. Fraudes se transformam em estornos. Usuários legítimos voltam e realizam novas transações.
Mas quando você bloqueia alguém, o sinal desaparece.
A reclamação fica enterrada em uma fila de suporte. O cliente nunca tenta novamente. O evento nunca se torna um rótulo. E, de repente, o seu pipeline de análise de fraude nem faz ideia de que o erro chegou a acontecer.
Esse é realmente o problema central que este episódio explora.
Mais especificamente, como as equipes de fraude podem começar a medir as taxas de falsos positivos usando abordagens imperfeitas, mas práticas, como simulação de regras de fraude, revisão manual, resolução de entidades, grupos de controle, monitoramento de transações e feedback dos usuários.
Antes de poder reduzir falsos positivos, você primeiro precisa provar que eles existem.
O que você vai ouvir neste episódio:
- Por que a detecção de fraude com falsos positivos é difícil em sistemas construídos em torno de ciclos de feedback incompletos
- Como as transações recusadas desaparecem das análises de fraude e dos dados de treinamento de modelos
- Por que os dados de chargeback são mais fáceis de mensurar do que os usuários legítimos bloqueados
- Uma análise detalhada da simulação de regras de fraude e de onde a simulação falha operacionalmente
- Como a revisão manual ajuda a identificar falsos positivos ocultos em sistemas de detecção de fraude em pagamentos
- Por que a resolução de entidades se torna uma das ferramentas mais poderosas para vincular usuários bloqueados a comportamentos legítimos posteriores
- Como grupos de controle expõem fraquezas ocultas em sistemas de decisão contra fraudes
- Onde os ciclos de feedback dos usuários podem ajudar e onde eles se tornam perigosos
- Por que a estratégia de prevenção a fraudes depende de entender a redução de falsos positivos no nível operacional
- Como a gestão de risco de fraude muda quando as equipes entendem de onde os falsos positivos realmente vêm
Uma conversa sobre sistemas de fraude, erros ocultos, pontos cegos operacionais e por que medir falsos positivos é, na maior parte das vezes, um exercício de triangulação em vez de certeza.
Quem deve ouvir:
- Líderes de fraude e analistas de fraude
- Equipes de risco e conformidade
- Gerentes de operações de fraude
- Equipes de prevenção a fraudes em FinTechs
- Profissionais de detecção de fraude em pagamentos
- Equipes que gerenciam sistemas de decisão contra fraudes
- Equipes de ciência de dados e análise de fraude
- Qualquer pessoa responsável pelo monitoramento de transações, por ferramentas de prevenção de fraude ou pela redução de falsos positivos
Basicamente, se você já olhou para o seu sistema de fraude e se perguntou se está bloqueando mais bons usuários do que imagina, este episódio é para você.
Notas do episódio
Visibilidade
Os sistemas de fraude são muito bons em medir fraudes confirmadas. Eles são muito piores em medir clientes legítimos que desaparecem depois de serem bloqueados.
E isso cria um problema operacional estranho: as equipes de fraude muitas vezes otimizam com base no que conseguem ver, enquanto ignoram as perdas ocultas em transações recusadas.
Por isso, apresento vários métodos práticos que as equipes de fraude usam para estimar a taxa de falsos positivos, incluindo simulação de regras de fraude, revisão manual, grupos de controle, feedback dos usuários e resolução de entidades.
Nenhum deles é perfeito.
Esse é meio que o ponto.
Triangulação
Você combina vários sinais incompletos até que o quadro fique bom o suficiente para a tomada de decisão.
A conversa também aborda camadas de decisão de fraude, limites de modelos, monitoramento de transações, fricção operacional e por que as ferramentas de prevenção a fraudes muitas vezes criam pontos cegos que as próprias equipes não conseguem medir completamente.
E, sinceramente, quando você percebe quantos falsos positivos nunca viram rótulos estruturados, começa a entender por que tantas equipes de fraude subestimam o problema desde o início.
Principal conclusão
Zero falsos positivos é um mito.
O verdadeiro objetivo é entender de onde vêm os falsos positivos, quanto desse problema está realmente sob o seu controle e quais erros valem a pena corrigir operacionalmente.
Quando as equipes de fraude finalmente conseguirem medir corretamente os falsos positivos na detecção de fraude, poderão parar de adivinhar e começar a fazer concessões de forma intencional.
Menos mágico.
Provavelmente muito mais útil.
Ainda não está pronto para encerrar a conversa sobre o meu e, espero, o seu assunto favorito? Assine a newsletter The Saturday Fraud Strategist.
Episode transcript
Chen Zamir
00:11
Everyone agrees false positives are bad, but most teams don't know how to quantify them. Ask a fraud leader for last month's chargeback count, and you will get a precise number. Ask how many good customers were blocked by mistake, and you'll get a shrug, or worse, none. Why? Because the way we build fraud systems makes false positives invisible by design. Here's what I mean. If you approve a transaction, you get feedback. If it was fraud, you'll hear about it. If it was good, you will see the customer come back, spend more, log in again, do normal customer things. But if you block something, you get nothing. Best case, a customer complains, but even then that complaint will rarely make its way back into your data. It usually dies in a silo, buried in a support ticket, stuck in an inbox, or fading into the memory of a support rep. It almost never becomes a label attached to a specific event that fits into your model training pipelines or powers your dashboards. So, before we talk about reducing false positives, we have to talk about labeling them. So, how do you measure an error that the system is designed to hide? Let me start with a disappointing truth. There is no silver bullet. Measuring false positives is always an exercise in triangulation. You need to combine several imperfect methods to get a good enough picture for decision making. Let's get down to it. Teams naturally gravitate toward simulation because it's easy. You simply backtest your logic against historical data. Take a rule, or a rule set, or a workflow, and run it against the last six months of traffic. Look at all the events it would have declined, then cross-reference them against their final status, legit or fraud. Don't forget to account for fraud maturation, since chargebacks take time to materialize. Recent data is effectively unlabeled. A 30-day buffer is the minimum, and a 90-day window ensures your clean users are actually clean. Any transaction labeled clean that your rule would have blocked is a theoretical false positive. The appeal here is accessibility. You can run this in SQL or your fraud platform without deploying any new infrastructure. But simulation comes with important limitations. You are testing one rule in isolation, but in production that rule lives inside a complex decision stack with different rules, model thresholds, AI agents, manual reviews, third-party decisions, and so on. You also cannot backtest operational friction. Simulation assumes perfect code execution, but misses bugs, integration issues, or the behavior of upstream actors like issuers or processors. All of those can create false positives, and none of them are captured in a backtest. So, yes, simulate. It's a good start. Just be aware of its limitations. The second method is older, slower, and much more powerful. You manually review decline traffic. Instead of looking at the logic, you look directly at the impact. You take a sample of blocked events, either across the board or for a specific solution, and you ask an experienced fraud investigator to label them manually. This is essentially the same process you use to design your rules or hunt for new fraud patterns, just applied in reverse. Now, the limitations are obvious: it doesn't scale, it's time consuming, and it's expensive operationally. You also need experienced analysts, and even then, a percentage of events will remain as inconclusive gray cases. On the flip side, because this is an offline audit rather than a live decision, case-specific accuracy matters less. You also don't need to label everything. All you need is a sense of where you stand and where the worst offenders are. After all, it doesn't really matter if you conclude that the rule has an accuracy of 45% or 48%.
Chen Zamir
04:14
The third method is the one I consider the most underrated.
Chen Zamir
04:16
Linking. The idea is simple: in many cases, good users show up multiple times on your platform before or even after they get blocked. They may try again with a different card. They may reattempt onboarding with a different email address. They may transact again from the same device with an IP address you don't find risky. If you can link these events together and see the full sequence, you suddenly get an extremely strong signal that the original decline was a false positive. This linking can be straightforward: same user ID, different card. Same device, different email. Same IP and name, different device. Or it can be fuzzier, like a blocked user followed by a successful registration of their spouse. Once you have the basic infrastructure in place, the method scales beautifully. You simply run over your block population periodically and try to associate them with good events that happen before or after. The catch is that you need that infrastructure. You need to research a heuristic that would perform well. You need some kind of entity resolution solution. You need to be able to pull sequences of events over time. That is non-trivial work, but the ROI is massive. This really enables you to generate a continuous stream of high-precision false positive labels. It won't catch every false positive, but the ones it does catch will be very reliable. The fourth method is one that some might feel uncomfortable with: control groups. You take a randomized sample of traffic, typically around 1%, and whitelist it completely. No rules. No model-based declines. Normal reviews. Nothing. The population sails straight through to completion. Then you simply watch what happens. If your overall system performs well, you will see a meaningful amount of fraud in that control group. If, on the other hand, you notice that the control group's false positive rate is much higher than your expectations, then you know that you need to re-examine your setup. Control groups are powerful because they answer a very direct question: What would have happened if we removed the system entirely? The answer usually isn't pretty to look at, but it is an honest one. This approach, however, is only practical at scale. You need enough traffic that 1% or 2% still gives you statistically meaningful numbers in a timeframe that makes sense. You also need the budget to absorb the loss. It may not seem much, but if your incoming fraud pressure is 5% and you approve 1% of that, you just created five bps of loss before even considering chargeback fees. So this is a great tool to have, but it's likely only practical in your situation if your organization is mature and profitable enough. The fifth method, requesting direct user feedback, is a more opportunistic tactic, only available in select contexts. If you're looking at declined transactions on accounts you already know and trust, you can trigger a notification: We blocked a transaction ending in 1234. Was this you? But this isn't something you can do across the board. It's unusable at onboarding when you've never seen the user before. Asking a stranger if they are a fraudster doesn't really make much sense. Additionally, and I cannot stress this enough, do not give fraudsters a way to mark their own attempts as good. If a “yes, it was me” button automatically lifts the block, you haven't built a feedback loop. You've built a back door for fraudsters to find and exploit. But as an additional source of ground truth in very specific situations, such as failed login attempts or contact detail changes, this can be surprisingly helpful. If your conclusion is none of these methods are perfect, you are correct. That is the nature of the beast. But remember, you're not looking for a single magic metric that tells you the exact number of false positives in your system. You're looking to combine several independent indicators to build both accuracy and scalability. In practice, most teams end up with some combination of simulations for quick rule-by-rule sense checks, manual reviews for high-value flows and top offending solutions, linking as the main automation workflow once the engineering work is done, control groups for large populations where you can tolerate some control losses, and finally, user feedback in narrow, carefully chosen contexts. And if you mix these properly, you will end up with a workable estimate of your false positives broken down by at least a few important dimensions: product flow, traffic type, decision layer, and so on. And once you can measure the problem, you can finally start asking the right questions.
Chen Zamir
09:10
It's one thing to know that you have a false positive problem. It's another thing to know where it comes from. Is it mostly from rules? Your model threshold? Your human reviewers? A specific integration that corrupts IP addresses on mobile? A third-party payment processor that's hyper-aggressive on a specific region? In the second part of this series, we'll take the false positives you've managed to detect and bucket them by root cause. Wrapping up, let me just say that zero false positives is a myth. But you can still get to a point where every remaining false positive is either consciously accepted or outside of your sphere of influence. And that clarity alone is worth a lot.
View full transcript
