The Saturday Fraud Strategist
Masterclass sobre Falsos Positivos, parte 2: Como identificar de onde os FPs se originam
9 min
Um dos erros mais comuns que vejo as equipes de fraude cometerem é atacar diretamente os falsos positivos.
Um cliente reclama. O CEO diz que o modelo está bloqueando demais. Alguém abre um painel, ajusta o limite do modelo de fraude, talvez faça alguns ajustes em algumas regras de fraude e, de repente, todo mundo sente que o progresso está acontecendo.
Sinceramente, não pega bem.
Não porque reduzir falsos positivos seja um objetivo errado. É absolutamente o objetivo certo. O problema é que a maioria das equipes parte imediatamente para o tático. E, se você for apenas tático na forma como reduz falsos positivos, provavelmente só deverá esperar ganhos táticos.
Neste episódio, entramos na segunda parte da masterclass sobre falsos positivos: como decompor modelos de detecção de fraude com falsos positivos em grupos que você realmente possa priorizar e corrigir. Analisamos de onde vêm os falsos positivos, quais são gerados por modelos de detecção de fraude, regras de fraude, revisão manual, parceiros upstream, analistas de fraude, problemas de qualidade de dados, sinais de fraude corrompidos e fluxos de trabalho de detecção de fraude em pagamentos.
Quantificar falsos positivos é útil.
Mas isso não é um plano.
O que você vai ouvir neste episódio:
- Por que reduzir falsos positivos exige análise de causa raiz, e não apenas ajuste de modelo
- Como identificar quem realmente recusou o evento: uma regra, limite de modelo de fraude, agente de IA, analista de fraude, equipe de revisão manual, emissor, adquirente ou fornecedor de soluções antifraude
- Por que parceiros de pagamento upstream podem gerar falsos positivos que seus próprios sistemas de prevenção a fraudes não conseguem corrigir diretamente
- Como a tomada de decisão em fraudes se desdobra entre detecção de fraude em pagamentos, pontuação de risco de fraude e fluxos de trabalho operacionais
- Por que a otimização de sistemas de fraude começa pela identificação dos piores infratores
- Como problemas de qualidade de dados, sinais de fraude corrompidos e deriva de modelo criam falsos positivos que parecem risco de fraude
- Como as equipes de operações de fraude podem priorizar os grupos que são grandes o suficiente, solucionáveis o suficiente e valiosos o suficiente para serem tratados primeiro
Quem deve ouvir:
- Líderes de operações de fraude que buscam melhorar a precisão da detecção de fraudes
- Analistas de fraude que trabalham em filas de revisão manual
- Equipes de risco que gerenciam regras de fraude, limites de modelos de fraude e pontuações de risco de fraude
- Equipes de ciência de dados responsáveis por modelos de detecção de fraude e por monitorar o desvio dos modelos
- Equipes de detecção de fraude em pagamentos que lidam com recusas de emissores e decisões de parceiros a montante
- Equipes de prevenção a fraudes que buscam reduzir falsos positivos sem aumentar as perdas
- Qualquer pessoa que já tenha ficado olhando para um painel de falsos positivos e pensado: “Tá, e agora?”
Notas do episódio
Um padrão frustrante, mas familiar
As equipes de fraude muitas vezes só reagem aos falsos positivos depois de uma reclamação de cliente, de uma escalada para a diretoria ou de uma preocupação repentina de que “o modelo está bloqueando demais”.
Ok, justo.
Mas agora você precisa se perguntar: você está resolvendo a causa raiz ou apenas ajustando a parte do sistema que é mais fácil de enxergar?
Essa distinção é importante.
Os sistemas modernos de prevenção a fraudes oferecem muitas ferramentas às equipes
Modelos de detecção de fraude, regras antifraude, pontuação de risco de fraude, revisão manual, agentes de IA, sistemas de detecção de fraude em pagamentos e analistas de fraude desempenham todos um papel na interrupção de atividades suspeitas.
O mesmo sistema que protege você também pode bloquear bons usuários se você não entender de onde cada decisão realmente está vindo.
O grande problema é a falta de visibilidade
Um falso positivo pode vir das suas próprias regras de fraude. Pode vir de um limite conservador do modelo de detecção de fraude. Pode vir de uma análise manual. Pode vir de um emissor, adquirente, processador, fornecedor de soluções antifraude ou outro parceiro a montante.
Se você não sabe quem disse “não”, não tem como saber o que corrigir.
Falsos positivos não são apenas erros do modelo
Bons clientes são bloqueados. Analistas de fraude perdem tempo revisando casos que poderiam ser evitados. As filas de revisão manual aumentam. As equipes de detecção de fraude em pagamentos perseguem o problema errado. Líderes de operações de fraude têm dificuldade para explicar por que o atrito com o cliente está aumentando.
E em algum lugar no meio de tudo isso, um usuário perfeitamente legítimo está se perguntando por que o seu sistema decidiu que ele parecia suspeito.
Nada bom.
Problemas de qualidade dos dados
Às vezes, a lógica de detecção de fraude está correta. O problema é que o sistema está operando com sinais de fraude corrompidos. Talvez o campo de IP esteja errado. Talvez os IDs de dispositivo estejam faltando. Talvez os metadados de pagamento nunca tenham sido transmitidos. Talvez um bug no SDK móvel esteja deixando o tráfego de iOS com uma aparência estranha.
De repente, seus modelos de detecção de fraude começam a desviar. Suas regras de fraude falham. Sua pontuação de risco de fraude parece pior do que realmente é.
E agora a equipe está “otimizando o modelo” quando o verdadeiro problema é uma entrada com erro.
Clássico.
O caminho a seguir
O melhor caminho é agrupar os falsos positivos por agente, parceiro, jornada do usuário, fluxo do produto, plataforma, método de pagamento, região geográfica e tipo de problema de dados.
Priorize os problemas que são:
- Grande o suficiente para ser relevante
- Realmente sob o seu controle
- Não causado por problemas de dados que você não consegue corrigir
- Conectado a regras de fraude de alto impacto, limites de modelos ou políticas de revisão manual
É assim que a otimização do sistema de fraude se torna estratégica em vez de reativa.
Principais aprendizados
Reduzir falsos positivos não se resume apenas a limites de modelos de fraude, regras de fraude ou revisão manual. Trata-se de entender todo o sistema de decisão de fraude, incluindo parceiros, plataformas, fluxos de trabalho, analistas, sinais de fraude corrompidos e problemas de qualidade de dados.
Quando você souber de onde vem o problema, poderá finalmente decidir o que corrigir primeiro.
E, sinceramente, esse é um lugar muito melhor para se estar.
Ainda não está pronto para encerrar a conversa sobre o meu e, espero, o seu assunto favorito? Assine a newsletter The Saturday Fraud Strategist.
Episode transcript
Chen Zamir
00:08
One of the common mistakes I see fraud teams make is attacking false positives head on. They'll get a customer complaint, and they'll now trace back why this false positive happened and how to make sure it doesn't happen again, or the CEO complains that the model is blocking too much, and now they try to optimize it. The problem, of course, isn't the fact that they're trying to minimize their system's false positives. It's the fact that they're being tactical about it. And if you're tactical about how you do things, you can only expect tactical gains at best. In part one of the false positives masterclass, and if you missed that one, the link is down below, we accepted the reality that your system isn't built to report its own mistakes. We discussed several methods that can help you work around that and generate a usable picture of your false positives. Then comes the next problem. Quantifying false positives is merely an observation. It is not a plan. To reduce false positives in a meaningful way, you have to go one level deeper and ask a different set of questions. Where do they come from? Which are driven by my own system, and which my partners? Which are actually data quality problems masquerading as fraud risk? And most importantly, where should I start? That's what the second part of the series is about, breaking down your false positives into buckets that you can prioritize and act on. How do you do that? Here's my seven-step process. The first and most important step is to attach every false positive to the actor that made a decision. When a transaction is declined or an onboarding attempt is blocked, someone or something said no. That someone might be a rule in your system, machine learning model threshold, an AI agent, human analyst, manual reviewer, even a third-party partner, an issuer, acquirer, or fraud vendor. Without this categorization, you will default to optimizing the things you can see, usually your rules and your models. That's how teams spend months fine-tuning rules, only to later learn that most of their declines were coming from an issuer they never spoke to. So, your first task is to map every decline to the actor that made that decision. Sometimes it will be a single rule. Sometimes it will be a decision based on a certain score threshold. Sometimes it will be a human decision. Sometimes it will be a response coming back from a payment partner. The point is that you don't have to do this perfectly on day one. Even a rough breakdown makes an enormous difference, because it will help you get a sense for where the most value lies. Once you have that basic map, you can ask the following question: How much of this is even under my control? This is particularly important in card payments, where a single card transaction can pass through half a dozen hands before the issuer finally says yes or no. Between the cardholder and the issuing bank, you often have the merchant or platform itself, a payment service provider, an acquirer, an acquiring processor, a card network, an issuer processor, and finally the issuing bank. And that's even without counting the third-party fraud vendors that some of these actors plug into their own stack. Each of these actors can decline a transaction. Each has its own risk logic, and each contributes its own false positives. Now, if 60% of your false positives are driven by upstream partners, then your maximum sphere of influence is capped at 40%. That doesn't mean you should give up and walk away, but it does mean you should rethink your goals, how you set expectations internally, and where you spend your energy. And I think that this is an important point that I see a lot of fraud leaders miss. You cannot tune rules you don't own. You can, however, quantify the impact, make it visible, make sure everyone in the organization understands where the limits of your influence actually are. And trust me on that. This saves a lot of frustration later. And also, strategically, if you're unhappy with your partner's performance and believe it's substandard, you can always work to replace them. Once you have your decision sorted into high-level buckets, it's time to go one level deeper.
Chen Zamir
03:50
For example, you take a rule engine bucket that is responsible, say, for 25% of all false positives, and you break those 25% down to individual rules. And you do that as best you can for each one of your buckets. Now, when I say as best you can, what I mean is that realistically this can easily become time-consuming and low-value work. But let's try to break it down into the top five offenders in each category, and remember that even if a rule is relatively accurate, high volume usually means it is a primary source of false positives. High-volume decision is a good rule of thumb in terms of where to start. The point is that you want to identify the specific actors which are responsible for the most amount of false positives in absolute terms. These are your quick wins. At the same time, your gut might tell you that some offenders are flying below radar, legacy rules, outdated policies, or solutions that were never properly validated. Don't ignore that gut instinct, even if these actors have low volumes, or at least don't ignore it before you collect data on it. So, we now know what is blocking your users, but we still need to know where those users are coming from, even within the part of the stack you control. False positives are rarely evenly distributed. They tend to cluster around specific flows, such as mobile versus web, iOS versus Android, maybe different products or payment methods, or maybe even like new versus established users. If you only look at the actor that declined the event, you will see a rule or a model misbehaving. But if you look at which flow the user went through, you might find a deeper issue. Specific flow that produces corrupted or missing data, causing your entire fraud stack to misfire. Here's an example. Imagine you have an integration bug in your mobile SDK that sends incorrect IP data for iOS signups. You don't see that bug at first. What you see is that several geo-based rules suddenly have higher false positive rates on that platform, or a model that uses IP-based features also seems to drift, or maybe analysts complain that events coming from mobile look weird. If you only look at the rule level, you waste weeks recalibrating rule sets. But breaking it down by flow, you quickly realize the logic is fine on web, and the issue is isolated to iOS. Now, keep in mind that this isn't always a data bug. It might also be that a specific flow concentrates many good users who behave differently than your general population, which on its own can drive false positives up. But the point is, once you've bucketed your false positives by actor, do the same by user journey. Which product, which platform, which payment method, which geography, which specific funnel? I guarantee you'll see patterns emerge very quickly. The moment you start seeing patterns by flow, you will almost always run into the same culprit: data quality. Sometimes the underlying fraud logic is actually fine, and yet on a particular slice of traffic, your performance tanks. That's often because the system is operating on corrupted or missing inputs, default IP addresses, where the real IP failed to capture placeholder emails or garbage values, device IDs that reset to null on some OS versions, payment metadata that is never passed through for a certain method, timeout, or integration errors with third-party intelligence sources. You want to locate the exact data fields that are affected in those population segments you've identified. A reliable method I use all the time is to simply group data points by their value and look for values that have a suspiciously high count. And if you think about it, how many times have you done exactly that to uncover fraud patterns just to stumble upon a bug? Once you identify data quality issues, you have some detective work to do. First, we need to remember that often corrupted data points have cascading effects. Corrupted email field would, of course, corrupt email velocity checks that are based on it. So, your first task is to identify all the impacted data points and link those to your misbehaving rules or models. For most organizations, this exercise can prove incredibly hard, but you have a shortcut: your list of top offender solutions from step three.
Chen Zamir
07:41
All you need to do is to cross-reference your corrupted data points with the inputs used by your worst-performing rules. This will save you days of analyzing data skills by associating data issues with solutions. You will be able to complete the last link in the chain, attributing a value to each of these issues, just as you've done with offenders. You have a dollar amount, which you can put on that email bug, at least in terms of false positives. And now it's time to bring it all together. Now you have a complete map, not only of how many false positives you have, but also where and why they are generated. With that, prioritization becomes much more straightforward. You start with buckets that are large enough to matter, under your control, and not caused by data quality issues that you cannot fix. In my experience, in many organizations, these are likely to be a small number of high-volume, high-impact rules, one or two model thresholds that were set conservatively, maybe a specific case management policy that encourages overblocking, a couple flows where your system produces data issues. These are exactly the areas we'll focus on in part three, where we'll get into the mechanics of fixing your decision logic to be less trigger-happy. The work would only be effective because you've done the root cause analysis first and know how to invest smartly. For now, if you've gone from, we have false positives, to, we know where most of them come from, and which parts we can actually fix, you are already well ahead of most teams, and that's a good place to be at.
Chen Zamir
09:07
You.
View full transcript
