53 min
Há algumas semanas, apresentei o modelo para a jornada de transformação com IA. Esta conversa com Lalitha Rao, diretora de riscos da Imprint, é o próximo passo natural. É uma visão real e prática do que é realmente necessário quando uma equipa se compromete a expandir a IA agêntica para além da fase teórica.
Nos nove meses desde que a sua equipa de risco começou a desenvolver seriamente a infraestrutura de IA agêntica, aumentou a carteira de crédito em 150%, multiplicou as receitas por 2,5 e conseguiu reduzir as taxas de perda nas coortes mais recentes. Tudo isto sem eliminar um único posto de trabalho. A equipa de Lalitha não encarou esta iniciativa como um exercício de redução de custos, mas sim como uma forma de criar mais valor para os membros e parceiros da Imprint. E os resultados falam por si.
Se você já está em um estágio mais avançado da sua jornada de transformação com IA e quer ver um exemplo real de como é, na prática, escalar a IA agêntica quando a estrutura teórica é aplicada em campo, esta é a conversa certa.
O que você vai ouvir neste episódio:
- Por que a plataforma centralizada de IA da Imprint, desenvolvida pela equipe do CTO, acabou sendo mais importante do que qualquer agente individual que eles tenham criado
- Como a ARIA, uma agente autônoma de políticas de crédito, analisa dados de bureaus de crédito e dados internos para identificar sinais precoces de deterioração da carteira e expandir a fronteira eficiente, em vez de apenas otimizá-la
- Por que a IA agêntica viabilizou algo que uma consulta SQL ou um modelo independente não conseguiriam fazer: processar várias fontes de dados em diferentes formatos e integrá-las em um fluxo de trabalho único, coerente e replicável
- Como um agente de mineração de resíduos identifica continuamente erros do modelo, cria novas variáveis e retreina modelos usando testes fora do período para evitar o sobreajuste
- Por que a Imprint nunca mediu o sucesso pela redução do quadro de funcionários e o que construiu, em vez disso, com o tempo liberado da equipe
- Por dentro do Project Shield, um conjunto de dez agentes antifraude que opera por seis horas, custa cerca de US$ 300 em tokens e evita centenas de milhares de dólares em perdas decorrentes de fraudes
- Como um repositório compartilhado de competências e uma definição clara da responsabilidade pelos dados permitiram que uma equipe de riscos sem formação em engenharia criasse e mantivesse agentes prontos para produção
- A avaliação franca de Lalitha sobre desenvolver internamente ou comprar, e os fornecedores específicos dos quais a Imprint ainda depende, incluindo Sardine, LexisNexis e Proof
- Os verdadeiros obstáculos que a Imprint enfrentou ao longo do caminho, propostas excessivamente extensas, a falsa sensação de certeza nas análises geradas por IA e como criaram mecanismos de proteção para resolver ambos os problemas
Você deveria ouvir este episódio se:
- Já avançou na jornada de transformação com IA e quer entender, de forma concreta, o que é realmente necessário para ganhar escala além da estrutura inicial
- Lideram uma equipe de risco, fraude ou crédito e estão tentando decidir quais casos de uso de IA agêntica devem priorizar primeiro
- Estão avaliando se devem desenvolver ou comprar agentes de IA e querem um modelo mental claro para entender em que situações os dados proprietários mudam essa equação
- Temem que expandir o uso da IA signifique reduzir a equipe e querem conhecer um contraexemplo real
- Quer detalhes práticos sobre como capacitar profissionais que não são engenheiros a criar e manter agentes de IA prontos para produção
Notas do episódio e principais conclusões
Construir primeiro a infraestrutura de IA agêntica
Cerca de 80% do trabalho necessário para ampliar o uso de agentes de IA na área de risco consiste na infraestrutura de dados para IA agêntica: uma plataforma de IA centralizada sobre a qual as equipes de fintech possam realmente desenvolver soluções, com acesso padronizado a dados e ferramentas. No caso da Imprint, isso significou uma conexão MCP com Snowflake, Notion, Linear e Slack, além de uma documentação clara sobre a linhagem dos dados. Um repositório compartilhado de competências para agentes de IA, revisado por pares e com controle de versões, foi o que viabilizou uma iniciativa de desenvolvimento de agentes de IA conduzida por profissionais que não eram engenheiros, tornando a equipe de risco a segunda maior usuária interna de agentes de IA na empresa, atrás apenas da própria equipe de engenharia. Essa combinação representa, de fato, a lista completa de pré-requisitos para desenvolver agentes de IA que devem ser levados a sério antes de qualquer outra coisa ser criada.
O que o ARIA possibilita e outras ferramentas não conseguem
A ARIA, agente de inteligência de risco de aquisição da Imprint, atua como analista autónoma de políticas de crédito, analisando em paralelo o desempenho da carteira e os dados das agências de crédito em todos os programas de parceiros, com vista a otimizar a avaliação de risco de crédito através de IA agêntica e a detetar sinais precoces de deterioração antes de estes se traduzirem em incumprimentos. Um segundo agente de IA, dedicado ao risco residual, trabalha em segundo plano para identificar falhas nos modelos, gerar novas variáveis e gerir o retreino dos modelos de IA agêntica através de testes fora do período de amostragem, nos quais os agentes de IA se baseiam para evitar o sobreajustamento. Em conjunto, estes dois agentes não se limitaram a levar a Imprint a um ponto mais otimizado da fronteira eficiente da IA agêntica. Expandiram a própria fronteira.
Sem redução do quadro de funcionários, melhor qualidade de vida
Este foi, na minha opinião, o ponto mais importante. A Imprint nunca apresentou o trabalho da equipa de risco na implementação de agentes de IA como um projeto de eficiência ou de redução de custos. Os analistas de fraude, que antes passavam as primeiras horas de cada dia a eliminar falsos positivos óbvios, agora dedicam esse tempo a procurar padrões de fraude verdadeiramente novos, o que representa um avanço real na redução de falsos positivos com agentes de IA. Segundo a própria equipa, a sua qualidade de vida melhorou não apesar da implementação da IA, mas graças a ela.
O Project Shield amplia o conhecimento institucional sobre fraudes
A equipa antifraude da Imprint criou um conjunto de dez agentes, denominado Project Shield, que funciona durante cerca de seis horas de cada vez, custa aproximadamente 300 dólares em tokens e trata da geração de regras antifraude por IA, permitindo poupar centenas de milhares de dólares por ano. O conjunto inclui um agente que propõe novas regras e funcionalidades e outro agente, criado especificamente para assumir uma postura cética, que verifica a precisão e as taxas de falsos positivos antes de qualquer solução chegar à produção. Os primeiros problemas dos agentes de IA em produção, como avaliações geradas por IA que pareciam mais confiantes do que deveriam, levaram a equipa a adotar práticas concretas de medição do grau de certeza dos agentes de IA e de gestão dos riscos associados à explicabilidade, apresentando em cada resultado um nível de certeza e o raciocínio subjacente, para que um revisor humano pudesse identificar exatamente onde faltavam dados antes de tomar a decisão final.
Desenvolver ou comprar — e a mudança de mentalidade mais ampla
A estrutura de Lalitha para decidir entre desenvolver ou comprar agentes de IA é simples: desenvolva quando os dados proprietários criarem uma vantagem competitiva real e compre quando um fornecedor já tiver escala e conhecimento especializado para fazer melhor. É por isso que a Imprint continua usando Sardine, LexisNexis e Proof em conjunto com seus próprios agentes internos. Seu conselho final resume bem essa ideia e mostra, de fato, como escalar agentes de IA na gestão de riscos sem perder o foco. A maioria das equipes começa essa jornada perguntando como pode se tornar mais eficiente em sua posição atual na fronteira de eficiência, inclusive nos primeiros casos de uso de personalização com agentes de IA em fintechs. A maior oportunidade está em deslocar completamente essa fronteira e manter a flexibilidade em relação ao que já foi desenvolvido, pois o ritmo das mudanças faz com que quase nenhuma solução continue sendo a melhor por muito tempo.
Conclusão final
Se há algo que eu gostaria que todos os líderes das áreas de fraude e risco levassem desta conversa, é que ampliar o uso da IA agêntica nunca foi realmente uma questão de substituir pessoas. A equipe de Lalitha não diminuiu. Ela passou a ter a oportunidade de realizar um trabalho que realmente importa: identificar novos padrões de fraude em vez de descartar falsos positivos e pensar na estratégia de crédito em vez de preparar relatórios semanais. Isso não é um efeito colateral de fazer esse trabalho bem. É justamente esse o objetivo.
Recursos e links
Ainda não quer encerrar a conversa sobre o meu assunto favorito — e, espero, o seu também? Assine a newsletter The Saturday Fraud Strategist.
Conecte-se com Lalitha Rao | LinkedIn
Diretora de Riscos, Imprint
Conecte-se com Chen Zamir | LinkedIn
Apresentador do The Saturday Fraud Strategist
Ajudando fintechs a criar defesas mais inteligentes contra fraudes
Coautor de “The Fraud Fighter’s AI Playbook”
Episode transcript
Lalitha Rao
00:00
We have entirely rebuilt our risk systems to be geared around AI.
Chen Zamir
00:07
Did you guys actually downsize the size of the team?
Lalitha Rao
00:11
That's not sort of been our philosophy around this. The quality of the life actually uh on the team is one thing that you know has improved so much. Are actually saving us hundreds of thousands of dollars in fraud losses. And I think that is actually you know 80% of the game, is to
Chen Zamir
00:33
Welcome everybody to another episode of The Saturday Fraud Strategist. With me today is Lalitha Rao. Uh Lalitha, you're the chief risk officer at Imprint, correct?
Lalitha Rao
00:44
Yes, that's correct. Excited to be here and to join you today.
Chen Zamir
00:47
Yeah, the pleasure is all mine. Uh Lalitha, we had a call a couple of weeks ago. We talked a lot about agentic AI, which will be the topic of our conversation today. Got me very excited. We geeked over it for about 30 minutes. Today we have a bit longer than that. So, I'm very excited about our uh conversation today. Before we dive into it, maybe you want to share a bit about yourself and your background and what led you to to Imprint.
Lalitha Rao
01:13
Yeah, absolutely. Uh I've been in the risk field now for about 16 years. Uh spent a bulk of my career at large banks like Capital One. And more recently at fintechs like Stripe and Square. Uh joined Imprint about uh now 19 months ago to lead their risk function. Uh at Imprint we are a loyalty and payment platform for brands to deepen engagement. We work with a number of large brands. Uh like H-E-B Booking.com, Crate & Barrel, and Rakuten just to name a few. And really my remit on the risk team is to enable safe and resilient growth. Uh to do that while staying within our risk appetite. And to create best-in-class member experiences to really deepening engagement for our partners.
Chen Zamir
02:16
Yeah. Well, and you have like uh very small partners with almost no fraud, right? That uh that's how it sounds like.
Lalitha Rao
02:25
Fraud is definitely you know a a place that is evolving at uh such a rapid clip. And uh you know I'm excited that we have a really strong team and a really strong Data driven approach uh to continue to make strengthen our fraud system so we can really serve our customers our customers with the least amount of friction as they interact with our experience.
Chen Zamir
02:50
Before we deep dive into it, we had Brian Davis from Safeguard on the Pod a couple of weeks ago. And Brian and I talked a lot about his framework for uh the transformation journey, the AI transformation journey that he sees uh in mainly FinTech and I'm guessing banks as well. And I think that conversation was you know very theoretical. And very kind of like you know like high level framework doctrine methodology. And I think you know first of all if you haven't listened to it yet, go and check it out. But I think if you did listen to it and you liked it I think this conversation is exactly the next step. Because when we talk Lalitha um you know you your experience with rolling out agentic frameworks at Imprint is super interesting. The easiest way to start I'm guessing would be to just plainly ask what did you guys build? So today when you're looking at your kind of like agentic landscape what do you guys have in place?
Lalitha Rao
03:59
Yeah, I was going to say I definitely tuned in for uh the podcast with uh Brian. Really enjoyed the episode. I think you know such a great framework uh that he shared. That you know especially for uh you know as uh different companies banks and fintechs are evaluating how you know they go on this journey. I think it was just like such a great session. Uh so uh you know excited to kind of um continue to build on that as well. Uh so uh you know I'm this is actually such a topic that's like just so near and dear to my heart. Because we have entirely rebuilt our risk systems to be geared around AI. Uh the day in the life of a risk analyst today looks so different from it from what it looked even 6 months ago. If you think about it, uh there's, you know, a lot of time spent on, you know, creating reporting, uh, aggregating data from different data sources, getting reporting together, creating, uh, reporting around SLAs's around ensuring that uh, we have the right governance and model validation documentation. And uh, now we've been able to just create so much operational efficiencies around a lot of that work. That it just frees up the team's time to do really strategic work. That's actually moving the needle uh for our members and our partners as well. Uh so it's been really exciting. Uh if you think about uh you know what we've built, uh I I want to also call out that you know one of the reasons we've been successful at uh you know really leveraging AI in a manner that is I think becoming a very durable and compounding advantage for us over time is because our you know chief technology officer and our AI team. Really created an operating platform form across the company. That enabled you know the teams to be able to use the AI in a manner, where you know they had access to standardized data. Uh they had access to standardized tools through an MCP connection. You you name it it was like snowflake, notion, uh you know linear, slack. Uh and as well as you know documentation around data lineage. So that when uh you your agent is looking for uh definitions and data, they know exactly where to go. And there is a canonical source of truth for where to find information. And I think that is actually you know 80% of the gam, is to have that platform and structure in place. And I think that really enabled, the you know different teams and the risk team to go deep on their domain main expertise. Uh so like I said, you know, we've really changed our workflows across credit, fraud, collections, disputes, uh you name it. And it looks very different today. Uh we've also built a suite of very specialized agents for our highest leverage work, uh that actually frankly are doing work that it would take teams of uh data scientists to do. Uh so excited to tell you more about that.
Chen Zamir
07:38
Yeah, please do. I mean, first of all, it's super interesting uh to just hear you echoing, you know, one of the main things that Brian said in in our conversation. That, you know, it starts with setting up the infrastructure and setting up the connections and the data flows. Um and it and it definitely sounds like, you know, this is not at Imprint at least that's not like a risk thing. That's technology across the board. Uh so super interesting. Um but yeah, I I would definitely love to hear more about what you've built. And understand like is it more around trying to streamline specific use cases or workflows? Or is it more about trying to just pick up the repetitive work and try to automate that away across the board? Like how did you how did you guys approach it?
Lalitha Rao
08:30
Yeah. No, I think that's uh a really good question. And you know as you know when we started on this journey I thought you know that's where a bulk of the opportunity was. Uh which was you know being able to gain a lot of efficiencies, automating a lot of you know repeatable work that can be tedious. And we have definitely done that. But you know what we found is uh that the gains go so much beyond that. Uh and like I said we've built you know some very specialized agents, that are helping us do some of our highest leverage work. So I'll give you an example of this. We've built an agent that we uh have named Aria. This is our acquisition risk intelligence agent. Uh so think about Aria as an autonomous credit policy analyst. And what uh this agent enables us to do is it scour our portfolio, looks at performance based on honest data, but is also looking at data from the bureaus. To find early signals of deterioration, that have not yet manifested on our side in the form of delinquencies. And it comes up with recommendations for policy changes that uh bring us back to the most optimal point on our efficient frontier. And it runs this in parallel, across all of our partner programs. In the old world, you know, this would take an analyst, you know, several days to do for a single program. And now Aria runs these recommendations. Uh gives us a full audit trail. You know we are able to see the reasoning, the logic, we can see the source of truth in the form of the SQL scripts, the data sources. Uh understand you know the really the uh the reasoning behind the recommendation. And we surface it to a human who's then empowered to make the decision, on whether this is a you know a change that we should deploy or not. And it has dramatically improved the velocity with which we're able to rate on you know early signals that we are seeing in our portfolio. And continue to improve our underwriting. Uh in fact we've already seen the results from this in terms of being able to expand our approval rates, expand access to credit to more of our members. So like like I said, I think the biggest insight for me is just you know, uh the value here is in terms of like you know not just moving to a more optimal point on that efficient frontier. But really pushing that frontier out. And you know we've seen several examples of this. I think this is one example of that uh that uh you know that kind of work.
Chen Zamir
11:29
That that's super interesting. I I wonder you know what was when you look at Aria as a use case why was an AI agent the solution here and not for example I don't know um an SQL query, or some sort of a tool. Or some sort of machine learning model? Like how did an AI agent unlock potential that you couldn't unlock with other means?
Lalitha Rao
11:57
Yeah I think what the agent here does is you know it has all the context of our uh of Imprint embedded and Imprint risk management embedded. And it can actually use all that context. So it actually un knows exactly what our current credit policy is. It has access to you know the data sources around performance as well. It understands you know what are our overall guard rails in terms of our risk appetite. What are our you know partner commitments as well. And it's able to run that optimization, uh and that tradeoff between uh you know risk and uh reward. Uh in a in a continuous manner, across all of our programs. I think the other reason this was especially impactful in our context is, you know we have brands across so many different verticals. You know we have very large partners in the grocery vertical. We also have very large partners in the travel vertical. And all of that means we have to customize our uh uh credit and fraud underwriting to the context of uh our members in those verticals. And given that you know Aria is actually able to do all these optimizations with all that unique context in parallel, and you know generate a report. Like I said we were we were doing the same things we were just doing them much slower. And it's allowed us to you know iterate so much more rapidly uh continue to increase the data sources we are using as well. And just drive better outcomes for our portfolio.
Chen Zamir
13:42
I think that you touched a very important point here and I'm trying to kind of like generalize it also to the audience. Because they are probably thinking about like very different use cases. And it goes back to something that you said before. That you know you you think about AI and you think first about automation. But then you find out that it's actually not about automation. And I think here if if I get you correctly, the the main benefit of using AI or the thing that agentic AI unlocks rather. And um or or as opposed to other technologies is the fact that you can query multiple different data sources in different environments and in different formats and digest them all into like one coherent repeatable
Lalitha Rao
14:30
Yes
Chen Zamir
14:31
Workflow.
Lalitha Rao
14:32
I I think that was very well articulated. Uh you know I'll also say that you know we we found this kind of leverage uh on in a few different areas as well. You know one of the things at Imprint that you know we are very excited about is, we sit very uniquely at the intersection of both uh the customer and the merchant. And what that means is we have access to very interesting data, that is uh you know very unique and proprietary here in terms of understanding uh for instance if you're looking at an application from a Rakuten customer, we can actually see their history with Rakuten, how long have they been a member with Rakuten. How have they been you know transacting over time. What has been their trajectory what does you know the basket composition look like, and that is such great signal that is just completely orthogonal to the information that's available on a traditional credit bureau report. Uh you know it's like a fraud team's uh dream uh data right, because you can steal an identity uh but it's hard to like really mirror a full digital footprint with a brand. So you know we use uh this proprietary data to build custom models uh across uh you know many different use cases, be it you know uh risk. But even outside of risk, spend. Attrition. And so on, and we what we've all done is we've built an agent that we call our residual mining agent. What this agent does is it is continuously looking at the errors in the model. Where is the model failing in its predictions. And continuing to optimize the model. Uh and this applies across our modeling suite. It finds new features. It actually is able to like look at the underlying data, create new features as well. And uh find opportunities to continue to improve our models.
Chen Zamir
16:33
These features are handed over to the data science team to actually develop. These are like leads for new features.
Lalitha Rao
16:40
Exactly. So that you it's basically and it goes even beyond features. Because it'll actually retrain the new model as well. Uh and show you you know that this is sort of uh you know the new features. This is the retrained model. It'll actually do this in a manner where you know one of the things we were very worried about is it going to just overfit to the data. And uh generate a lot of biases in the predictions. So we actually you know created the agent in a manner where it does uh what we call out of time testing. So you build the model on a different time frame, and you test and validate it on a completely different time frame. So you know we did we used that technique. We did an out of sample testing as well. Just to make sure that the new model that's been generated is not overfit. We understand the benefits of the new features. And you know we found this to be very uh impactful. In fact for uh you know one of our core portfolio management models, I think about a fifth of the features are now found from uh this technique.
Chen Zamir
17:48
Super super interesting. Um yeah and I and again I think that these are exactly the kind of like super advanced use cases. It's really fascinating to see a team that already uh have that running. So, you've built all of these agents, right? And you said so yourself beforehand. Uh you know, we got rid of a lot of work and our uh analysts, or our our our team members now have a lot of time to do the high leverage work and so on. Did you guys actually downsize the the the size of the team? Because I think that you know in today's world when we're speaking about adopting AI in most cases, if not all cases, we're speaking about it in the context of uh creating efficiencies in the budget. And not only in um in the team uh the team's work. So like have you've seen this uh uh team size reduction?
Lalitha Rao
18:47
No actually uh uh that's not sort of been our uh you know philosophy around this. What we have seen is you know that uh we have looked at this from the lens of what else could we be doing for our members and partners with you know the same team in place. And the results here have truly been remarkable. Uh like just to give you a a sense like you know over the last year year and a half, uh we have grown our loan book by about 150%. We have, you know, increased revenues by 2.5x. And we've done this while actually reducing our loss rates, on a uh for all of our newer cohorts. Like our newest cohorts are about, you know, 40% lower in terms of early delinquency rates and loss rates than they were a year ago. So we are actually seeing all the the great work here in terms of being able to iterate more rapidly. Uh you know detect uh opportunities for optimization sooner, but also implement this changes so much faster. Translate into such tangible results. That for us it's just been you know what all are we now able to accomplish that we were not able to do uh with the same size of team. Uh you know I'll give you an uh another example here to make this just really real. You know I think our fraud team is you know again uh you know just like uh we have a lot of thought leaders on our fraud team with a lot of experience from their years of like you know uh fighting fraud at many different institutions. And what they've done is they have actually found a way to really put all that context, uh that you know and all that uh experience as well, and leverage uh that in a manner that really scales. They've we call this uh you know project shield. And uh this is a suite of 10 different agents that run in tandem. And they're looking at our you know most recent fraud disputes. And finding newer rules, uh to actually uh be able to better mitigate that fraud. Uh so this is something that we are we run for about 6 hours at a time. And uh it uses about I think $300 of tokens. And the rules that are generated from this uh are actually saving us hundreds of thousands of dollars in fraud losses. And inbuilt in that agent suite is you know there's an agent that's actually proposing new features, that's proposing new rules. There's an agent that is specifically designed to be a skeptic. Uh that is actually you know evaluating those rules and seeing you know what is the precision of these rules. Is the false positive rate here actually meeting the hurdle. You know is this something that is, you know was working a few months ago but is not capturing the more recent trends. So you know it is uh and it it's so amazing to see all that rich deep context and experience. Um also be reflected uh in our agentic work Now.
Chen Zamir
22:13
That is so cool. I I it's interesting. And I I I I wanted to ask you, you know, if if you're not measuring the success of the project by the amount of uh headcount that you managed to cut, which I'm glad that that was never on the table. Um I'm guessing that you know in some cases measuring the the value that you create is quite straightforward. Taking you just described it, right? Taking um a list of rules that were created for $300 and a bunch of uh you know days that we had to develop this. Uh against the hundreds of thousands of dollars that you save yearly in fraud, that is pretty straightforward. But I'm guessing that there are some effects that might even be more strategic. That because you talked about it earlier when you talked about Aria as a as as a as um as a capability that lets you push the portfolio and may maybe accept clients that you wouldn't have otherwise. Uh how do you how do you measure these very tricky kind of like value ads?
Lalitha Rao
23:29
No, I I think that's like a uh you know question that we think about deeply as well. You know, I think like for us, we've been able to definitely quantify it because, you know, like I guess in the form of fraud losses, in the form of our delinquency curves for newer cohorts, uh, you know, looking better and demonstrating better credit quality. But beyond that, what we've been able to do is, uh, you know, we're on this journey where, uh, like I mentioned, we have access to very rich data and context about, uh, our, uh, customers. And we're on this journey to really deepen uh personalization in every interaction with the customer through the life cycle. And uh you know that what that does is of course it deepens engagement, uh it uh you know also helps us optimize lifetime value. And expand monetization uh both for us but also our partners, and I'll give you a great example of this. Like you know and this is you know something we is only possible with really leaning in on AI. So, you know, you think about it. We have a, you know, Booking.com customer. They're in, you know, the the checkout flow and they're looking to book uh a flight. And uh you know, the flight would bring them close to their credit limit. You know for us what we are building you know through all these tools is, how do we in that moment in that high intent moment actually give the our customer a credit line increase. So that they are able to not only book their uh you know flight uh with Booking.com but they can also book their hotel they can also book their rental car and uh and you know with some room to spare. So I think that's the kind of work that uh you know we are uh driving here, is uh just and we you know see that already translate into such tangible value. But we expect that to just continue to compound over time.
Chen Zamir
25:30
[Hey folks, I want to take a quick break to speak about today's sponsor, me. If you're finding this useful, do me a small favor, like and subscribe. It really helps with the algorithm. And if you're not already on my newsletter, the Saturday fraud strategist, you should definitely check it out. Every Saturday, I break down fraud trends, real cases, and strategies from my own experience of building and operating fraud prevention systems. Whether you're new to the space or a seasoned practitioner, I'm sure you'll find it interesting. Check the link in the description. Takes 2 seconds. You'll thank yourself next Saturday. Now, back to the video.]
Chen Zamir
26:03
How do you uh measure and monitor your your agents? Like in what kind of ways? I'm guessing, you know, are these agents uh up or not? Uh are they costing me $300 per run or $500 per run? But what are the kind of monitoring frameworks that someone should have in mind when they are uh uh rolling out a ticket at scale?
Lalitha Rao
26:32
Yeah, definitely. And I'll, you know, this actually goes back uh in some ways to what I talked about is that centralized platform that should be in place as you kind of roll out and you know build more specific domain specific use cases out. So for us uh you know what uh we have uh like the central uh AI and tech teams have enabled is they you know all of the agents and the skills we've built live in a you know in a centralized GitHub repository. They have you know very clear owners. Uh there's a you know rigorous process, in terms of uh when you want to change any make any changes to code. In the form of you know peer review, uh review of the code as well. Before any changes are actually productionized. So I think that is like you know just part of our machinery and exists as well. What that also means is that you know all the data sources we're using have you know like I mentioned uh you know canonical definitions. They have clear owners uh who are accountable for uh you know is the data consistent? Is it uh timely? The freshness of the data as well, and uh and we have audit trails around that as well.
Chen Zamir
27:58
And and these owners are like fraud odd specialists on the team? Or or are these engineers that build the agents according to the specs that the risk team gives them?
Lalitha Rao
28:11
So we have actually uh different use cases. So if you're looking at data uh that is in, so we have the data on the raw production the raw data in our data mods is owned by our centralized data team. So they are ensuring that. But then there's you know tables that the fraud team has built, on top of that uh the data mart. And there you'll have somebody with that fraud context that is the owner. They have actually you know you know what is uh how do you define a dispute rate? You know what exactly is the numerator? What exactly is the denominator? Making sure that all that semantic context is also well documented, and uh that would be owned by somebody who has that domain expertise on the risk side.
Chen Zamir
29:00
On the data scheme I understand and I agree. I think that's the right uh the right approach. I But who actually builds out the agents themselves? Is it the risk team or is it like are they only given the specs?
Lalitha Rao
29:13
Yeah. No, I think the risk team actually builds the agents themselves. We have uh you know actually across the team, we are building agents. And uh like I said we've you know the the central team has enabled us in terms of you know uh the the best practices, and the tools. Before we can actually you know push something to production. We have a peer review uh of the code as well. Uh by an independent data scientist. Uh it you know it could be a data scientist on our central data team. It could be another data scientist on the risk team. But the agents are developed by the risk team.
Chen Zamir
29:51
So let let me ask you about this. Because I think you make it sound very straightforward and trivial. But I like I don't know from my experience, this can be like a very very hard shift right. Because in order to build, and deploy, and maintain code. A agents are code. You need to have like a very very different mindset. Than what risk specialists usually have. Uh so how do you how do you deal with that? Because like it sounds like what you guys have been, uh like you know how you've rolled it out, sounds like very professional it sounds like engineering grade. How did you manage to get your team, who are not engineers, to actually comply with these best practices?
Lalitha Rao
30:45
Definitely. So I think one thing I will call out is that you know the the team, uh the risk team at Imprint, is a a team that is uh you know we've built a team with very strong data chops as well. So we have uh a lot of data scientists on the team as well. So they are also we have our very own machine learning function within the risk team as well. So I think what we have is a set of folks that have worked in risk, but are highly specialized in terms of their ability to work with data to understand uh you know data structures to be to be build to even build features over time. To train models as well over time. And I think that actually has been a big part of why I think the risk team was one of the first functions, after engineering, to really adopt the the AI wave. So uh you the risk team is uh I think the number two user of AI today at the company after engineering. And I think uh a lot of that is driven by the fact that uh you know the team is uh, has a more uh you know uh has more technical experience as well. And that also uh enabled uh adoption more,
Chen Zamir
32:04
Yeah, still kudos I've seen very strong uh risk teams. I don't think it is that uh easy uh even when they have uh technical chops. Um what do you think are the prerequisite for a non-engineering team? So they can actually build and maintain an AI program that doesn't you know like degrade, and rot very quickly, and end up just being a waste of time and budget for everyone.
Lalitha Rao
32:34
Yeah, I think uh that's a that's again a question that we think about deeply. And I'll actually uh add to one of the other uh you know things that's been very valuable in terms of uh adoption for within the risk team. And really scaling this program. Is what again I go back to the what the central AI team has enabled for us here right. Is they have created what we call a shared skills repository across the company. So that's a you know set of skills that has gone through the entire uh you know rigorous process. Of uh code review, peer review, uh making sure that everything is version controlled in GitHub as well. And that's available for different team members to build upon. You know, we actually have skills that also call out like these are the gotchas, right? As you're building your skill, watch out for this. And uh make sure you're kind of adding that into your context as you build more agents as well. Uh so uh you know that that's where I think the the foundational work, that uh the central tech teams have done have really enabled this. On the data side as well. You know you can imagine that if you are trying to build an agent that is needs to grab data from your data mod, it it can be uh it it's not a straightforward task right. Like which table should you use? Which field is the correct field? And uh you know you can get results that are just not correct. But are stated in a very confident manner. Which is actually uh you know which is uh not an ideal outcome at all. So I think the fact that the data team is also putting in so much time and investment into really creating uh that uh that semantic layer. Where uh you know we have well- defined data tables, with canonical definitions. So that the the agent knows exactly which data mod to query which field to look at. Uh I think all that really was a big part of why you know the risk team, which was you know uh which has had a lot of technical chops, was enabled to be able to kind of uh to build on this.
Chen Zamir
34:58
You know it it all comes down to the infrastructure right. I mean where we started, and it's uh and it's very fascinating to see the difference the difference that it can make on how well you and execute on your program. Since we started touching on it. Let's talk about uh like the build decision here. Because you know there the there is no lack whatsoever of companies out there that offer you know agents including for risk. You know like yes, or they also have, 35:33 you decide to build it on your own. And I think uh that I would say I I see a lot of teams that go down this route. Actually probably all of them if most of them, right? Um they they go down this route because um I think it's very it's very sexy and very enticing. Everybody wants to play. And it also sounds um relatively uh uh easy. When did you guys start actually? When when was when did you guys start with this uh journey?
Lalitha Rao
36:06
So we started this uh I think in Q1 of this year is when it really took off.
Chen Zamir
36:14
Wow. Wow.
Lalitha Rao
36:15
And started getting a lot of momentum. But actually even in Q4, uh you know is sort of when the uh we started to kind of explore the early use cases. So I'll give you an example of an early use case right. The first thing we did is we have a weekly business review report. That needed contribution from the credit team, it needed contribution from the fraud team, from the collections team. And then everybody would put this like giant you know PDF together and ship it. And that's the first piece of work that and use case that we used. And and it demonstrated you know now it's a click for us right. And you know we've also built it in a manner where that the output is deterministic. There's no uh you know we've uh there's uh we can replicate it 100% of the time. And it has just saved so much time for so many teams. That it just made that use case like so obvious and it got that momentum going. So I think we did that actually in Q4 of last year itself. And then you know in Q1 like I said is when we really started to kind of think about what does this mean for how we operate. And it's just been you know a game changer since then.
Chen Zamir
37:32
Yeah
Lalitha Rao
37:33
Like quality. Yeah, I was going to say the quality of the life actually uh on the team is one thing that you know has improved so much. Uh because they're doing the work that you know they really want to do. They are spending time engaged on uh you know problems uh that are uh you know thought provoking. Uh that are strategic. Uh you know we have uh an ops uh agent who is uh saying that you know they don't miss the fact that you know they spend the first few hours of their day reviewing obvious like false positives uh that now they spend way less time on.
Chen Zamir
38:13
This is fascinating because I think and and this is such an important learning. We really are uh indoctrinized to see AI as a threat, as an enemy, and as the thing that would take our jobs. But what I'm hearing from you is that not only that it didn't take anyone's job, despite you being like very advanced. Uh but not not only that but it also makes your lives better.
Lalitha Rao
38:44
That's actually been really uh you know fun to see. Like you know so much of the work that was uh you know our fraud uh reviewers, now can spend time actually finding newer patterns of fraud. Uh focusing on emerging threats. And uh that's just like you know uh of course helps us in terms of uh our uh improvement in our performance and our frameworks. But the the team uh also enjoys add so much more.
Chen Zamir
39:16
Uh, incredible. Incredible. Um, I I think this is, you know, if there's anything for you guys to take out of this conversation, I think this is it. Um, you know, we can speak about how to roll out AI, or how to build it or whether to build it or buy it for hours upon hours. But I think this goes to you know like a very key part, which is motivation versus. Also because we're talking about the transformation journey about a change management process. Uh about resistance. And I think that like seeing this example of of uh Lalitha's team being on one hand so advanced and on the other hand having like pure gains, not only on the business side, but really on the individual quality of life uh of the employees, that that I think is the most important takeaway. And I'm I'm already in the summary, but really this is like I'm I'm stopping in the middle just to highlight this takeaway because I think it is so so important.
Lalitha Rao
40:20
No, I I think that's super resonates. I was going to say, you know, it's it's not come without hiccups. We've definitely had our hiccups along the way.
Chen Zamir
40:30
Tell me more. Tell me more.
Lalitha Rao
40:32
Uh you know so I I'll tell you like for instance as we uh you know were rolling initially had a lot of adoption. Uh on the team what we saw is that proposals were getting much longer. Like three times the length of the proposals. But uh you know the the number of charts in the proposals had come down by half. Or actually you know in many cases we didn't have charts because you know claude works better with tables. So uh you know one thing that you know we were worried about is is like you know claude doing too much of our thinking for us. And how do we make sure that for all new proposals we have standardized best practices. You know it is uh you know good to be able to look at visually at charts and see whether results are intuitive or not. And uh longer proposals are not necessarily better quality as well. So I think what we did is once we saw that you know we as a team recognized that. And we were like okay how do we make sure we create best practices around uh proposals in this new era.
Chen Zamir
41:39
A kind of a human in the loop right. It it is not as um formalized as in in an investigation process. But it just shows how in the end it's very very hard to separate humans from AI. And how like and even if something is automated, yes it is automated but it is still monitored. It is still uh assessed and audited by humans. And that's something that you cannot really take away or take out of the process ever.
Lalitha Rao
42:10
Totally
Chen Zamir
42:11
You can minimize it. Yeah.
Lalitha Rao
42:12
Yeah. Definitely. Like another example here was you know as we were looking at our operational reviews that were you know again you know had uh were quite manual sometimes. It would entail people you know our expert reviewers going in opening up five different tabs. Gaining context from each tab. And trying to kind of make that decision. Uh and then document that decision. And uh you know obviously this was a use case that was just ripe for uh reinvention. And we definitely took that opportunity. But what we saw is our first pass at it, you know we had agents write up uh the review in a manner where uh you know it looked very you know comprehensive, very polished. Almost a sense of false certainty in the outcome of the review. So as you know we we saw that, we were like we went back and reflected on you know what's the the right way to actually make this something that is a a really good uh tool for our uh agents. But also with has the right guard rails. So we actually started creating uh uh within the output, uh we started measuring certainty associated with the decision making. You know where did you what data sources were behind the decision? Where was there uh you know missing data? Or potentially conflicting uh sort of uh set of ideas as well? And exposing that in the output. So that when the reviewer looks at it they can look at all those areas and really bring in that human judgment. And make a better more informed decision.
Chen Zamir
44:00
I started asking you before about build or buy. And you've you guys have been on this insane journey for less than a year. So rewinding to Q4, help me kind of like understand your um frame of thought when it comes to whether you now invest in building it yourself or whether you try at least to some extent um to to outsource and maybe uh buy some uh agents from from from vendors. How did you guys approach this decision?
Lalitha Rao
44:32
Yeah. No, I think for us the way we approach we are all you know for uh buying as well. Uh where it makes sense. And we do that as well. So for our you know mental model here is, you know where we have access to proprietary information, uh that is part of our mode. Uh that allows us to uh really improvise the workflows. That's when we hone in on building. So uh but otherwise we're all about you know buying as well where uh uh you know other uh providers have the advantage of scale, have uh the advantage of uh potentially more subject matter experience in that domain. So a great example of this is we use Sardine for all of our transaction fraud uh decisioning. We use their rule engine. We use their feature store as well. And uh uh we also you know purchase external data from a variety of data sources. You know we uh we use uh uh you we use a host of vendors. We use Lexus Nexus, Prove, uh you know Sardine of course. And uh on the uh credit bureau side as well. To complement the data that we have internally. And like I mentioned the very unique uh data we get access to by virtue of our partnerships to build proprietary models. Uh because that truly is part of our competitive mode. So that's sort of how we think about the build versus buy decision. You know some of these uh agents that we've built are really specialized workflows that have so much of our unique context. In the form of you know what is our credit policy, what is our comprehensive set of fraud rules, uh you know what are our requirements for our partners and our risk appetite? So that is so unique to us. That we are able to really bring in that additional layer of intelligence there. And drive uh you know more value and incrementality.
Chen Zamir
46:39
I I wonder if you guys use also agents that you've, you know from external vendors, without going into names uh more from a perspective of if you do that like how do you how do you manage these two programs simultaneously side by side.
Lalitha Rao
47:00
We actually do use external vendors. A great example of this is you know when we uh look at things like even uh external information. Any uh as we look at entity resolution within our KYC workflows. Uh and uh look at adverse media, screening and things like that. There's you know a lot of vendors that have really figured this out, in a very systematic manner. And we definitely lean in on those agents as well. So I think it really comes down to that, you know, where do we have that uh proprietary data and unique context that gives us that advantage? Versus where someone else has already figured this problem out, and has uh you know done that at scale.
Chen Zamir
47:41
Yeah. And it it sounds like between the lines and maybe maybe I'm wrong here, but the the use cases where you saw a lot of success is really where you tackled very heavy, very complex internal processes. That in the end vendors cannot cannot really touch, and definitely not throughout the entire chain.
Lalitha Rao
48:07
Yeah, I I think that's actually well summarized.
Chen Zamir
48:10
Yeah. No, that's uh that's very interesting. Um okay, so Lalitha, maybe last question. I'm sure that there are a lot of uh like fraud leaders that are listening right now, you know, thinking, okay, yeah, yeah, yeah, agentic. Okay, I need to do something with it. Um what would be the one piece of advice, uh you know, with this three and a half Qs, uh on your belt. Um for for leaders who are just starting this journey,
Lalitha Rao
48:47
I think the the you know biggest takeaways for me are, like I said going into this, you know I was really thinking about uh you know how do you uh optimize and make your uh move to a more optimal point on the current efficient frontier uh by bringing in more operating leverage and efficiencies. But really I think the opportunity is just so much bigger to just completely shift that frontier. You know like we are doing with being able to just bring in so much more personalization. In every interaction with our customers. And that is you know so valuable in terms of deepening engagement. Uh and uh expanding monetization as well down the line. So I think that is ultimately, I would say you know it's important to have that perspective. Uh aside from that, I would say you know the way things are changing, the pace of change is is you know really really rapid as well. So I think not being too attached to anything you build, is and expecting that to continue to change. And really you know uh again I'll say this it's like you know uh it's it's more about driving those outcomes. And continuing to iterate on uh how you kind of adopt uh these tools to drive better outcomes.
Chen Zamir
50:09
Lalitha , we covered so many things. I want to maybe rehash some of the takeaways, uh because you said some very eye opening uh uh uh things, or I collected some very eye opening insights here. Where do I even start? Um, I think maybe when it comes to like the prerequisites. Or the the enabling enablement uh part. And I think this is super interesting because this was exactly the the red line throughout my conversation with Brian. I think you mentioned two things that I think all teams should think about and understand. Whether these are boxes they already checked before they go off and and and start building their own agents. You mentioned one is uh the data infrastructure itself. Um which is uh almost almost a given almost, trivial when we speak about. But I still see many many teams who don't really don't really think about it at all. Uh and really to make sure that all of the different data, and when we think data sources we tend to think databases. But all of the like all of the different systems that hold data uh uh on your business. Whether it is sales force or Google cloud or slack or whatever, all of that is is connected and connected well. But the other thing that you said is even more interesting. Um and that is the fact that you have a centralized, you call it the AI team. It's a centralized engineering team that has developed the architecture. Uh and also kind of like the the the the building blocks. The skills. The the sub agents. Around really the the scaff scaffolding for creating engineering uh AI agents. Uh by non-engineers. Which is I think a very interesting notion. The first time that I'm hearing about it. Um and I'm sure that this has a lot to do with the insane velocity that you managed to reach in just three and a half basically, within 9 months or so.
Lalitha Rao
52:26
Definitely. I would say you uh it it might look uh you know like this was an easy switch. It certainly was not an easy you know switch here. And I think uh the reason we were able to do this, was like to your point, because we had this dedicated you know AI team. That uh you know was actually explicitly one of their remits was. How do you enable non- tech teams to do this. And what's the right sort of scaffolding and guard rails to really enable that. Uh you know we call this uh our harness platform internally, uh to be able to do this in a manner that's safe and uh encourages adoption as well. How do you you know make it easy for teams? So I think uh without that you know this would certainly uh not have been something that we would have been able to really you know run as a system. And it would have become so much more uh you know ad hoc, project based and not as impactful.
Chen Zamir
53:24
Yeah absolutely. That's how it sounds like. Um absolutely. Um we also talked a bit about, um well actually. Let me start with the build or buy. We talked a bit about build or buy just now. I think uh you made the point that you know where you want to focus. I mean you can build a lot of things. And when you where you want to focus your efforts is really around where you have proprietary data that you don't want to or can’t share with uh external parties. And these are probably the uh the areas around which you want to build. But you also made another point, uh when we talked about what are the use cases that you want to start with. Um and you mentioned like that specifically agent AI as means for automation. Uh what it solves, or where it brings unique capabilities is is around use cases that have multiple varied data sources in all sorts of formats. And the uh um an AI's ability to basically just kind of like digest all of that as one piece. Uh these are probably kind of like the the the use cases that that should pop uh first in priority. Um, you I think I think. Okay. And here here's I think the one of the things that I loved most about our conversation. Vision. Um, I think that a lot of the teams that enter the AI transformation journey, enter it with a very specific frame. And that is how can we be more efficient? How can we cut cost? How can we downsize the team? And your vision, your lens was completely 180 degrees different. And that was how can we make our product better? And I love it. Because it like it it is I mean first of all of course you would see uh better gains. Uh and of course you would be able to measure these gains in a very straightforward manner right. Um there are no like second order effect uh uh value here. It's like this is what you're setting up to do. I'm setting up to create more value to the company.
Lalitha Rao
55:56
And I think for us like what was top of mind was like how do we create more value for our members. Uh as well as our partners. And uh like I said you know it's just you know so much part of our mission is to be able to like uh deepen our ability to understand each and every member very holistically. Across all the interactions we've had with them. What was that last interaction with our customer service team? How do we bring that context into every interaction? So, we really deciding what is the you know next best action for us? Uh an interaction for our member uh that will help deepen engagement as well.
Chen Zamir
56:44
And on top of that, I think that the the almost unexpected uh thing that you found out is that while we're making the company more profitable, or growing faster, and while we're making the lives of our partners and members better, also our own team uh is is doing better. Their quality of life has been uh has been improved by that. And I think this is such, you know it's almost it's funny to say, it but it is such a positive story around AI integration. Which has no, I mean it's s it's like win win win. And I really really think that this is a great example of not only how to do things, but also how to set up the vision. And how to align the this like AI uh transformation vision between you as a leader, the company leadership, but also your team. And how everybody can win uh out of that. That I think is very very inspiring and um super important. Um, the other thing that I really liked uh that you said, and also came up uh in my 58:02 conversation with Brian is that you know it's very easy to look at all of the cool things that you've built. And this you know this agent that trains the model on its own. And builds the features on the fly. And so on. And it and it sounds very scary. Like how can I do that. But I think the fact that you mentioned that you started out with automating a silly report. And just doing like the smallest thing. Just to show that it can work. And just to kind of like get your bearings and understand like how to even approach this technology and how to approach such an endeavor. Uh this is also like super important and I really love that you shared it. Um, I also love that you shared um, you know, the hiccups or kind of like the guard rails that you've learned um, on the fly that you need to introduce. And you mentioned both human governance and the fact that you know while we are saying that we're automating um, and that is true that doesn't mean that the human is suddenly uh, absent from the process. Uh, the human is still there. It's like a major part of of of monitoring and measuring. Uh and the second thing that you mentioned in that uh in that context was uh, and I personally also use it and I found it to be a gamechanger working with AI, is to um measure certainty and expose the certainty and the I would that's my experience. The reasoning for that certainty, whether it's high or low, to the human. And that is a gamechanger in terms of how you do governance
Lalitha Rao
59:56
100%. I think everything has to be explainable. Like it that's like table stakes for us. Especially you know when we're looking at making risk decisions. Um so they're very aligned
Chen Zamir
60:09
So many great insights. Uh Lialitha uh first of all thank you very much for having this conversation and sharing from your experience. It it it is really really interesting to hear and also like super interesting to see how much you can learn in 9 months. And I'm just I'm I'm wondering if we'll talk in 9 months from today. Uh how much more things you will have to share. Uh and we should definitely do that. Um yeah, I I really enjoyed the conversation. Uh thank you very much for joining us today.
Lalitha Rao
60:42
Thank you so much Chen. And I really enjoy it. I'm, you know, a fan of the podcast. So, it's such a pleasure to be here today.
Chen Zamir
60:50
No, no, the pleasure is all mine. I assure you. Uh, and folks, I I I really hope that you enjoyed it. Uh, I really hope that you uh took some takeaways with you. And I'll see you next Saturday.
View full transcript
