---
title: When a Data Exec Walks Into a Fraud Team, with Shachar Meir
source_page: https://www.sardine.ai/media/the-saturday-fraud-strategist/episodes/llm-analytics
canonical: https://www.sardine.ai/media/the-saturday-fraud-strategist/episodes/llm-analytics
format: text/markdown
date: 2026-08-03T04:01:00.000Z
description: LLM analytics can help fraud teams move faster, but only with strong data governance, clean foundations, and better questions before AI adoption…
---

**Quick links:** [Human page](https://www.sardine.ai/media/the-saturday-fraud-strategist/episodes/llm-analytics) · [Home](https://www.sardine.ai) · [Customers](https://www.sardine.ai/customers) · [Blog](https://www.sardine.ai/blog) · [Demo](https://www.sardine.ai/demo)

---

# When a Data Exec Walks Into a Fraud Team, with Shachar Meir

**Published:** 2026-08-03T04:01:00.000Z

LLM analytics can help fraud teams move faster, but only with strong data governance, clean foundations, and better questions before AI adoption…

So this episode starts with a data executive walking into a fraud team.
Which sounds like the setup to a very niche joke. And maybe it is. But honestly, it also describes a real problem most fraud teams know too well: we rely on data for almost everything, but the relationship between fraud teams and data teams is often messier than anyone wants to admit.
In this episode, I’m joined by Shachar Meir, a data advisor and former director of data at Meta, to talk about LLM analytics, self-service analytics, data infrastructure, and why fraud teams cannot just throw AI on top of messy data and hope it becomes strategy.
Because yes, LLM-powered self-service analytics sounds amazing. Ask a question in plain English, get an answer, move faster, avoid waiting three weeks for a data team with 900 priorities. Great. I want that world too.
But then you’ve got to ask yourself: where did the answer come from? Which tables did it join? What definition did it use? Did it understand the business context? Did it hallucinate? Did you ask the right question in the first place?
Not a small detail.
This conversation is really about the gap between the promise of AI analytics tools and the operational reality of fraud analytics. Fraud and risk teams work in adversarial environments. The problem keeps changing. The signal-to-noise ratio is bad. The cost of getting it wrong can be massive, whether that means letting fraud through or blocking legitimate users. So if you want LLM analytics to actually help, you need more than a shiny interface. You need data foundations, governance, semantic clarity, and the humility to start small.
What you’ll hear in this episode:
Why fraud and risk teams are unusually complex data customers
How fraud teams can work better with data teams instead of requesting endless point solutions
Why self-service analytics failed so often before LLMs entered the picture
What changes, and what does not change, when LLM analytics becomes the interface
Why asking the right data question matters more than getting a fast answer
How data governance, semantic layers, and data quality shape AI analytics results
Why fraud teams should start AI adoption with one curated table, one use case, and one measurable outcome
How to think about the one-hour, one-day, one-week, and one-month versions of a data project
You should listen to this episode if you:
Work in fraud operations and depend on data teams to ship risk or fraud analytics projects
Are considering LLM analytics or AI agents for data analysis inside your fraud stack
Have been promised AI data querying that sounds too easy, because it probably is
Need better ways to align with data engineering, analytics, or risk data infrastructure teams
Want a practical way to think about AI data governance before deploying tools that affect real users
### Episode notes & key takeaways
Fraud teams are not normal data customers
One of the main points Shachar makes in this episode is that fraud teams are different from most business stakeholders. And I say that with love. Mostly.
Marketing, sales, and finance teams absolutely need good data. But in fraud and risk, the complexity is different. You might run huge batch processes over years of historical data, but the result still needs to support a decision in milliseconds. Is this account legitimate? Is this transaction risky? Is this behavior normal, or is it the start of an attack?
That is a very strange operating environment.
The signal-to-noise ratio is usually bad. Fraud patterns shift. Legitimate users behave in ways that look weird. Fraudsters behave in ways that look normal. And sometimes the difference between a stolen account and a real user traveling across borders is not obvious at all.
This is why the collaboration between fraud teams and data teams has to be tighter than a standard request-response relationship. Fraud teams cannot just throw jobs over the fence and hope the data team runs them in production. Data teams need context. They need to understand the mission. They need to know why a decision matters and what happens when the system gets it wrong.
Otherwise, everyone is technically doing their job, and somehow the system still fails. Not ideal.
Point solutions create long-term data problems
Shachar talks about one of the most common mistakes he has seen in fraud and risk teams: asking for pointed solutions instead of infrastructure.
A fraud team sees a problem. Returning malicious sellers. Returning abusive buyers. A specific scam pattern. A new risk flow. So they ask the data team for a solution to that one problem. That feels reasonable in the moment. But if every request becomes a one-off project, the entire system becomes harder to maintain, harder to scale, and harder to reuse.
The better question is: what infrastructure would make many of these solutions easier?
A lot of fraud analytics problems need similar foundations. You need to calculate backend signals from historical behavior, store them somewhere accessible, and make them available in real time or near real time. If you build that pipeline properly once, you are not starting from zero every time a new fraud pattern appears.
That does not mean every problem has the same solution. Obviously not. Fraud would be much easier if that were true. But it does mean that fraud teams should align with data teams around reusable infrastructure, not just individual requests.
That is how you make the next project cheaper, faster, and less painful.
The last mile of fraud models matters more than teams admit
There is a very familiar moment in the conversation where Shachar describes teams building sophisticated models and then using a static threshold almost randomly.
Okay, the model score is greater than 800, decline the transaction.
Why 800?
Often, the answer is not great.
This is one of those things that can make a fraud person quietly stare into the distance. A team spends months building a strong model, measuring performance, tuning features, and evaluating statistical outputs. Then the actual business decision layer gets much less attention.
But the last mile is where the model touches users. It is where you decide how much fraud you stop, how many false positives you create, and what tradeoffs the business is actually making.
AUC, or area under the curve, can be useful. But it does not tell you whether the model is being used well. It does not tell you whether the threshold makes sense for different regions, segments, products, or user types. It does not tell you whether the business outcome is improving.
That is why the strategy layer matters. If you do not periodically validate how the model is being used in production, a lot of the work that went into the model can go to waste.
Self-service analytics failed before AI for a reason
Self-service analytics is not new. It existed long before LLM analytics. Dashboards with filters. BI tools. Purpose-built internal tools. Dropdowns everywhere. The dream was simple: let business users answer their own data questions without needing SQL, Python, or an analyst.
And yet, it often failed.
Why? Because self-service analytics depends on assumptions that are rarely true.
You assume users know what they want. You assume they can explain it clearly. You assume words have deterministic meaning inside the company. You assume “revenue” or “decline” or “active user” means the same thing to every team. Cute idea. Often false.
You also assume users understand the complexity of the data. They usually do not. And that is not an insult. Data has tribal knowledge buried everywhere. Sometimes an analyst knows to exclude traffic from a specific country for a certain report because otherwise the results are noisy. The business user may never know that decision happened, but the report only makes sense because someone made it.
Self-service also assumes people want to be accountable for pulling their own numbers. Sometimes they do not. If a number is wrong in front of the CEO, some people would very much prefer the analyst to be the person under the bus. Human behavior. Not elegant, but real.
So when we say LLM analytics will solve self-service analytics, we should pause. The interface changed. The underlying problems did not disappear.
LLM analytics can make bad questions more dangerous
The scary part of LLM analytics is not that it can answer questions in plain English. That part is impressive.
The scary part is that it may answer the wrong question very confidently.
With traditional dashboards, the definitions are usually hard-coded. You may hate the dashboard, but at least it is pulling from a known source and applying known logic. With AI data querying, the system may decide which tables to use, how to join them, how to interpret the request, and how to present the answer.
If the user does not understand SQL, data structure, metric definitions, or business context, how do they validate the answer?
Exactly.
This is where AI hallucinations in analytics become a real operational risk. The answer may look polished. The query may look plausible. The result may even be close enough to feel right. But if the question was poorly framed or the data context was missing, the output can still be wrong.
And in fraud operations, wrong answers are not just academic. They can lead to bad rules, bad reviews, bad model decisions, bad policy changes, and real impact on users or losses.
This is why Shachar’s point is so important: the job of an analyst is not just to answer questions. It is to help ask the right questions.
Honestly, that is the part we should be automating last, not first.
Data governance is not optional for AI analytics tools
One of the most useful parts of this conversation is the distinction between two approaches to AI analytics tools.
One approach is: give the AI agent access to the warehouse and let people ask anything.
That is the version that makes everyone anxious, or at least it should.
The other approach is: build strong data foundations first. Define the metrics. Create a semantic layer. Add metadata. Clarify what each field means. Curate the tables. Monitor quality. Then use LLM analytics on top of a controlled data environment.
That second approach is less exciting in a demo. It is also much more likely to work.
AI analytics tools need the basics even more than humans do. A human analyst can sometimes notice when data looks strange, remember tribal context, or ask a follow-up question. An LLM may just produce the most probable answer. And if the underlying context is messy, that answer can be messy too.
So if a fraud team wants to adopt LLM-powered self-service analytics, the first project should not be “connect the entire warehouse.” It should be one trusted use case, one highly curated dataset or set of tables, clear definitions, good metadata, and a way to validate whether the tool is actually helping.
Very boring. Very necessary.
Start with one use case, not the entire data universe
Shachar’s advice for fraud team AI adoption is refreshingly practical: do not boil the ocean.
Start with one use case. One repeatable question. One trusted dataset. One table or one small set of curated tables with definitions the team actually understands.
Build the validation layer. Build the context layer. Add metadata. Define what “declined transaction” means. Is it declined by your fraud system? Declined by the card issuer? Declined because of insufficient funds? These distinctions matter.
Then test whether the LLM analytics tool helps. Does it answer the right question? Does it use the right data? Does it reduce time to insight? Does it support a safer decision? Does it make the fraud team better, or just faster at being wrong?
That last question matters.
If the first use case works, expand gradually. Add dimensions. Add metrics. Add more questions. But do not start with the fantasy that AI can duct tape together every messy database, environment, table, and definition in the company.
That is not a strategy. That is a very expensive shrug.
AI can help with context, brainstorming, and sharper thinking
This episode is not anti-AI. That would be lazy. Also, wrong.
Shachar gives a useful distinction: AI can be very good at providing external context, brainstorming, ideation, and challenging your thinking. If you need a sense of the typical range for a business metric, AI may help you get directional context. If you want to think through what you might be missing, AI can be useful. If you want to pressure test a plan, it can help.
The key is to use it as augmentation, not delegation.
That means AI can help you sharpen your thinking, but it should not replace your thinking. It can suggest questions, but you still need to know which ones matter. It can help frame options, but you still need to understand the decision. It can help draft a query, but you still need to validate the logic if the result affects real users.
This is especially true in fraud and risk, where decisions can trigger declines, reviews, friction, user harm, financial loss, or compliance exposure.
So yes, use AI. But maybe do not hand it the keys to production and walk away. Seems fair.
The one-hour version is often better than the one-month version
Toward the end of the conversation, Shachar shares a simple but useful framework: every project has a one-hour version, a one-day version, a one-week version, and a one-month version.
This is worth stealing. Respectfully.
A lot of technical teams, including fraud and risk teams, see a problem and immediately imagine the one-month version. Every edge case. Every integration. Every perfect data source. Every complexity. And then the project becomes so large that it stalls, gets deprioritized, or dies quietly in a roadmap document.
Instead, ask what the one-hour version looks like. Can you create directional value quickly? Can you estimate the metric? Can you make progress without pretending it is final?
Then improve it. The one-day version is cleaner. The one-week version is automated. The one-month version is more complete.
This is not new. It is MVP thinking. It is agile. But apparently we all need to relearn simple things every few years under a new name. Fine.
For fraud teams working with data teams, this mindset is especially useful because data resources are limited. If you can show value quickly, validate the direction, and then build from there, you are much more likely to get support than if you propose a two-year project nobody fully understands.
### Final takeaway
LLM analytics can absolutely help fraud teams. But only if the foundations are not a mess.
If your data definitions are unclear, your semantic layer is missing, your metrics mean different things to different teams, and your users do not know how to ask the right questions, then AI will not magically fix that. It may just make the confusion faster.
That is the slightly annoying but useful takeaway from this episode: fraud teams should not wait for perfect data, but they also should not pretend AI removes the need for data governance, data quality, and smart collaboration with data teams.
Start small. Build one trusted use case. Align on the infrastructure. Ask better questions. Use AI for context and thinking support, not as an accountability escape hatch.
Anyway, if your LLM analytics strategy is “connect everything and see what happens,” I guess technically that is a strategy.
Just not one I would want to explain after it breaks.
Resources & links
Not ready to stop the conversation about my, and hopefully your, favorite subject? Subscribe to The Saturday Fraud Strategist newsletter.
Connect with Shachar Meir | LinkedIn
Data Advisor
Former Director of Data @Meta
Connect with Chen Zamir | LinkedIn
Host of The Saturday Fraud Strategist
Helping fintechs build smarter fraud defenses
Co-author of “The Fraud Fighter’s AI Playbook”