Manual review is a human looking at cases the automated system cannot resolve confidently, inspecting the evidence and making the final call. It handles ambiguity and produces high-quality labels, but it is slow, costly, and hard to scale.
What is manual review, in plain English?
Manual review is where a human analyst steps in on the cases automation cannot decide with confidence: the gray zone between clearly good and clearly fraudulent. The analyst pulls up the evidence, the device, the identity data, the transaction history, any documents, and makes the final approve, decline, or escalate decision.
Review earns its keep on ambiguity and edge cases, the situations too nuanced or too novel for a rule or model to call cleanly. It also does something valuable for the whole system: every reviewed case becomes a high-quality label that can feed model training, sharpening the automation over time.
In the detection stack manual review is the safety valve behind scoring and rules. It is powerful but expensive: it is slow, it costs analyst time, and it does not scale the way software does, so how much you route to it is a decision in itself.
How a case flows through review
A reviewed case moves from an uncertain score to a human decision and back into the system as a label.
- Route — Land in the queue. A case the system cannot resolve confidently is routed to a review queue instead of auto-decided.
- Inspect — Gather the evidence. The analyst reviews device, identity, history, and documents to understand the case.
- Decide — Make the final call. Approve, decline, or escalate, applying judgment the automation could not.
- Feed back — Create a label. The decision becomes a high-quality label that trains and tunes the models behind it.
What it looks like in practice
In practice
A review queue that used to clear comfortably starts backing up. The manager's first instinct is to ask for two more analysts, and the SLA pressure makes it feel urgent.
Before hiring, an analyst pulls the reasons cases are landing in review. Most trace back to a single rule that started firing far more often after a recent threshold change, dumping a flood of borderline-good customers into the queue. The queue was not growing because there was more real ambiguity; it was growing because the thresholds and features upstream had gotten sloppier. Fixing the rule drains the queue overnight. Adding headcount would have paid to keep clearing a problem that never needed to exist.
Why a rising queue is a symptom
Manual review is managed with a handful of numbers: review rate, or what share of traffic gets routed to humans; queue SLAs, or how fast cases get cleared; inter-analyst agreement, or how often reviewers reach the same call on similar cases; and decision accuracy against later outcomes. Watched together, these tell you whether review is healthy or straining.
The key insight is that over-routing to review is usually a signal that your thresholds or features need tuning, not that you need more people. A rising queue tempts managers to add headcount, but that treats the symptom. If too many cases land in the gray zone, the fix is usually upstream: sharper features and better-set thresholds that let automation confidently decide cases it is currently punting to humans. Throwing people at the queue keeps clearing cases that should never have been ambiguous.
What to watch in the data
- Review rate creeping up. A rising share of traffic going to humans usually points to upstream threshold or feature problems, not a need for headcount.
- SLA breaches. Cases sitting too long mean customers waiting and fraud aging; watch queue time, not just volume.
- Low analyst agreement. When reviewers disagree on similar cases, your guidelines or evidence are unclear, and labels get noisy.
- Decision accuracy. Track how review decisions hold up against later outcomes; consistently wrong calls point to training or evidence gaps.
- Queue composition. If most reviewed cases turn out good, you are routing too aggressively and burning analyst time on safe traffic.
Quick questions
When should a case go to manual review?
When automation cannot decide it confidently: genuine gray-zone cases where the score is ambiguous or the pattern is novel. Cases the system could resolve on its own should not be there.
Is a growing queue a staffing problem?
Usually not. A rising queue more often means thresholds or features upstream need tuning, dumping cases that should have been auto-decided. Adding headcount treats the symptom, not the cause.
Why is manual review valuable if it is slow?
Because it handles ambiguity software cannot and produces high-quality labels that improve the models. It is a precision tool for hard cases and a source of training data, not a bulk decisioning engine.
How do you measure review quality?
With inter-analyst agreement and decision accuracy against later outcomes, alongside review rate and SLAs. Agreement shows consistency; accuracy shows whether the calls were right.
Can review decisions train the model?
Yes, and that is a major benefit. A careful human decision on a hard case is a high-quality label, and feeding those back helps the automation learn to handle similar cases on its own.
Go deeper
- FFIEC BSA/AML Examination Manual ↗ — The manual US examiners use to assess BSA and AML programs.
- FTC Consumer Advice: Scams ↗ — US consumer guidance on current scams and fraud, and how to report them.