Key takeaways every fraud team needs to know:
- A fake government ID costs about $50 in bitcoin and takes seconds to generate. The site that sold more than ten thousand of them had customer support, a Telegram channel, and a signature motif. It was not hard to find.
- Real-time video deepfakes now run on consumer gaming hardware. The premium tier comes with a technical onboarding call, a support team, and a vetting process for new buyers.
- Voice biometrics can be bypassed with a five-minute clone built from publicly available audio. A podcast, a press interview, or a public talk is enough training data.
- These AI scam vectors are not new. What changed is that someone packaged them together into one coordinated identity attack that any low-skill operator can now run.
- The tools that enable these AI scams are sold in the same Telegram channels used by scam compounds where trafficking victims are forced to run operations at industrial scale. It is one supply chain, not two separate worlds.
- The assumption that only elderly or naive people fall for AI scams is the assumption that will get more people scammed. The tools are cheap enough that the pool of people who can run a convincing attack just got significantly larger.
Watch Joseph Cox's full keynoteJoseph Cox opened his SardineCon keynote in San Francisco by describing something he admitted sounds ludicrous.
Let’s say you get a call from your spouse, or your son, or your daughter. The cadence is right, the rhythm is right, but there's something ever so slightly off that you can't put your finger on. So naturally, you ask for a video call on FaceTime (or Zoom if you're a very corporate family). They pick up and it's them, except the movements are also a little wrong. Which then makes you ask what Joseph called a genuinely insane question: can you send me a photo of your ID?
They're offended, because what a ridiculous thing to ask a family member. The photo arrives anyway, taken on a sofa and you relax. That's when the ask for gift cards or the wire comes.
Joseph has done every single step of that himself. He co-founded 404 Media, and his method is to buy what the criminals are buying and test whether it works. He calls each piece trivial. The only part he hasn't seen anyone do yet is link them together.
A fake ID costs about $50 in bitcoin and takes seconds
Joseph walked through a site called OnlyFake, run by someone using the handle "John Wick" on Telegram (yes, they were obsessed with the movies, signature and all). He loaded up about $50 in bitcoin, picked a country and a document type, typed in a name and date of birth, uploaded a photo, and had the fake document in seconds. There's even an API, so a buyer could feed it a CSV and produce them in batch.
The first version came out flat, which was a problem, because a bank running Know Your Customer checks wants a photo of the physical document. Naturally, OnlyFake added background so buyers could pick from a library, upload their own, or generate one at random. The library included a piece of outdoor furniture and, according to Joseph, “one of the most disgusting carpets ever”. Either way the ID ended up on a carpet or a table or someone's lap, looking like a photo a person actually took.
Joseph made himself an old-style British passport and tested it against a cryptocurrency exchange's identity verification. It passed. He only tested the document, never the face-matching step.
The FBI arrested the operator and charged him with selling more than ten thousand fake ID documents. Neither the arrest nor the charges were announced until after he had pleaded guilty, leaving an unexplained window that Joseph is still digging into.
But of course, shutting the site down didn't remove the capability. Researcher Jane Wong used a commercially available tool to produce both a document and a selfie of herself holding it, and other people are generating passports with Google's Nano Banana Pro. Each time a vendor patches a jailbreak, buyers move to whichever mainstream model hasn't been patched yet. The same guardrail cycle that runs through every other AI safety problem is now pointed at government identity documents, and patching just relocates the issue.
Real-time video deepfakes now run on a gaming laptop
Joseph showed footage of Nigerian scammers, working the romance-scam angle known as Yahoo boys, transforming themselves live on video calls. A Black man becomes an older white man with facial hair, mid-conversation. Another switches from man to woman on a mobile phone.
The output is rough. There's a ring light involved, sometimes a wig (the software still can't do hair well), and the quality is closer to a bad Zoom filter than a movie effect. But it still works, and people are getting scammed by it on a daily basis.
The premium tier comes with a support team
Then there's the tier above that. Joseph played a clip of someone stress-testing their own face swap. Light moved across the face, a cheek pinched so the skin stretches, eyes closed one at a time. Nothing broke the illusion.
They were using a Chinese-language tool, sold with actual customer support, 24x7 response, and a technical onboarding call. It needs a genuinely powerful gaming rig (a 4080 Super at minimum, 16 gigabytes of DDR5) and runs about $2,000 a year. So, naturally, Joseph bought it.
The process took over a month because the sellers vet buyers first, partly by asking for PC specs to filter out time-wasters. As someone whose most powerful device is his PS5, Joseph had to message his co-founder Emanuel Maiberg, a massive PC gamer, and ask for screenshots of his rig to forward to the vendor as proof he was a serious buyer.
When he finally got a demo call, the person on the other end was live-deepfaking into a Hong Kong actor's face, then switched to a woman. A custom model of a specific person costs a few hundred more, so Joseph paid for one and sent over a handful of the worst selfies he has ever taken. Then he watched someone wear his own face on a live call. He showed it to his mother and even she was convinced by it.
That's the bar now. A service business, with a support team, selling identity theft as a subscription.
This software is marketed in the same Telegram channels used by the scam compounds in Myanmar and Cambodia, where trafficking victims are forced to run these operations at industrial scale. One supply chain serves both ends of the market.
Voice biometrics didn't stop an AI clone
Joseph's bank in the U.K. lets customers authenticate by saying "my voice is my password." He used ElevenLabs to clone his own voice, reading out the script it generated for him, which (for reasons nobody has been able to explain) was a steamy romance novel he had to perform at his desk in the office. Then he played the cloned phrase into a phone at the bank's voice prompt and it worked. He got into his own account, checked the balance, the whole thing.
The clone wasn't even that good. There's a slightly wrong cadence in how it says "password," like the emphasis lands in the wrong place. It didn't matter because the system accepted it anyway. And that's with a five-minute instant clone. ElevenLabs also offers a 30 minute "professional" version for cleaner output, which is nothing if the target has ever given a public talk, hosted a podcast, or done a press interview. Any AI native fraud team relying on voice authentication should treat this as a documented failure mode, not a theoretical one.
What ties it together
Fake IDs, deepfakes, and voice cloning have all existed for years. What's changed is that someone packaged them for buyers who don't want to learn any of the underlying technology. Joseph found a market that sells the document, the face swap, and the voice as separate purchases a buyer can stitch into a single call. The Chinese tool he bought ships with a real-time translation layer, so an operator who speaks no English can still run the call in English.
And while there is a market for the premium tools that run a couple thousand dollars a year, the free ones (disguised as prank apps on the Play Store) are still good enough to fool a lot of people already.
Joseph kept the warning broad. He doesn't think we're ready for AI-powered fraud, and he didn't just mean as an industry or a company. He meant as a society. The idea that only old or naive people fall for scams is, in his words, bigoted and wrong. The tools are now cheap enough that the pool of people who can run a convincing scam is a lot bigger, and the pool of people who can be convinced didn't shrink at all.
Frequently asked questions
What should an AI native fraud team understand about fake ID generation tools?
Fake ID generation services like OnlyFake have made high-quality document fraud accessible to anyone with $50 in bitcoin and a Telegram account. These tools generate convincing government documents in seconds, place them on realistic backgrounds, and have been tested successfully against live cryptocurrency exchange verification flows. For fraud teams running document-based identity checks, this is a documented bypass, not a hypothetical one.
How do real-time video deepfakes work and why do they matter for fraud prevention?
Real-time video deepfake tools replace a person's face live on video calls using consumer gaming hardware. The premium tier operates as a subscription service with customer support and buyer vetting. While the output quality varies, it is sufficient to deceive people in live conversations. For fraud teams using video-based liveness checks or remote interview processes, these tools represent a documented attack vector that is actively being used at scale.
Can voice cloning bypass bank voice authentication systems?
Yes. A five-minute voice clone built from publicly available audio, such as a podcast episode or press interview, successfully bypassed a major UK bank's voice authentication system in a documented test. The clone was imperfect but was accepted by the system anyway. Any fraud team relying on voice biometrics as a primary authentication control should treat this as a known failure mode requiring a layered defense.
What makes AI scams harder to stop now than they were two years ago?
The individual components of AI scams, fake documents, video deepfakes, and voice cloning, have existed for years. What changed is that they are now packaged together into coordinated identity attacks that require no technical skill to execute. Low-cost tools are available through mainstream app stores and Telegram channels. Premium tools come with customer support and onboarding. The barrier to running a convincing AI scam has dropped far enough that the pool of people capable of executing one has grown significantly.
