The Dark Side of AI: What’s Actually Being Used Against Australian Small Businesses

Australian businesses lost $2.18 billion to scams last year. That number is not a prediction or a worst-case scenario, it’s what actually happened, and it’s 7.8% higher than the year before. Scamwatch’s 2025 Targeting Scams Report makes the trend clear: scammers are getting better, faster, and harder to spot. The reason is not hard to find. They are using the same AI tools your business is considering.

This is not about robots taking jobs. It’s about criminal operations that now run at industrial scale, produce phishing emails tuned to your suppliers’ writing style, and can put your CEO’s face and voice on a video call they never had. If that sounds far-fetched, keep reading.

The Numbers Don’t Lie: Australia Lost $2.18 Billion to Scams in 2025

The top five scam categories by loss in 2025, according to Scamwatch:

• Investment scams: $837.7 million
• Payment redirection: $166.8 million
• Romance scams: $139.9 million
• Phishing: $97.6 million
• Remote access: $69.9 million

Payment redirection, criminals intercepting or faking supplier invoices, is the one that hits businesses hardest. It costs Australian businesses $166.8 million a year and it’s the category AI has made dramatically worse.

The ASD Cyber Threat Report 2023-24 puts the average cost of a cybercrime incident for a small business at $49,600, up 8% from the year before. For a medium-sized business, it was $62,800. A single successful attack can break a small operation. And these attacks are getting harder to detect.

The New Toolkit: What Criminal AI Actually Does

The criminals targeting your business are not working alone at a keyboard writing bad emails. They are running automated operations with AI tools purpose-built to deceive. Here’s what the toolkit looks like right now.

Jailbroken AI (WormGPT, WolfGPT): Phishing at Industrial Scale

WormGPT was uncovered by reformed hacker Daniel Kelley and sold on dark web forums. Unlike ChatGPT or any consumer AI, it has no ethical guardrails. It will write targeted phishing emails with the psychological precision of a con artist, at the speed of software. The Age reported in May 2026 that models like WormGPT can craft phishing targeted to individuals with “cunning psychological accuracy, at scale.”

One model, obtained by a researcher and seen by The Age, was being sold on Telegram and offered to run “live” frauds with victims simultaneously across multiple messaging platforms. This is not a lone scammer. It’s a software product sold to criminal operators.

The practical upshot: a phishing email that mimics your accountant’s writing style, references your real supplier names, and arrives on a Monday morning when you are expecting an invoice is not hard to produce anymore. It takes seconds.

Deepfake Video Calls: When Seeing Is No Longer Believing

In early 2024, a finance worker at a Hong Kong company was tricked into transferring $US25 million after a deepfake video call. The criminals created AI-generated replicas of the company’s CFO and other colleagues. The worker sat through what appeared to be a legitimate multi-person video conference. Then he sent the money.

That was 2024. By May 2026, The Age was reporting that “live” hyperrealistic deepfake video frauds are operational, with criminal gangs impersonating executives and suppliers in real-time calls. The step from static image to interactive video deepfake is complete. These are not obviously fake. They are convincing enough to fool people who are paying attention.

Australia’s own extremist groups have already demonstrated how quickly deepfakes can be manufactured and deployed. The Age reported that in the hours after the Bondi attack, one neo-Nazi group was already posting realistic deepfake videos. Separately, a fake video placed a neo-Nazi speaker at a One Nation fundraiser he had actually been excluded from. If that kind of content can be manufactured that fast, a fake video call from your “CEO” is well within reach.

Voice Cloning: Your “Boss” on the Phone

Voice cloning requires only a short sample of someone’s voice, a YouTube video, a podcast, a voicemail. The output is good enough that most people cannot distinguish it from a live call. The Age reports that criminal gangs use voice cloning as part of their CEO impersonation operations, calling accounts staff with urgent payment requests.

Cybersecurity expert Matt Kriner put it plainly: “The guard rails are not there like people believe… we’re all being encouraged to use it as if it’s perfectly safe and secure.”

A call from your boss’s number, in your boss’s voice, asking you to transfer money urgently, that is not a hypothetical. It’s a documented attack pattern.

The Australian Angle

The $166 Million Invoice Fraud Problem

Payment redirection scams work by substituting real bank details with criminal ones. AI has made this far easier. A criminal can now scrape your supplier’s public information, study their email style from phishing data or purchased lists, and generate a near-perfect fake invoice email. Your accounts team has no reliable visual cue that something is wrong.

The volume of these attacks explains why payment redirection sits at $166.8 million a year and climbing.

The Pig-Butchering Playbook

Investment scams at $837.7 million dwarf every other category. Most of the sophisticated ones now use what’s called a pig-butchering structure: criminals build a relationship over weeks or months before introducing an investment opportunity, then drain the victim’s funds when trust is established.

A live example operating in Australia is auadstest.com, a Chinese-origin fake Facebook advertising revenue platform. It mimics the legitimate Facebook Ads interface closely enough to fool business owners. Victims are recruited through social media, an AI persona maintains the relationship over time, and the money is gone once it is transferred. If you or any of your staff receive an unsolicited approach about a “Facebook ad revenue sharing program” or similar, it is a scam.

The Telegram infrastructure behind these operations is well documented. The Age reported that one Telegram-based model “offered to run ‘as live’ frauds with victims across multiple messaging platforms”, simultaneously, at scale.

Australian Extremists Are Already Using This Tech

This matters to businesses not because of extremism directly, but because it shows the speed of adoption. The Age reported that both far-right and Islamist extremist groups in Australia “routinely use Telegram bots for ‘vetting’ and first contact”, the same conversation-funnel approach used in financial fraud. The technology transfer from extremist use to criminal use is not a long journey.

UNSW cybersecurity professor Alana Maurushat is tracking an Eastern European criminal network currently building AI tools optimised for cybercrime profit, expected to deploy within two years. Her view on where this is heading: “Every workplace is rushing to integrate AI. Businesses see cost-savings in automation. [But] that’s a feeding frenzy for criminals.”

The Chatbot Honeypot Your Staff Don’t Know About

Here’s the threat nobody is talking about at the small business level.

National security researchers have documented adversarial AI chatbots, companion apps and “AI assistant” products, specifically designed to extract information from users who trust them. Scientific American has reported on this in the context of public servants, but the same dynamic applies to anyone who works with sensitive client data, financial details, or business strategy.

Your bookkeeper who asks their AI companion app “what should I do about this unusual invoice from our supplier?” is potentially exposing your supplier relationships. Your operations manager who pastes a contract into a free AI tool to summarise it may be handing your terms to a data scraper. Most consumer AI tools retain conversation transcripts. Many train on them. Some are built to harvest them.

This is not paranoia. It’s a documented attack vector. And most SMBs have no policy around it.

What Actually Protects You (AI-Specific, Not Generic)

Standard cybersecurity advice, use strong passwords, enable MFA, is still relevant but it doesn’t address any of the threats above. Here’s what actually does.

The Codeword Rule for Phone Calls

Pick a secret word, share it only with the staff who handle financial approvals, and require it to be spoken in any phone call that authorises an unusual payment. No codeword, no payment. This is low-tech, free, and works against voice cloning completely. The clone doesn’t know the word.

Payment Change = Video Call, Every Time

Any request to update a supplier’s bank details must trigger a video call, not a phone call, not an email reply. It has to be over your standard platform (Teams, Zoom) from a contact in your internal directory, not a new meeting link. Do this without exception and you eliminate the entire category of AI-generated invoice fraud.

What Your Staff Can’t Tell Their AI

Write a one-page policy that your team can actually understand. The core rule: no client data, no financial details, no internal strategy in prompts to public AI tools. That means no pasting contracts into ChatGPT, no asking Claude to summarise an internal report, no using Copilot on a document that contains client names. It also means no discussing work in AI companion apps.

This does not need to be sophisticated. It needs to exist and be communicated.

Australian Government Tools Worth Bookmarking

• Scamwatch, report a scam (ACCC)
• ReportCyber (ASD, for cyber incidents)
• ASIC MoneySmart, check a financial business (before engaging any investment platform)
• National Anti-Scam Centre (operational arm of the ACCC)

The Bottom Line

Alana Maurushat’s comparison is worth sitting with: “Asbestos was the start of huge innovation too when everyone started bringing it inside the house.”

AI is genuinely useful. The same tools criminals use to generate phishing also help your business write better copy, answer customer questions faster, and analyse data you wouldn’t otherwise have time to look at. This is not an argument against using AI.

But the people who want to steal from your business are already using it. They are using it to write better scam emails, clone voices, fake video calls, and run months-long relationship frauds at scale. The advice to “just use AI carefully” misses the point. The question is whether you have specific defences against specific AI-powered attacks, because right now, most Australian small businesses don’t.

The codeword costs nothing. The payment-change protocol costs nothing. The staff policy costs an hour to write. None of it requires a cybersecurity team. All of it works.

Sources

• ACCC Scamwatch, Targeting Scams Report 2025
• ASD Annual Cyber Threat Report 2023–24
• The Age, “ChatGPT’s Evil Twin: How Criminals and Extremists Are Using AI to Lay Traps” (8 May 2026)
• Scientific American, “Chatbot Honeypot: How AI Companions Could Weaken National Security”
• Wired, CJNG Cartel and AI (citing Interpol 2024 warning)
• Anthropic, Statement on US government directive to suspend Fable 5 and Mythos 5
• National Anti-Scam Centre (NASC)

More on this: AI News & Trends →

📖 More deep dives: News Deep Dives

📖 More deep dives: News Deep Dives

📊 Compare AI tools side by side | 💼 Free resources & AI prompt packs

More hot takes: Hot Takes →