What Can Go Wrong With AI Agents: Risks, Guardrails & When to Stay Human

We’re pro-AI. We’re also honest about what can go wrong.

There’s a version of AI coverage that treats every tool like it’s magic and every risk like it’s fear-mongering. That’s not useful to anyone running an actual business.

AI agents work, and some of them will save you real time and money. But they can also get things wrong in ways that cost you clients, expose your customer data, or quietly blow out your tech budget. This post covers the five most common failure modes, what they look like in practice, and how to guard against each one.

Risk 1: Hallucinations and wrong outputs

What it is: AI models don’t look things up the way a search engine does. They generate plausible-sounding responses based on patterns in their training data. When they’re wrong, they’re often confidently wrong, with no visible sign that anything has gone sideways.

A real example: In one widely reported case, consulting firm Deloitte produced an AI-assisted report for an Australian government client that contained fake academic citations, references to research papers that didn’t exist. Deloitte was required to refund part of the contract. Source: TechRadar, 2025.

For small business, the stakes look different but the failure mode is the same. An AI agent drafting a quote with wrong pricing. A customer chatbot giving outdated hours or prices. A legal document draft that gets a clause wrong because the model guessed.

Guardrail: Never let an agent publish or send anything without a human check first, at least until you’ve run it long enough to trust its accuracy on that specific task. For factual or legal content, that human check isn’t optional. Keep “human in the loop” as the default and only remove it when you have evidence the outputs are reliable.

Risk 2: Data privacy and the Australian Privacy Act

What it is: When you give an AI agent access to your CRM, booking system, or email, and it processes customer names, contact details, health information, or financial data, that data often leaves your systems and gets processed on overseas servers.

Why this matters now: Australia’s Privacy Act is expanding. The $3 million annual turnover exemption that sheltered most small businesses from Privacy Act obligations is being phased out. Once fully in effect, the changes will bring around 95% of Australian businesses (over 2 million SMEs) under the Act’s requirements. A new statutory tort for serious privacy invasions start in June 2025, giving individuals the right to sue for damages directly. Source: Schiller Legal, 2025.

If your AI tool is processing customer data and storing it on US or EU servers, you need to check whether you have a lawful basis for that transfer under the Australian Privacy Principles.

Guardrail: Before connecting an AI tool to any system containing customer data, read the vendor’s privacy policy. Find out where data is processed and stored. Ask if they have an Australian Data Processing Agreement. If they don’t, or if the answer is vague, treat that as a red flag. For health or financial data specifically, get legal advice before proceeding.

Risk 3: Over-automation, when the agent runs without the right guardrails

What it is: Agents do what they’re told, not what you meant. If the instructions aren’t precise, or the trigger conditions aren’t right, an agent will confidently execute the wrong thing at scale.

A scenario: A cafe owner sets up an AI agent to respond to all Google reviews with a personalised reply. The agent’s instructions include: “if the review mentions food poisoning or illness, apologise and offer a refund.” A bad-faith review arrives. The agent apologises and publicly offers a refund. The cafe didn’t have a food issue, but the public response looks like an admission.

Or simpler: an agent is set up to follow up on unpaid invoices automatically. The payment had already come through, but the bank feed synced slowly. The client gets a payment demand 20 minutes after they paid.

Guardrail: Define what the agent cannot do without human approval. Sensitive actions, public responses to negative reviews, cancellations, refunds, legal or financial communications, should require a human sign-off regardless of how confident the agent is. Write these limits into the agent’s instructions explicitly, not just as a mental note.

Risk 4: Cost blowouts

What it is: Most AI tools charge per usage, per API call, per message processed, per token. When a process runs more than expected, or when something breaks and an agent loops, the costs can multiply fast. Misconfigured agents have generated thousands of dollars in API charges in a matter of hours.

Guardrail: Set hard spending limits in your AI provider’s billing settings before you do anything else. OpenAI, Anthropic, and most platforms let you set monthly caps or alerts. If your agent is using a third-party platform, check whether it has usage limits built in and turn them on. Review your AI spend weekly for the first month of any new agent to catch anything unexpected early.

Risk 5: Vendor lock-in

What it is: You build your workflows around a specific AI tool. The tool changes its pricing, discontinues a feature, or gets acquired. You’re stuck because your processes depend on it and migrating is painful.

This happens more with AI tools than with most software because the space is moving quickly. Products pivot. APIs change. Free tiers disappear.

Guardrail: Prefer tools that export your data in standard formats. Don’t build automation that works only through one vendor’s proprietary interface if an open alternative exists. Keep your core data (contacts, bookings, financial records) in your own systems, the AI tool should be reading from them, not storing the source of truth. And check what happens to your data if you cancel the subscription. The vendor’s privacy policy should tell you.

When to stay human: a checklist

AI agents work well for tasks that are repeatable, low-stakes, and have a clear right answer. They work badly, or dangerously, when the stakes are high, the context is complex, or the consequences of a mistake are hard to reverse.

Keep a human in the loop when:

• The output could be published or sent to a customer without you seeing it first
• The decision involves money over a threshold you’d notice losing (set this threshold explicitly)
• The task involves personal or health information
• You’re responding to a complaint, a legal matter, or a conflict
• The agent hasn’t run this specific type of task before, novelty is where errors cluster
• A mistake would be hard to explain or hard to undo
• Someone would be embarrassed or harmed by a wrong answer

None of this means AI agents aren’t worth using. The tools in this series, from customer chatbots to document drafting to booking automation, deliver real value when set up with care. The businesses that get the most out of them are the ones who start with one task, watch it closely for a few weeks, and only expand automation after they’ve seen how it behaves.

Slow to trust. Quick to act when it’s proven. That’s the right posture.

If you’re just getting started, our AI agents guide for Australian small business covers the basics, and our industry breakdown shows what’s actually working for tradies, hospitality, and professional services right now.

Sources

• Deloitte AI report refund (Australian government contract): TechRadar, 2025
• Australian Privacy Act small business exemption removal: Schiller Legal, October 2025
• Privacy in Australia 2025/2026 enforcement overview: Spruson & Ferguson, January 2026
• AI hallucination rates in agentic workflows: Seekr, June 2026
• AI hallucination rates on legal queries (Stanford RegLab): Four Dots, citing Stanford research, June 2026
• OAIC, Privacy Act obligations for small business: oaic.gov.au

This post is part of our Agentic AI series, practical guides to AI agents for Australian small business.

📊 Compare AI tools side by side | 💼 Free resources & AI prompt packs

Also see: If You Use an AI Agent, You’re Liable for What It Does →