What Happens if Your AI Tool Gets Breached? Australian SMB Guide

If your AI vendor is hacked, a staff member pastes customer records into a consumer AI tool, or a shared AI workspace is accessed by the wrong person, your business may have mandatory notification obligations under Australia's Notifiable Data Breaches scheme. Most small businesses have a breach response plan for their own systems. Far fewer have considered what happens when the breach involves an AI tool they use every day. This guide covers the three most common AI breach scenarios and what you need to do in each one.

The three most common AI breach scenarios

Not every AI incident is a notifiable data breach. The key questions are whether personal information was involved and whether affected individuals face serious harm. Here are the three scenarios most likely to affect Australian SMBs using AI tools.

Scenario 1: Your AI vendor is hacked

If your AI tool provider suffers a breach affecting your account data or the content of your prompts, the NDB scheme obligation to assess the breach and, where required, notify the OAIC and affected individuals falls on your business. Not the vendor. As the entity that collected and used personal information, your business holds the NDB assessment obligation. The vendor notifying you about their breach is the vendor fulfilling their contractual obligation to you. They are not notifying the OAIC on your behalf. See the OAIC's guidance on NDB obligations at oaic.gov.au.

Check your service agreement with the vendor. Most enterprise-tier AI contracts include breach notification clauses committing the vendor to alert you within a set timeframe. Your 30-day assessment period under the NDB scheme starts from when you first suspect or become aware of a breach, not from when the vendor tells you about it.

Scenario 2: Staff share customer data with a consumer AI tool

Many consumer AI tools, including free tiers of general AI assistants, use conversation data for model training by default. If a staff member pastes customer records, patient notes, or financial details into one of these tools without disabling training, that data may have been shared with the vendor and incorporated into their training process. This may constitute an unauthorised disclosure of personal information.

Whether it rises to a notifiable data breach depends on the sensitivity of the information and the likelihood of serious harm. A customer's first name and a general question about their account has a lower harm profile than health records, financial account numbers, or identification documents. Assess each incident on its own facts.

Consider a recruitment agency owner whose office manager used to manually strip candidate ID numbers and dates of birth from resumes before filing them, a slow job that ate up twenty minutes per shortlist. Instead of doing that by hand, the office manager started pasting full resumes into a free AI chatbot to speed up candidate summaries, not realising the tool used that data for model training. When the agency owner found out, she had to review every resume the office manager had pasted, check whether driver's licence numbers or medical details were included, and decide within the 30-day window whether the OAIC needed to be notified. Twenty minutes saved on filing turned into weeks spent on a breach assessment.

Scenario 3: A shared AI workspace exposes data to the wrong person

Team-plan AI tools often allow multiple users to access shared conversations, saved prompts, and documents. If workspace permissions are misconfigured, or a former employee's account is not deactivated, someone outside your organisation may gain access to AI workspace content containing personal information. This is an access breach and it falls within the NDB scheme if the content qualifies as personal information and the likelihood of serious harm is present.

This scenario is preventable with basic offboarding procedures. Revoking access to AI tools and shared workspaces should be part of your standard staff offboarding process, alongside email and file system access. Review your offboarding checklist now if AI tools are not already included.

What counts as an eligible data breach

Under the NDB scheme, an eligible data breach has two elements. First, there has been unauthorised access to, unauthorised disclosure of, or loss of personal information held by your organisation. Second, a reasonable person would conclude that the disclosure is likely to result in serious harm to one or more of the affected individuals.

Breach scenario	Serious harm likely?	NDB likely to apply?
AI vendor hacked, prompts included customer names and email addresses only	Low to medium	Assess case by case
AI vendor hacked, prompts included financial account details, health records, or passwords	High	Yes, very likely
Staff pasted client medical records into a consumer AI tool	High	Yes, very likely
Staff pasted a customer's first name and general enquiry into a consumer AI tool	Low	Unlikely, but document your assessment
Former employee accessed a shared AI workspace after leaving	Depends on content	Assess based on what was accessed

The 30-day assessment window

When you first suspect a potential eligible breach, you have 30 days to complete a reasonable assessment of whether it qualifies under the NDB scheme. This 30-day window starts when you first become aware of the potential breach, not when you confirm it. Begin your assessment immediately. Do not wait for formal confirmation from a vendor or IT team before starting the clock.

If your assessment confirms an eligible data breach, notify the OAIC and affected individuals as soon as practicable. There is no fixed post-assessment deadline, but the OAIC interprets this requirement strictly. Delays of weeks or months without a clear justification will not be treated favourably during any investigation.

Steps to take when an AI incident occurs

1. Contain immediately. Suspend access to the affected AI tool or workspace. Revoke any compromised credentials. Ensure no further personal information is being exposed or shared.
2. Document what happened. Record the date you became aware, what information was involved, whose data was affected, and the likely cause. This documentation protects you during any subsequent OAIC assessment.
3. Start your 30-day assessment. Determine whether the breach involves personal information and whether serious harm is likely. You are assessing NDB eligibility, not fault. Be objective.
4. Contact your AI vendor. If the breach is on their systems, request a written incident report from them confirming what data was accessed and what steps they have taken. Keep this on file.
5. If eligible: notify the OAIC. Complete the Notifiable Data Breach form at oaic.gov.au. The form covers breach type, information involved, estimated individuals affected, harm assessment, and containment steps taken.
6. Notify affected individuals. Tell individuals whose personal information was involved what happened, what data was exposed, what harm they may face, and what steps they should take. Notify directly by email or letter wherever possible.
7. Review and prevent. Update your AI tool policies to prevent a recurrence. Add AI breach scenarios to your staff training if they are not already included.

When you do not need to notify

Not every AI data incident is a notifiable data breach. If the personal information has low sensitivity, if the data was properly encrypted and the key was not compromised, or if the likelihood of serious harm is genuinely low after a reasonable assessment, you may not have an NDB notification obligation. Document your assessment regardless. If the OAIC investigates later, a documented assessment showing you considered the question carefully is far better than no record at all.

The OAIC has a data breach self-assessment tool on their website to help you determine whether your incident is eligible. For incidents involving health information, financial information, or large volumes of personal data, consider getting a privacy lawyer to review your assessment before you decide not to notify.

Related reading: our can staff upload customer data to AI tools, our AI and the Privacy Act guide, and our AI data breaches and the NDB scheme.

Related reading: our Claude AI review for Australian business.

Try our free AI Privacy Risk Scorer to score your current AI tool setup against Privacy Act requirements.

Related reading: our AI governance by region.

Does the NDB scheme apply to my small business?

The NDB scheme applies to entities covered by the Privacy Act 1988, including organisations with an annual turnover above $3 million, health service providers of any size, credit providers, and businesses that trade in personal information regardless of turnover. If your business falls below the threshold and is not in a special category, the NDB scheme does not apply directly to you. However, APP 11 (security of personal information) obligations still apply to APP entities of all sizes, requiring reasonable steps to protect personal data from misuse or unauthorised disclosure.

Does my AI vendor notify the OAIC on my behalf?

No. If an AI vendor suffers a breach affecting your data, the vendor may have their own NDB notification obligations as an APP entity. But they are not notifying the OAIC on your behalf. You remain responsible for notifying the OAIC about any eligible breach involving personal information you collected and used. The vendor's notification to you is their obligation to you as their customer. Your notification to the OAIC is your obligation to the regulator.

What do I tell affected customers?

Your notification to affected individuals must include a description of the breach, the types of personal information involved, what steps the individual should take to protect themselves, and your contact details for further enquiries. Use plain English and focus on what happened to their data, not on technical AI system details. Where possible, notify directly by email or letter rather than via a general website notice. The OAIC allows website notification only when direct contact is not practicable.

What are the penalties for failing to notify?

Failure to comply with the NDB scheme is an interference with the privacy of an individual under the Privacy Act. The OAIC can investigate complaints, issue enforceable undertakings, and seek civil penalty orders. For serious or repeated privacy breaches by organisations, penalties can reach $50 million or more. The risk of an OAIC investigation following an unreported breach is real, particularly if affected individuals later complain directly to the OAIC.

Can I use an AI tool to help write the breach notification?

Yes, you can use an AI tool to help draft your notification letters and OAIC submission, provided you are not entering personal information from the breach into the AI tool when drafting. Use the AI to help with structure and plain-English drafting. Enter only generic descriptions of the breach type, not names, email addresses, or other personal details of affected individuals. Review the final draft carefully before sending and ensure it is accurate to the specific circumstances of your breach.