What to Do When AI Goes Wrong

Even well-chosen, well-implemented AI tools will sometimes produce bad outcomes. A hallucination. A wrong output that someone acted on before catching the error. A data breach involving a vendor. A bias-driven recommendation that disadvantaged a client. These events happen across all sizes and types of business, and how a business responds when they do determines both the scale of the harm and the legal and reputational consequences. This guide gives you a practical protocol to follow when something goes wrong with an AI tool in your business.

This guide addresses four types of AI incidents that occur most commonly in Australian SMBs: a wrong AI output that was acted on, a data breach or privacy exposure involving an AI tool, a hallucination with real-world impact, and a bias-driven outcome affecting an individual. Each has a slightly different response profile, but all four follow the same five-step protocol. Understand the protocol first, then read the incident-specific sections for the variations that apply to your situation.

The Five-Step Protocol

Step 1: Stop

As soon as a potential incident is identified, stop the activity that caused it. This may mean stopping use of the specific AI tool, pausing whatever task produced the wrong output, or suspending the process that led to the exposure. The goal at this step is to prevent the problem from getting worse while you assess what happened. Do not attempt to correct or minimise the problem before you understand it. Premature correction actions taken without a full picture of what occurred can compound the harm, complicate the assessment, and in some cases create additional regulatory exposure.

In practice, Step 1 might mean: stopping sending client communications that were drafted using the AI tool, halting a workflow that uploaded documents to the vendor, or immediately reviewing whether the AI-assisted output that is in question has been shared externally. The scope of what you stop should be proportionate to what you know at this point. Stop what is clearly implicated. Leave the rest running until you have a clearer picture.

Step 2: Assess

Understand what happened before deciding what to do about it. The assessment has four questions: What specifically went wrong? Who or what was affected (which clients, what data, what decisions)? What is the current state of the harm (is it ongoing, or was it a one-time event)? And what was the cause (tool error, human error, policy gap, vendor issue)?

Keep the assessment focussed and timely. For serious incidents involving personal information, the assessment needs to be completed quickly enough to meet any regulatory notification timelines that may apply. Under the Notifiable Data Breaches (NDB) scheme, a business must take all reasonable steps to complete its assessment of a suspected breach within 30 days of becoming aware of it. That is a ceiling on the assessment, not the notification deadline: once an eligible data breach is confirmed, notification to the OAIC and affected individuals must happen as soon as practicable, which is normally much faster than 30 days. The assessment step is where you determine whether the incident meets the NDB threshold. See the data breach section below for how to assess this.

Step 3: Contain

Once you understand what happened, take the steps needed to prevent further harm. Containment actions depend on the incident type. For a wrong output that was acted on: identify all instances where the wrong output has been relied on and take steps to correct or withdraw the reliance. For a data exposure: take steps to prevent further access to the exposed data, which may include contacting the vendor, revoking access, or changing credentials. For a bias-driven outcome: stop the process that produced the outcome until you can assess what drove the bias and whether it can be corrected.

Containment should also include preserving evidence. Do not delete logs, emails, AI outputs, or other records related to the incident while it is under assessment. These records may be needed to reconstruct what happened, to demonstrate to a regulator that you acted appropriately, or to support any legal proceedings. Document what you are doing and when throughout the containment phase.

Step 4: Document

Create a written record of the incident and your response. At minimum, record: the date and time the incident was identified, who identified it, what the incident involved (the tool, the use case, the output or exposure), who was affected, what containment steps were taken and when, whether notifications were made and to whom, and what the current status of the response is. This does not need to be a lengthy document. A clear, factual chronology is more useful than a lengthy narrative.

Good documentation has two purposes. It allows you to demonstrate, if asked by a regulator or a client, that you identified the problem, assessed it appropriately, and took proportionate steps. And it creates an institutional record that informs what changes to make to prevent recurrence. Many businesses discover in the documentation step that the same incident was possible for reasons they had not fully understood during the assessment, and that the containment actions they took were necessary but not sufficient to address the root cause.

Step 5: Report

Report the incident to the parties who need to know. This has three possible groups: affected individuals (clients, staff, or others whose data or interests were affected), regulators (OAIC for eligible data breaches, sector regulators for incidents with sector-specific implications), and internal stakeholders (ownership, board, senior management). Not every incident requires external reporting, but every incident requires internal reporting so that the business can learn from what happened and decide whether to change its processes or policies.

When in doubt about whether an incident requires regulatory notification, err on the side of seeking advice rather than assuming it does not. The cost of a conversation with a lawyer is much lower than the cost of a failure to notify that is later identified by a regulator. The OAIC's guidance on the NDB scheme is publicly available and describes the eligibility criteria in plain language. Read it before concluding that notification is not required.

Incident Type 1: Wrong AI Output Acted On

This is the most common AI incident type. An AI-generated output (a summary, a quote, a recommendation, a document) contains an error that was not caught in review, and someone acted on it: sent it to a client, used it as the basis for a business decision, or relied on it in a legal or financial context. The wrong output may have been a hallucination (the AI invented a fact that was not in the source material), a misrepresentation (the AI summarised something inaccurately), or a classification error (the AI assigned the wrong category or value to something).

The severity of this incident type depends entirely on the consequences of the error. An AI-drafted email with a factual error that is caught by the recipient and corrected immediately is a minor incident with minimal impact. An AI-generated quote with a pricing error that has been signed and is now part of a contract is a more significant commercial incident. An AI-generated clinical note with an error that influenced a treatment decision is potentially a serious patient safety incident with regulatory implications. Assess the consequences first, then decide on the response proportionate to those consequences.

For this incident type, the most important containment action is identifying all instances of the wrong output that have been shared, relied on, or incorporated into something consequential, and taking steps to correct or withdraw them. Notifying the affected parties directly and promptly is almost always better than waiting for them to discover the error themselves. The business's ethical and legal obligations to correct a material error exist regardless of whether AI was involved in producing it; the involvement of AI does not create a defence or reduce the obligation to act.

Incident Type 2: Data Breach or Privacy Exposure Involving an AI Tool

This incident type occurs when an AI tool has exposed personal information in a way that was not intended: a vendor has been breached, data was uploaded to a tool in violation of its terms or the business's policy, a tool has shared or retained data it was not supposed to, or a prompt injection attack has caused a tool to output information it should have kept confidential. The Privacy Act 1988 and the NDB scheme establish the obligations that apply when personal information is involved.

To assess whether the NDB scheme applies, consider three questions. First, has there been unauthorised access to, disclosure of, or loss of personal information? Second, is it likely to result in serious harm to any of the individuals whose information was involved? (The OAIC's guidance describes this as the serious harm threshold, and it includes financial harm, physical harm, reputational damage, and serious psychological harm.) Third, does the Privacy Act apply to your business? (It applies to businesses with an annual turnover above $3 million and to certain other categories of business regardless of turnover, including health service providers and businesses that trade in personal information.) If the answer to all three questions is yes, an eligible data breach may have occurred and notification obligations may apply.

For AI-specific data breaches, the most common complication is the vendor relationship. If the breach occurred at the vendor's end (the vendor was hacked, or the vendor's staff accessed your data inappropriately), the vendor has their own notification obligations, but your business may also have obligations to notify affected individuals. The fact that the breach was the vendor's fault does not extinguish your obligations to the individuals whose data you were responsible for. Your data processing relationship with the vendor, and the terms of any data processing agreement, will affect what notification responsibilities each party bears.

The OAIC's Notifiable Data Breaches guidance is at oaic.gov.au/privacy/notifiable-data-breaches. For incidents involving personal information where you are uncertain whether the NDB threshold is met, the OAIC accepts voluntary notifications for incidents that may not meet the eligibility threshold, and seeking advice from the OAIC directly is available via their website.

Incident Type 3: Hallucination With Real-World Impact

An AI hallucination is an output where the model generates a confident-sounding statement that has no basis in the source material and is factually incorrect. Hallucinations that are caught in review before they reach a client or affect a decision are a quality management issue, not an incident. Hallucinations that are not caught become incidents when they result in real harm: a wrong fact in a published document, a fabricated legal citation in a lawyer-reviewed document that was not independently checked, a non-existent regulation cited as the basis for a business decision, or a made-up contact or organisation name that a client tried to follow up on.

Hallucinations are a design characteristic of large language models, not a bug that will be patched. They occur with varying frequency and severity depending on the model, the use case, the quality of the source material, and the specificity of the query, and some tools document their own accuracy safeguards in detail, as our Claude AI review for Australian business does. The risk management approach for hallucinations is structural: design your workflows so that AI outputs are reviewed against authoritative sources before they reach clients or inform consequential decisions. The incident response for a hallucination that has caused harm is the same five-step protocol above. The root cause analysis should identify whether the workflow design permitted an unreviewed AI output to reach a consequential use, and what structural change would prevent recurrence.

Incident Type 4: Bias-Driven Outcome Affecting an Individual

AI bias incidents occur when a tool produces outputs that systematically disadvantage individuals based on protected characteristics such as gender, age, ethnicity, or disability, and those outputs influence decisions that affect the individual's access to employment, services, credit, or other significant outcomes. Unlike wrong-output incidents, which are typically identifiable at the level of a single output, bias incidents are often only visible in patterns across many outputs: the tool consistently recommends against certain names, consistently scores certain profiles lower, or consistently produces content that reflects assumptions about certain groups.

If a single instance of a potentially bias-driven outcome is identified, the immediate steps are to stop the use of the tool for the affected process and to review whether the individual who was affected has a remedy. If the outcome affected employment, tenancy, credit, or access to services, the affected individual may have rights under applicable anti-discrimination law that exist independently of the AI involvement. The business's obligations to provide a remedy for a discriminatory outcome do not change because AI produced the output that contributed to it.

The Australian Human Rights Commission has published guidance on AI and human rights. Sector regulators including ASIC and APRA have also indicated attention to algorithmic bias in financial services. For bias incidents in regulated sectors, seeking regulatory advice early is advisable rather than attempting to resolve internally before considering whether there is a reporting obligation.

After the Incident: Preventing Recurrence

Every AI incident is an opportunity to identify a gap in your governance, risk management, or operational controls and close it. The most useful post-incident question is not "how did this happen" but "what structural change would make this type of incident less likely to occur or less severe if it does?"

Common structural changes that emerge from AI incident reviews include: adding a review step for AI outputs before they reach clients in high-consequence use cases, updating the data handling policy to address inputs that were found to be going into tools they should not be in, updating vendor contracts to include breach notification and data deletion requirements, or changing the tool's use case scope to exclude the types of tasks where errors were most consequential.

The incident documentation you created in Step 4 should feed into a brief review of your AI governance policy. If the incident revealed a gap in the policy (a scenario it did not address, a tool it did not cover, or a use case that had not been assessed), update the policy before closing the incident response. Our AI risk assessment checklist and AI staff policy template cover the structural controls that prevent most common incident types.

Last reviewed: June 2026 | Next review: June 2027

Related reading: our can staff upload customer data to AI tools and our AI data breaches and the NDB scheme.

Try our free AI Compliance Checker to check whether your AI tools meet your compliance obligations.

Does the NDB scheme apply to my business?

The Notifiable Data Breaches (NDB) scheme applies to organisations covered by the Privacy Act 1988. This includes private sector businesses with an annual turnover above $3 million. It also applies to certain businesses regardless of turnover, including health service providers, businesses that buy or sell personal information, businesses that provide residential tenancy database services, businesses that provide credit reporting services, and certain contractors to the Australian Government. If your business is below the $3 million turnover threshold and does not fall into one of the other categories, the NDB scheme may not apply to you, though you may still have obligations under other applicable law. The OAIC's website has a self-assessment tool to help determine whether your business is covered at oaic.gov.au.

What if the AI incident was caused by the vendor, not by my business?

The allocation of responsibility between your business and an AI vendor depends on your contract and the specific circumstances of the incident. However, your business's obligations to the individuals whose personal information was involved generally exist regardless of whether the breach was caused by the vendor. If a vendor held personal information you provided, was breached, and that personal information was exposed, your business may have NDB obligations even though the breach occurred at the vendor's end. Your data processing agreement with the vendor (if one exists) may include breach notification obligations on the vendor's part, indemnity provisions, and defined responsibilities for notifying affected individuals. Review your contract as part of the assessment step. If there is no data processing agreement, that is a gap to close with any vendor handling personal information on your behalf going forward.

What should an AI incident report document contain?

A practical AI incident report should cover: the date and time the incident was identified and by whom; a factual description of what occurred (what tool, what use case, what went wrong); the scope of impact (which clients, staff, or third parties were affected, and what data or decisions were involved); the steps taken in response and when; whether any regulatory notification was made, to whom, and when; and what changes were made or are planned to prevent recurrence. It does not need to be lengthy. A clear factual record of what happened and what was done about it is more useful than a lengthy narrative. Store the incident report in a location that can be retrieved if needed for a regulatory inquiry, legal proceeding, or insurance claim.

Do I need to tell my clients when an AI tool was involved in an error that affected them?

The obligation to notify clients of errors that affect them exists regardless of whether AI was involved. If an error in a deliverable, a quote, a document, or advice has materially affected a client, the business's general obligations around honesty, contractual performance, and professional conduct apply. Whether you are required to specifically disclose that AI was involved in producing the erroneous output is a more nuanced question that depends on the nature of your relationship, the applicable professional standards in your sector, and the specific circumstances. In most cases, the more important disclosure is the error itself and what you are doing to correct it. The involvement of AI is context that may be relevant to explain how the error occurred, but the obligation to notify and correct does not hinge on whether AI was the proximate cause.

How do I prevent the most common types of AI incidents from occurring?

Most AI incidents in small businesses fall into two preventable categories. The first is wrong outputs that bypass review: the fix is a workflow design that requires human review of AI outputs before they are used in consequential contexts, and a clear policy on what counts as consequential. The second is data inputs going into tools they should not: the fix is a clear policy on what data can and cannot go into each AI tool, communicated to all staff who use it. Both of these are covered by a basic AI acceptable use policy. Our free AI staff policy template includes sections covering both. Our AI risk assessment checklist covers the pre-deployment assessment that identifies which use cases carry the highest incident risk before you commit to them.