When AI Acts on Its Own, Who Is Responsible for Mistakes?

Agentic AI is crossing a line that matters a lot to lawyers: it's no longer just assisting people with tasks — it's starting to act on the business's behalf, moving through workflows and responding to events with minimal human involvement. That shift is the subject of a Deloitte Legal report, authored by Melinda Upton, Richard Punt, and Sebastiaan ter Wee, and their framing is direct: the more autonomy AI gains, the more human oversight matters — not less.

The core principle: the system can be wrong, but it can't be responsible

The report's central point is simple but easy to overlook in practice: when an AI system acts within a business process and makes a bad call, responsibility doesn't transfer to the technology. The organization still owns the outcome. That means human oversight can't just live at the design and launch stage — it needs to stay embedded in how the system runs day-to-day and how its actions get reviewed after the fact.

Three things companies are actually trying to avoid

The authors frame the risk around three concrete failure modes:

• Unclear accountability. When an agentic system makes a poor decision, it's often unclear afterward whether the root cause was the design, the deployment, oversight gaps, training data, or escalation rules — and those questions are far harder to answer once something has already gone wrong than if they'd been settled in advance. The report argues legal teams should help define, ahead of time, who owns what and where the approval thresholds sit.
• Regulatory and contractual exposure. The report points to concrete industry examples: in financial services, an agentic tool negotiating terms could expose a company to bad commitments or compliance breaches; in healthcare, a system mishandling patient scheduling or information could create serious downstream consequences. Their recommended fix is a human-in-the-loop model for higher-risk use cases, so sensitive decisions and exceptions get reviewed before they escalate into legal problems.
• Loss of trust. Trust, the report argues, is often the first casualty when agentic AI launches without clear guardrails — customers, regulators, and partners are far more willing to accept autonomous decisions when a company can actually explain how they were made and who remains accountable. Reputational damage from a mishandled case, the authors note, can outlast the original mistake itself.

Three moves that prevent this

To get ahead of these risks, the report lays out three actions for legal departments:

• Define accountability before deployment — deciding in advance what a system can do autonomously, where human approval is required, who owns each business process, and when issues must escalate. Waiting until after launch, the authors warn, often means retrofitting governance around choices that have already been baked in.
• Embed legal into design, not just review — rather than acting as a final checkpoint, legal should sit inside the project team from day one, shaping how agents are used, how their actions are logged, and how the business responds when something needs review. The report frames this as a chance for legal to be seen as an enabler of innovation rather than a gatekeeper.
• Treat oversight as ongoing, not one-time — since agentic systems evolve, a narrow use case can expand into new data, customers, or transactions over time. The report recommends continuous monitoring, explainability built into the system, and governance frameworks that get revisited as scope changes, rather than a single sign-off at launch.

What companies actually gain from doing this early

The report's argument isn't purely defensive — it frames early legal involvement as a speed advantage. Companies that build governance in from the start avoid late-stage redesigns when gaps surface, build more explainable (and thus more trusted) systems, and create a governance model that scales across new AI use cases instead of starting from scratch each time.

Putting it into practice

For legal leaders looking to act on this, the report suggests a fairly concrete starting sequence: map every agentic AI system already in use or development across the business (many legal teams, the authors note, are surprised how many exist with little visibility), flag the highest-risk use cases — those touching customers, contracts, regulated activity, or sensitive data — and get legal a seat at the table for those projects from the outset, not as a final reviewer.

From there, the report recommends setting a small number of non-negotiable standards: what the system can do on its own, what needs human sign-off, how decisions get logged, and who owns the process when something breaks — paired with a regular review rhythm (quarterly, or at major system milestones) so governance evolves alongside the technology.

The report is also candid that this isn't purely a legal skill-building exercise. CLOs need enough technical fluency to engage credibly with product and engineering teams, and need close working relationships not just with CTOs and CIOs, but with Chief Risk Officers and business unit leaders closest to where the AI is actually deployed — plus a direct line to the board, since questions of liability ultimately land there.

The takeaway

The report's closing argument reframes the whole exercise: this isn't really a story about restraining AI adoption. It's a story about legal functions that move early on governance being the ones positioned to move fastest on the technology itself — accountability as an enabler of scale, not a brake on it.