How to govern agentic AI: what to retain from traditional AI governance, and what new controls you need to implement to have an effective agentic AI governance framework for your enterprise.
Introduction to The Agentic AI Governance Framework
Definition: an agentic AI governance framework is the set of policies, controls, and oversight mechanisms an organisation needs to deploy autonomous AI systems responsibly, ensuring agents act within defined boundaries, remain accountable, and auditable.
There is a risk that agentic AI systems may be deployed faster than governance can keep pace – allowing agents to plan, act, and drift across systems without sufficient oversight and controls.
In such a situation, despite not knowing everything that is going on in your organisation, you can be sure you are responsible for it.
On top of the specific components of the Agentic AI Governance Framework, avoiding this will require three general shifts to the way we view governance:
- Enforcement – from descriptive governance (e.g. “We have a policy that prohibits X”) to operational governance, (e.g. “The system cannot do X because of control Y.”)
- Continuity – from a project with an end state to a permanent, ongoing, and evolving organisational capability built into your operating model.
- Scope – from parallel governance for humans vs AI to unified governance for ‘combined operations’ that comprise human and non-human workers.
For many firms, achieving this will mean evolving their governance: keeping what already works in traditional AI governance and iterating it for agentic AI.
Therefore, the goals of the Agentic AI Governance Framework – a practical guide to how to govern agentic AI – are to identify the parts of traditional AI governance that will remain relevant for agentic AI, identify the new parts you will need, and highlight where the field has not yet settled on answers.
In three parts, we cover:
- The five foundations that still remain relevant – five foundations of traditional AI governance (driven by the main regulations and standards like NIST AI RMF, ISO 42001, and the EU AI Act) that remain valid for agentic AI and should be built on.
- Nine new components needed for agentic AI because even though firms are deploying agentic workflows, the regulations and standards above were not designed for agentic AI and, at the time of writing, none of them even mentions it yet.
- Practical mitigations for five unresolved problems where the field does not yet have complete answers.
Governance applied to adversarial conditions – security threats are a first-order category of agentic risk and are inseparable from governance. Agents can be compromised via prompt injection, manipulated through poisoned data, subverted at the orchestrator level, and exploited through insecure external protocols. The Agentic AI Governance Framework embeds security controls into the governance component to which they most directly apply, e.g. agent identity, execution boundaries, multi-agent trust, and lifecycle management.
House views – as well as the descriptions and explanations of best practice, in every section, we provide our ‘house view’ because while understanding the theory is essential, it only has value if you can also implement it in practice.
Drill-through to implementable controls – completing the theme of ‘theory + practice’, in every section, we also provide a ‘controls checklist’ that refers you to itemized, implementable, and relevant controls across the five categories of agentic risk in the framework of Enterprise-Wide Agentic AI Controls:
- Individual AI Agent Risks.
- Multiple AI Agent Risks.
- AI Agent Security Threats.
- AI Agent Governance Failures.
- Human Capabilities for AI Agents.
Regulatory Mapping Tables – these mapping tables (pdf) demonstrate that the Agentic AI Governance Framework covers every requirement in ISO 42001, the NIST AI RMF, and the EU AI Act, as well as operational requirements that are vital for safety.
Immediate Actions for an Agentic AI Governance Framework
- If you would like some help navigating the various factors to consider, our Agentic AI Governance Design service will create the customised specifications for embedding agentic controls into your organisation.
- Download the Agentic AI Governance Framework in pdf format.
Part 1: Which Foundations of Traditional AI Governance Still Hold?
Traditional AI governance frameworks like NIST AI RMF, ISO 42001, and the EU AI Act are excellent and provide the foundations on which agentic AI governance can be built.
In this section of the Agentic AI Governance Framework, we highlight five aspects from them that remain valid for agentic AI. This is important because many organisations run their first agentic deployments on governance frameworks designed for non-agentic AI, like machine-learning models or generative AI.
1. Policy and Principles
Governance without a policy is aspiration.
Principle: Every organisation deploying AI should define, at board level, what responsible AI use looks like and where accountability sits.
Explanation:
- A policy and principles layer sets the organisational intent: acceptable use rules, risk classification tiers, and named accountability at executive level. Without it, governance exists only as individual judgement.
- Even though a low-risk task is not the same as a low-risk agent, the EU AI Act’s risk classification is useful – unacceptable, high, limited, minimal.
- The test of this layer, therefore, is whether it specifies, for each risk tier, who is responsible and what they are required to do.
- For agentic AI, additional contents should include permitted autonomy levels, in-scope and out of scope workflows, appetite statements covering each category of agentic risk, staff responsibilities, and governance arrangements.
Example: A financial services firm deploys a credit decisioning model. Its AI policy defines credit decisioning as high-risk, names the Chief Risk Officer as accountable, requires an ethics review before deployment, and sets a quarterly review cadence. Without that policy, the deployment might proceed based on informal consensus between technology and business teams – with no clear owner when something goes wrong.
Our House View
A failure at this layer can stem from a gap between a passive policy and an active one.
Many organisations have AI principles that name values – fairness, transparency, accountability – without specifying what those values require anyone to do, by when, and with what consequence if they do not.
Under a regulatory investigation, therefore, such a policy could be more of a liability than an asset.
Our test: Does the policy name owners, define risk tiers, outline how agents in different tiers should be treated, and specify what happens when this is breached?
If not, we help organisations close that gap before adding agentic layers on top.
For organisations that do not yet have a board-level policy on delegating autonomy to non-human systems, our template Agentic Risk Appetite Statement and Adoption Strategy is a good starting point.
Objectives and Planning: you can then translate it into a set of measurable governance objectives you can plan and monitor, for example, percentage of agents with verified identity, time-to-detect behavioural drift, or oversight-review completion rates.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls for Board-level oversight (#23) and the management of AI-related regulatory risk (#24).
2. AI Inventory and Lifecycle Management
You cannot govern what you do not know about.
Principle: Organisations should maintain a central, current register of all AI systems in use, mapped to risk tier, with documented validation and version history.
Explanation:
- A well-maintained inventory is the starting point for governance. Without it, governance lacks a subject, because you cannot control a system you do not know about.
- ‘Shadow AI’ – the use of AI tools outside formal governance processes – is consistently identified as one of the most significant practical risks in current deployments.
- As part of its AI inventory, each organisation should document the resources that support its deployed agents: the data sources they rely on, the tooling and integrations they use, the computing infrastructure they run on, and the human expertise required to govern them.
- For traditional ML and generative AI models, lifecycle management means documented validation at deployment, version control, and periodic review.
- Because agentic AI can learn and adapt, these requirements intensify for agentic systems, highlighting the importance of this part of the foundation.
Example: A professional services firm discovers during a governance review that 23 distinct AI tools are in active use across its business units, of which only 8 were known to the governance team. The remainder had been procured informally by individual teams, with varying degrees of data protection review, risk assessment, and oversight. In this case, the inventory exercise revealed where governance was absent, that is, where unapproved risks were being taken.
Our House View
In our experience, the number of deployed agents is typically greater than the governance team is aware of and is almost never lower.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to agentic lifecycle management (#1) and the implications for vendor relationships (#18).
A specific concern for 2026: shadow agents – AI tools with agentic capabilities that have been connected to business systems without formal review – are an emerging and underappreciated variant of the shadow AI problem. They carry materially higher risk than shadow gen AI use (like taking credit for an LLM’s work) because “they create outcomes, not just outputs.” For mechanism-specific detection methods and a classification protocol for shadow agents discovered in production, see controls 1.08 and 1.09 in the Agent Lifecycle Management section of the Enterprise-Wide Agentic AI Controls.
3. Data Governance
‘Rubbish in, rubbish out’ still applies, and its impact can be magnified.
Principle: The data that AI systems ingest, process, and act upon should be governed with at least the same rigour as any other significant data asset.
Explanation:
- Data governance covers data quality, provenance and lineage, privacy compliance (e.g. GDPR, CCPA, and sector-specific regulations), and bias monitoring.
- The questions this layer asks do not disappear for agentic systems: Is the data fit for purpose? Is sensitive data protected? Are outputs tested for discriminatory outcomes?
- In fact, they intensify, because agents typically require broader data access than static models, and because data breaches via agentic systems can involve active exfiltration rather than ‘just’ passive exposure.
Our House View
It is not possible to achieve good agentic governance without adequate data governance. So, we are clear with clients who have data governance shortcomings: deploying agentic AI on top of them amplifies existing risk.
Our view on the sequencing: organisations that want to deploy agents fast sometimes want to treat data governance as a parallel workstream rather than a prerequisite. We believe this attempt at speed is slower and more costly in the long run: sequencing matters.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to data quality (#10), data protection (#11), the bias monitoring and fairness testing aspects of data governance (#4), and the legal protection of fundamental human rights (#32).
4. Human Oversight and Controls
Monitoring ≠ control.
Principle: Effective human oversight of agentic AI requires checkpoints at which humans can review, override, or halt the system, calibrated to the stakes involved.
Explanation:
- Traditional human oversight frameworks specify who reviews AI outputs, under what conditions, and how decisions are documented.
- Therefore, dashboards, audit logs, monitoring tools, internal reports, and committee reviews are a vital part of your foundations.
- The key distinction for agentic workflows is that recording an undesirable behaviour and preventing it are not the same thing: monitoring ≠ control.
- However, agentic governance also requires the ability to refuse or halt activity when agents operate faster than human review cycles. We address this in the section on ‘Execution Boundaries and Pre-Action Control’ in Part Two.
Example: A bank deploys a credit risk model. Its oversight framework requires a qualified analyst to review every rejection above a defined exposure threshold before it is communicated to the customer. The analyst cannot delegate this review and cannot batch-approve without individual assessment. This is meaningful oversight: a specific human engages at a defined point in the process, with a genuine ability to intervene.
Our House View
A weakness can occur when risk tiers are documented but not encoded as machine-enforceable controls.
Our test: audit whether oversight mechanisms are genuinely differentiated by risk level, and whether humans are actually exercising oversight rather than rubber-stamping outputs. These are different questions, but it can be tempting to only ask the first.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to human intervention (#29) to ensure you do not lose control of your agent (#16).
5. Accountability, Evidence, and Audit Trails
Regulators will audit what your system did, not your principles.
Principle: Agentic governance should produce defensible, reconstructable evidence of your agent’s decisions – not just policies, committee minutes, and plans.
Explanation:
- Here, the question is not whether you have a governance framework, but whether you can reconstruct any specific AI decision from, say, six months ago.
- To put it differently, do you know the inputs, the model version active at the time, the human review that took place, and the reasoning behind the outcome?
- If you need to provide it under regulatory scrutiny, your obligations will depend on your regulator but, for example, the EU AI Act requires a 10-year retention period and specific documentation types, with penalties reaching 7% of global annual turnover.
- The underlying assumption is that each agent has a named owner, which we expect regulators will want to be a Senior Manager or equivalent. If you cannot name them before go-live, you are not ready to deploy.
Example: an airline’s AI chatbot provided incorrect information about bereavement fares. When the matter reached a tribunal, the organisation could not demonstrate that any mechanism existed to ensure the AI cited current policies. Despite having policies, they had no evidence, and the tribunal held them liable. The lesson is not that better policies were needed – it is that policies are just the start of governance: they drive risk appetite and, therefore, the controls you need.
Our House View
Can you reconstruct an AI decision as it occurred – inputs, model version, human review record, and outcome? We believe this test is vital, but in our experience, not every organisation can pass it.
Objective assurance: organisations should establish a scheduled, impartial internal audit programme that periodically verifies agents are behaving within approved parameters and that governance controls remain effective.
Our view on framing: we position this not as a compliance burden but as risk management. The organisations that can prove what their AI did and learn from any incidents are the ones that will survive regulatory scrutiny and emerge from legal challenges intact. In practice, this means that ‘provability’ shifts from being about housekeeping and audit trails to being about regulatory defensibility.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to accountability, explainability, and monitoring (#22), external disclosures (#25), and agentic incident management (#21).
If you would like some help with your agentic AI governance framework, our Agentic AI Governance Design service will create the customised specifications for embedding it into your organisation.
Download the Agentic AI Governance Framework in pdf format.
Part 2: What New Controls Does an Agentic AI Governance Framework Need?
The five foundations of traditional AI governance above remain valid, but if you are deploying agentic AI, they will be insufficient on their own.
This is because only machine-enforceable controls can constrain an autonomous AI that operates faster and at higher volumes than traditional controls can handle.
For a system with autonomous capability, the question changes from, “What output did it produce?” to, “What outcome did it create, in which systems, with what impact, and could we have prevented it?
These more sophisticated questions require more sophisticated controls.
To ensure you are ready, we recommend adding nine new components to your AI agent governance controls.
Security Posture
Security threats are a first-order category of agentic risk: with the Enterprise-Wide Agentic AI Controls dedicating an entire risk category (C: AI Agent Security Threats) to the controls that govern your exposure. Its eight risk areas are:
- Dependency on Data Quality (#10).
- Unauthorised Data Access (#11).
- Unauthorised Data Modification (#12).
- Malicious or Injected Instructions (#13).
- Orchestrator Subversion (#14).
- Agent Fails Under Attack (#15).
- Loss of Control (#16).
- Protocol-Related Risks (#17).
Examples: Restricting agent permissions is both an identity governance decision and a security control. Enforcing execution boundaries is both an operational guardrail and an adversarial defence. Monitoring reasoning chain integrity is both an auditability requirement and a tamper-detection mechanism.
Controls checklist: for each governance component in Part 2, the Controls Checklist in the House View box identifies the specific Category C controls that apply. Click-through for a stand-alone deep-dive into the management of agentic AI security threats.
1. Agent Identity, Permissions, and Prohibitions
Principle: Every agent should have a verifiable, unique identity with time-bound, least-privilege permissions that are managed dynamically and auditable.
- Traditional identity and access management (IAM) systems were designed for human users with stable, pre-defined roles.
- Agentic systems do not fit this model:
- An agent will act on your behalf with specific permissions and prohibitions.
- In multi-agent systems, agents may delegate to sub-agents, creating recursive delegation chains that current authentication systems do not always handle robustly.
- The principle of least-privilege – giving each agent only the minimum access needed for its specific task – is both essential and technically challenging to enforce at scale.
- It is our opinion that current standards are maturing but not yet complete.
- The implication for governing agentic AI is that you should deliberately design your agent identity architecture so you can strengthen it as standards develop.
Our House View
Agent identity is the governance component most likely to require technical engagement alongside governance work. Many organisations deploy agents under generic service account credentials with broad permissions – an architecture that makes accountability impossible and damage limitation very difficult.
On permissions and prohibitions: autonomy is a ‘freedom coin’ that has two sides: freedom to do something your own way, as well as freedom from other prohibitions. When we delegate autonomy, therefore, ambiguity about freedoms becomes a risk factor, which means risk managers must manage ‘both sides of the coin’:
- What could an agent do wrong?
- What else might it choose to do that we should explicitly prohibit?
Permissions and prohibitions need boundaries: define explicit capability boundaries in system terms: data access (read-only vs write), permitted actions, systems the agent can interact with, and financial / operational limits. Enforce them via role-based access, API permissions, and hard-coded constraints, not by prompts alone.
Our view on current tooling: standards for agent identity and dynamic permissioning are maturing but incomplete. In response, we help clients apply least-privilege as rigorously as current tooling allows, and we build a roadmap to strengthen identity management as standards evolve. For clients who want to deploy, we help them mitigate this known risk.
The accountability point: agent identity determines who is responsible for each agent’s actions, making it foundational to the accountability layer that regulators will scrutinise.
2. Dynamic Risk Assessment
Principle: Pre- and post-deployment risk assessments for agentic AI should be grounded in the specific characteristics of each agent: its autonomy level, action-space, access to data and external systems, reversibility of actions, and potential for error propagation.
Explanation:
- Traditional risk assessment treats AI systems as fixed artefacts with stable, predictable behaviour.
- This is an unsafe assumption to make about agentic systems because they can learn and develop new behaviours you may not have tested before deployment.
- Therefore, an agentic risk manager should always ask, “What can this agent do today, and what is the realistic damage if it acts incorrectly, is compromised, or behaves in a way its designers did not intend?”
- Create a baseline through a pre-deployment agentic risk assessment and institute live monitoring of your execution boundaries (see the section on this).
- For agents classified as high-risk under the EU AI Act, you should also create formal technical documentation before deployment and maintain it throughout the agent’s operational life:
- At a minimum this must cover: the system’s intended purpose and architecture; its components, tool integrations, and protocols; training, validation, and testing methodology; monitoring and control measures; and test results against defined metrics.
- Deployers that are public bodies, financial institutions, or insurers and are subject to the EU AI Act should also conduct a Fundamental Rights Impact Assessment (FRIA) before first use of any high-risk AI system.
- To ensure you keep your knowledge up to date, we recommend designating a review cycle or a named role for incorporating evolving scientific and technical knowledge about agentic AI capabilities, failure modes, and emerging risks into your risk assessments.
Our House View
We see scope for the unwary to underestimate how an agent’s design could introduce freedom for its risk profile to change over time. For example, if you loosely control an agent that has write access, operates autonomously, and can take irreversible actions, even if you give it a low-risk task it has the potential to drift to a higher-risk area that you may not have explicitly prohibited.
Our approach:
- In our Pre-Deployment Agentic Risk Assessment, we work through the agentic risk factors with clients systematically before assigning any risk tier. We examine four broad sources of risk to identify the risks before structuring and scoring them, designing the controls and KRIs, and inputting them for testing.
- And our Post-Deployment Agentic Risk Assessment uses evidence-led risk flags to surface agents that are operating at a higher risk level than anyone in the organisation had recognised – sometimes including agents that have been running in production for months.
On the importance of variety: to reduce the risk that governance blind spots reflect the narrow view of a single team, risk assessment and governance should actively seek input from a diverse range of perspectives, including those of individuals from groups most likely to be affected by an agent’s decisions.
Regarding shadow agents specifically, the dynamic risk assessment process is also the right moment to surface agentic tools operating outside formal governance. Effective agentic AI risk management requires a complete inventory: the assessment cannot be completed accurately if it is incomplete.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to unpredictability and errors (#1), behavioural drift (#5), regulatory risk (#24), agentic risk management (#27), and legal protection of fundamental human rights (#32).
3. Pre-Execution Boundaries, Control, and Security
Principle: Agentic governance should include the ability to refuse inadmissible actions before they execute – not merely to record and review them afterwards.
Explanation:
- Traditional AI governance is predominantly observational: it monitors outputs, reviews logs, and conducts audits.
- This is appropriate when a human decides what to do with an AI output, but not when the AI itself takes the action across systems – and at speeds humans cannot match.
- Because you cannot watch every step, effective oversight must move away from action-level review to constraint-level governance:
- In practice, agentic governance requires execution boundaries – limits on what an agent can and cannot do that you define pre-deployment and encode into them, so they are machine-enforceable at the point of execution.
- For lower-stakes actions, the boundary may be automated rather than human-reviewed – but it should exist and be enforceable.
- For higher-risk or irreversible actions, such as financial transactions, data deletion, external communications, access to sensitive systems, and data exfiltration, a pre-execution control should be able to halt the action pending human review.
- Set anomaly thresholds in your KRIs that trigger automatic pause and escalation.
- Require human approval for any action outside pre-approved parameters.
- Execution boundaries are a primary security defence against adversarial action:
- An agent that cannot be instructed to exceed its boundaries by a malicious input is more resilient than one that relies on its own reasoning to reject that input.
- This means your boundaries must be enforced architecturally, not behaviourally: prompt injection attacks, injected tool outputs, and poisoned external content should all be unable to cause the agent to act outside its approved parameters, regardless of how convincing the instruction appears.
Example: A customer service agent is authorised to process refunds up to £500 without human approval. An attempted refund of £4,800 triggers an execution boundary: the action is suspended, a reviewer is notified, and the agent continues with other tasks. This boundary is built into the architecture. If the agent is later compromised and instructed to process a large fraudulent refund, the boundary halts the instruction regardless of how convincing it appears. Contrast this with an architecture where the same policy exists in a document (not code), giving the agent practical discretion in the refund system. This is the difference between descriptive and operational governance that we call for in the Introduction to this document.
Our House View
We believe the distinction between descriptive governance and operational governance is the most important conceptual shift in moving from traditional to agentic AI governance. Descriptive governance says: “We have a policy that prohibits X.” Operational governance says: “The system cannot do X without approval.”
Prompt injection attacks should be a particular concern that you should manage proactively by sanitising all input formats (not just prompts) before they reach the agent’s reasoning layer.
A diagnostic question: if the agent you are designing received a malicious instruction to take a harmful action, what would prevent it from executing? If the answer relies on the agent’s own judgement or a policy document, there is a control gap; you might consider performing a Pre-Deployment Agentic Risk Assessment.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to unpredictability and errors (#2), unauthorised data modification (#12), malicious or injected instructions (#13), agent fails under attack (#15), protocol-related risks (#17), and collateral damage (#19).
4. Reasoning Chain Integrity
Principle: Governance should apply to intermediate reasoning steps, not just final outputs.
Explanation:
- Agentic systems can alight on correct outcomes despite flawed intermediate reasoning.
- This is a ‘near miss,’ the chance of which should be managed by governing an agent’s reasoning process as well as the outputs.
- Controls should require agents to produce inspectable reasoning artefacts at defined stages: planning, execution, and verification.
- Each stage should be logged before progression to the next.
- Where latent reasoning is used (reasoning that is not verbalised or logged) compensating controls such as surrogate traces or bounded deployment scopes should apply.
Example: A compliance agent recommends approving a filing. The output looks correct, but the intermediate reasoning contains a factual error that happened to cancel out – a near-miss, this time. With intermediate-stage check, the flawed reasoning is caught before the agent is trusted with higher-stakes decisions. Without it, the error would be invisible, and the organisation might not be so lucky next time.
Our House View
Logging what agents produce is not the same as understanding how they got there. We help clients define which reasoning stages require inspectable artefacts and design release gates between stages that are calibrated to the risk of what the agent does next. For LLM-based agents where internal reasoning is partially opaque, we also help clients understand the limits of what they can and cannot audit – and build a defensible position around those limits.
If you would like some help with your agentic AI governance framework, our Agentic AI Governance Design service will create the customised specifications for embedding it into your organisation.
Download the Agentic AI Governance Framework in pdf format.
5. Multi-Agent Governance
Principle: When multiple agents interact, governance should address the system as a whole – mapping interaction patterns, trust boundaries between agents, and the conditions under which an error from one agent could propagate to others.
Explanation:
- Most governance frameworks address individual agents. But some agentic deployments involve multiple agents working together, for example, an orchestrating agent directing specialist sub-agents, a sequential pipeline where one agent’s output becomes another’s input, or a group of agents operating in parallel.
- Each pattern introduces a new risk:
- An error from one agent can propagate to downstream agents that treat it as reliable.
- An agent compromised by a prompt injection attack can pass malicious instructions to agents that trust its outputs.
- Two agents optimising for related goals can interact in ways neither their individual governance rules nor their designers anticipated.
- Therefore, risk should be understood not just at the agent level but at the system level, and not just as a technical problem but as a stakeholder impact problem because failures that begin at the individual level can propagate to the organisational level.
- Furthermore, a specific multi-agent risk that is easy to overlook is uncontrolled agent replication, which takes place when agents to create new agents without explicit approval.
- Each unapproved replica is an ungoverned actor – inheriting the ability to act operation your behalf while inheriting none of the controls you have instituted.
- To avoid this, you should prohibit agent creation without explicit prior approval, enforce quotas through a central registry, and apply automatic expiry timers to any approved replica agents to limit persistence and propagation risk.
- Another multi-agent risk is orchestrator subversion – where faulty or malicious control logic causes agents to run endlessly, coordinate unsafely, or act outside approved boundaries. Controls include continuous validation of orchestrator logic, orchestration rate limits, and emergency stops that function even if the orchestrator logic itself is corrupted.
Example: A supply chain system deploys three interconnected agents: demand forecasting, inventory management, and procurement. The forecasting agent hallucinates an anomalously high demand figure. The inventory agent, treating this as fact, flags a shortage. The procurement agent autonomously places a large emergency order with an external supplier. By the time a human reviews the procurement log, the order has been executed and cannot be reversed without penalty. In this scenario, no individual agent behaved outside its defined parameters – the failure was at the system level.
Our House View
It is easy to ‘sleep-walk’ into multi-agent risk: you begin with single-agent pilots and accumulate complexity incrementally until agents are interacting in production, at which time traditional governance controls may be unable to keep the in check because of the speed with which they can operate and find new ways forward.
What we recommend: map agent interactions explicitly before deployment. This means documenting which agents can pass instructions to other agents, which agents can trust outputs from which sources without verification, and what the realistic blast radius of a single agent failure would be across the system.
On the liability dimension: multi-agent failures create particularly complex accountability questions. When a failure results from the interaction of two agents built and deployed by different teams – or by different organisations – the question of who is responsible may not have a straightforward answer. This is one of the reasons we address liability as an unresolved matter in Part Three.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to overlapping or conflicting agent actions (#7), inter-agent orchestration and communication (#8), uncontrolled agent replication (#9) and orchestrator subversion (#14).
6. Lifecycle Governance for Systems That Learn and Adapt
Principle: Agentic systems require ongoing, continuous governance across their entire operational life – not one-time validation at deployment followed by periodic reviews.
Explanation:
- Traditional model risk management validates before deployment and relies on periodic monitoring thereafter.
- This model assumes predictability: the model does not change, data distribution shifts gradually, and a quarterly review is sufficient to detect material drift.
- Agentic systems make these assumptions unsafe: they learn from and adjust to changing environments: new web content, updated external APIs, evolving tool integrations, changing user behaviours, and interactions with other agents that are themselves evolving.
- Therefore, effective lifecycle governance requires gradual rollout to production, continuous monitoring at machine speed, automated anomaly detection capable of flagging unusual patterns in real time, defined fail safes and kill switches.
- You should also extend your cost monitoring and controls: without machine-enforceable limits, agentic systems can drive up compute, token, and API costs through runaway loops, excessive retries, or deliberate resource exhaustion attacks. Because of this, unusual spending patterns become an important metric to watch.
- As a matter of good housekeeping, we recommend a) regular re-testing under conditions that reflect the current production environment rather than conditions at initial deployment, and b) a mechanism to enable and respond to stakeholder feedback on unexpected agent behaviours, policy concerns, or governance failures.
- Lastly, for AI systems classified as ‘high-risk’ under the EU AI Act, you should include a defined incident classification protocol that covers the Act’s specific external reporting obligations.
Example: An enterprise HR agent is deployed to handle employee queries, update records, and process leave requests. At deployment it passes a defined test suite. Three months later the organisation introduces a new HR policy. The agent does not get updated, so continues to process requests against the old policy. The issue is not detected until an employee challenge reveals a pattern of incorrect approvals. A continuous governance model would have flagged the policy change as a re-testing trigger and monitored for decision patterns inconsistent with current policy.
Our House View
A common gap can occur between pre-deployment testing and continuous monitoring: organisations invest significantly in testing before deployment and then treat deployment as the end of the governance process rather than the ‘end of its beginning’.
If you have deployed an agentic workflow without a formal pre-deployment assessment, you might consider a proactive Post-Deployment Agentic Risk Assessment to ensure is does not exhibit any red flags and the implementation of a reactive Agentic Incident Management capability.
Five distinct governance moments:
- Pre-deployment testing asks whether the agent is safe and fit for purpose in its intended environment.
- Gradual rollout asks whether it is behaving as expected in early production.
- Continuous monitoring asks whether it is continuing to behave as expected as its environment evolves.
- Regular re-testing and stakeholder feedback ensure you keep the agent’s knowledge sources up to date.
- Management review assesses whether its design remains adequate given changes in other deployed agents, operating conditions, regulations, and any incidents since the previous review.
Prompt and knowledge source version control:
Behavioural drift in agentic systems can originate from changes to the inputs it relies on – updated knowledge bases, modified prompt templates, or evolved tool integrations. Effective lifecycle governance therefore requires:
- Version control applied to all knowledge sources and prompt templates, with changes treated as deployment events requiring re-validation.
- Defined drift thresholds – quantitative where possible – that trigger automatic rollback or human review when agent behaviour deviates from its approved baseline.
- A prompt change log maintained alongside the agent registry, so that any behavioural change can be traced to its root cause without reverse engineering.
Our view on the governance model: effective agentic AI governance should be a continuously maintained operational capability.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to agent lifecycle management (#1), behavioural drift (#5), cost and resource overheads (#20), and agent incident management (#21).
7. Human Oversight Redesigned for Automation Bias
Principle: For human oversight to remain effective, its design should acknowledge that, unless managed proactively, automation bias, alert fatigue, and operational pressure may erode oversight quality over time.
Explanation:
- Automation bias – the tendency to over-trust systems that have previously performed well – is a well-documented psychological phenomenon.
- But it becomes a structural governance risk when humans are asked to oversee autonomous agents capable of generating high volumes of output. As agents tackle more work, the temptation to approve in bulk or reduce oversight checkpoints grows.
- An obvious move is to delegate the checking to another agent, which requires careful design boundaries and multi-agent risks.
- At some, though, a person will need to sign off, and, in their designs, organisations should recognise the qualities and limitations of being human:
- A clear taxonomy of agent action types mapped to oversight requirements.
- Training for human overseers on common agentic failure modes so they know what to look for.
- Contextual and digestible approval requests, rather than raw logs that humans cannot meaningfully evaluate.
- Regular audits of oversight quality, not just oversight volume.
Example: A compliance agent processes regulatory filings and flags items for human review. In the first month, the compliance officer reviews each flagged item carefully. By month six, the agent has a strong track record, and the officer has begun approving items in batches with minimal review. The agent then flags a genuinely problematic filing, which the officer approves without reading carefully. At this point, the theoretical control failed in practice.
Our House View
While organisations may deploy agents to reduce human workload, their increased capacity creates a meaningful oversight burden on the humans at critical points. We help clients resolve this by being specific about where oversight adds genuine value and where it does not.
Our view: routine oversight on low-stakes, reversible actions should be designed out of the process, but meaningful oversight on high-stakes, irreversible decisions is at the heart of risk management. Our goal with clients is to design well-calibrated human involvement at the points where it actually changes outcomes.
On automation bias specifically: we help clients train staff to understand AI limits and test their ability to operate without it. We recommend rotating AI challenge roles, promoting critical thinking, and reducing over-reliance on AI by recording why outputs are accepted. We also believe that the overseers of high-volume agents may need capacity limits on the number of approvals they can safely perform within a specified time period.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to detecting early warnings before you lose control (#16), human oversight and intervention (#29), and staff over-trust (#30).
8. End-User Responsibility and Transparency
Principle: Organisations deploying agentic AI should ensure that end-users – whether employees integrating agents into their workflows or customers interacting with them – have sufficient information and capability to use agents responsibly and exercise meaningful oversight.
Explanation:
- Traditional AI governance treats end-users primarily as recipients of AI outputs, like drafting an email you can edit and send.
- By comparison, agentic systems create outcomes, like responding to emails on your behalf.
- Therefore, when delegate tasks to agents we make governance decisions about what the agent is authorised to do, what data it can access, and when to intervene.
- This makes us responsible for the impact of an agent’s actions on other users:
- Transparency – others should know that they are interacting with an agent, what it can do, and how to escalate.
- Education – others who integrate agents into their workflows need to know its limits and common failure modes.
Example: An investment firm deploys a research agent. The research analysts who understand its limits use it to accelerate their research while applying professional judgement to its outputs. Analysts who do not receive adequate training may treat the agent’s outputs as definitive, fail to spot hallucinations, and submit investment cases to fund managers that contain inaccurate data and assumptions. The governance failure is in neither the agent nor the human but in the failure to equip users to exercise effective oversight of it.
Our House View
End-user responsibility is consistently underweighted in governance frameworks because it is seen as a training and communications problem rather than a governance one. We believe it is both.
On tradecraft erosion: as agents take on entry-level tasks, the foundational skills those tasks previously developed may atrophy. For regulated professions, this has professional qualification and liability implications that go beyond individual competence. We help our clients embed their response to this from the outset of their journey into agentic AI.
Our view on transparency to external users: users interacting with an agentic system have a right to know they are doing so, what the agent is authorised to do on their behalf, and how to reach a human if needed. We believe organisations that do not meet it are exposed – regulatorily and reputationally.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to external disclosures (#25) and staff over-trust (#30).
9. Organisational Readiness for Combined Operations
Principle: Organisations deploying agentic AI should treat workforce integration as a governance obligation. Specifically, you should manage the impact on human workers, preserve the professional capabilities that agents depend on, and build the organisational structures needed to run combined human and non-human operations sustainably.
Explanation:
- Agents are not tools in the traditional sense – they receive delegated tasks, take independent action, and produce outcomes their human colleagues are accountable for.
- This creates a new kind of working relationship that governance must address directly:
- Human workers whose roles are adjacent to deployed agents should be informed, involved, and supported through this transformational development.
- Getting this wrong and creating disengagement could be risky because, without knowledgeable and engaged human oversight and maintenance, the agents may fail.
- Preventing de-skilling must also be a priority, especially for roles where professional judgement, regulatory accountability, or domain expertise are required for effective oversight and, therefore, compliance.
- This requires deliberate design at the heart of governance: clarity about which decisions agents can make independently, which ones require human involvement, and how responsibility is allocated when agent and human contributions combine.
- Without this design, accountability can become diffuse, making governance unenforceable.
- Organisations should document the competencies required to govern agentic AI at each role level, assess current capability against that baseline, and allocate dedicated budget and resources to close identified gaps before deployment.
- Organisational readiness also has a change management dimension. Resistance to agentic workflows (whether from scepticism, anxiety, or concern about role displacement) is a governance risk if it drives shadow workarounds, inconsistent adoption, or selective disengagement from oversight responsibilities.
Example: A financial services firm deploys an agent to handle routine client onboarding, with human relationship managers retained for complex cases and exception handling. The deployment is technically sound, but the relationship managers receive no explanation of what the agent can and cannot do, no training on when and how to intervene, and no clarity on whether they are accountable for the agent’s decisions. Within three months, some managers have disengaged from oversight entirely, others are duplicating the agent’s work unnecessarily, and one has begun routing cases around the agent to preserve their workload. The root cause of this governance failure is not in the technology but in the absence of effective change management.
Our House View
Don’t just build agents; build an agentic capability.
Why? Because technical deployment is, at best, pointless without workforce integration, with consequences to oversight quality, escalation behaviour, and accountability.
On sequencing: workforce integration should begin in the agent design process to ensure the organisation is ready, engaged and aware of the capabilities and limits of agentic AI.
On the combined operations model: we believe the medium-term direction for many organisations is a genuinely mixed workforce, in which human ‘handlers’ oversee agents. The organisations that will be best positioned for that future will be those that treat the clear, distinct separation of labour between the human and the AI as a deliberate design choice, with rather than an outcome that will manage itself.
Governance infrastructure should be unified, not parallel: when humans and agents operate in the same environment and access the same systems, governance controls should be unified. This is because the alternative, parallel governance controls (e.g. audit trails, identity systems, and policy enforcement mechanisms) risks compliance gaps at every junction between them.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to agentic workflow capability (#26), agentic risk management (#27), organisational change management (#28), human oversight and intervention (#29), and staff over-trust (#30).
If you would like some help with your agentic AI governance framework, our Agentic AI Governance Design service will create the customised specifications for embedding it into your organisation.
Download the Agentic AI Governance Framework in pdf format.
Part 3: What Are the Unresolved Problems – and How Do You Mitigate Them?
It is easier to write about certainties, but we all know that agentic AI is an evolving technology, and, at the time of publication, we believe there are five important areas that are unresolved.
For those who cannot wait for these debates to conclude, our Agentic AI Governance Framework offers practical risk mitigations.
As the technology evolves, we will evolve our position in future versions of this document.
1. Liability Across the Agentic Value Chain
There is no settled legal or regulatory framework for allocating liability when an agentic system causes harm:
- Frontier AI developers argue that responsibility should sit with deployers – the organisations closest to where agents operate.
- Deployers argue that model developers design the guardrails and cannot disclaim responsibility.
Some regulators have taken explicit positions: India’s Reserve Bank has stated that regulated financial institutions remain liable for any customer loss regardless of how it arose.
But positions vary significantly across jurisdictions, and the question is unlikely to be fully settled without major litigation or regulatory enforcement action.
Our House View
For vendor-supplied agentic systems, the best available position to adopt for any regulatory or legal challenge is to have created a defensible position that demonstrates you took reasonable steps.
Our position is that organisations should maintain an AI supply-chain risk policy that maps the AI components, models, and data sourced from third parties, assigns risk ownership for each dependency, and specifies how third-party AI risk information is gathered, assessed, and acted upon:
- Document the accountability chain.
- Address liability for agent behaviour in the contract.
- Disclosure of the full agent architecture (including any sub-agents or chained models).
- Contractual right to audit action logs.
- Clarity on where the model’s instructions can be overridden and by whom.
- Confirmation that your data is not used for model training.
- SLAs covering agent failure and rollback.
Systematic framework for managing third parties: we recommend a documented policy governing all third-party relationships in the agentic AI lifecycle – covering procurement due diligence, ongoing supplier risk assessment, contractual obligations, and the allocation of governance responsibilities between the organisation, its vendors, and any sub-processors.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to bias and fairness (#4) and vendor / API instability (#18).
2. Pre-Execution Governance
This framework advocates for execution boundaries and pre-action control because we believe it is the right principle.
However, implementing full pre-execution governance across all actions in complex, fast-moving multi-agent systems exceeds what current tooling can reliably deliver.
Agent identity standards are maturing but incomplete:
- Dynamic permissioning for multi-principal scenarios remains technically challenging.
- The volume and speed of agent actions in production makes comprehensive real-time governance difficult to implement without creating operational bottlenecks that defeat the purpose of agentic deployment.
- The practical approach is tiered: rigorous pre-execution controls for high-stakes, irreversible actions; automated rule-based controls for medium-risk actions; autonomous execution with post-hoc audit for lower-risk actions.
This involves the management of residual risk that will need to be revisited as tooling matures.
Our House View
At the current stage of the technology’s development, we believe the most useful deliverable is a tiered architecture that applies the most rigorous controls where they matter most – with a clear roadmap for strengthening those controls as standards develop.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to agent identity confusion or exploitation (#6) and the prevention of collateral damage (#19).
3. How to Scale Oversight?
Every serious governance framework advocates for meaningful human oversight of agentic systems.
But as the number of agents, the frequency of their actions, and the complexity of their interactions increase, continuous human oversight becomes operationally impossible.
The alternative – automated monitoring with human review triggered by anomaly detection – is promising but raises its own questions: who monitors the monitoring system? And how do you prevent alert fatigue from eroding automated monitoring in the same way it erodes direct oversight?
There are no settled answers to how governance scales effectively from pilot deployments to enterprise-wide agentic infrastructure.
This is a particularly significant for regulated sectors, where regulators expect meaningful human oversight but have not yet specified what this means at scale. The organisations that have thought carefully about this in advance will be better positioned when regulatory expectations crystallise.
Our House View
We ensure that, from the outset, our clients design their oversight models with scale in mind and treat oversight quality – not just volume – as a governance metric. We also help clients document their oversight model explicitly, so they have a defensible position for a regulator.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to loss of control (#16) and human oversight and intervention (#29).
4. We Can Log What Agents Do. We Cannot Always Explain Why.
Minimum conditions for auditability require observability and ‘assessability’, which means the evidence should be interpretable, not just present.
This is an active area of technical development, and the current state of the art does not fully meet the standard that we believe regulatory scrutiny will require.
We believe that regulators and courts will want more than action logs, instead wanting the reasoning chain as evidence of what produced those actions, yet that chain may not be fully reconstructable with current technology.
- Observability – reconstructing what an agent did, in what sequence, with what tools is achievable with well-designed logging infrastructure.
- Explainability – understanding why the agent made the planning and reasoning choices it made is significantly harder, particularly for agents built on large language models whose reasoning processes are not fully transparent.
Because of this, when something goes wrong, it may not be possible to determine the root cause.
Our House View
We advise clients to 1) invest in logging infrastructure that captures the maximum feasible detail about agent actions, tool calls, and decision points, and 2) design governance programmes that can be strengthened in increments as technical standards and regulatory expectations in this area develop.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to inconsistent reasoning chains (#3) and accountability, explainability, and monitoring (#22).
5. Agentic Governance is an Ongoing Activity
The properties that make agentic AI valuable are the same properties that create governance risk – autonomy, scalability, the ability to act at machine speed across multiple systems.
This is a feature, not a temporary bug.
The result is a structural tension, so for as long as you use agentic AI you will need to manage that tension.
Our House View
Agentic governance needs to be a permanent, evolving, and ongoing effort to manage competing forces, not a one-off task that you can complete. As a result, we advise clients to embed governance into their operating models so that whenever they change the operating model they can check whether their governance remains fit-for-purpose.
Controls checklist: see the Enterprise-Wide Agentic AI Controls for itemised controls relating to agentic risk management (#27).
A Note on The EU AI Act
This Framework covers what is necessary for operational control and agentic AI compliance.
It is important to note that this exceeds obligations under regulations like the EU AI Act.
Therefore, an approach that focuses purely on following the rules will not bring operational effectiveness.
Indeed, if lower operational control creates issues, it may increase the chance of regulatory scrutiny.
There are, however, several prerequisites for lawful deployment under the EU AI Act that are not governance activities but that we note here for completeness:
- Prohibited AI Practices – before governance questions arise, deployers should also verify their agents do not engage in practices prohibited absolutely under Article 5 of the Act, particularly the prohibitions on the manipulation and exploitation of vulnerabilities, which apply regardless of risk tier and which execution boundary design should actively prevent.
- Market-placement obligations relating to CE marking, a conformity assessment and a declaration of conformity.
- Quality management system – where the EU AI Act’s Art. 17 specifies a QMS structure, this framework embeds the substantive elements into its sections on Policy and Principles, Dynamic Risk Assessment, Lifecycle Governance, and Accountability. Together, they address the required content needed to satisfy Art. 17.
- Obligations of deployers of high-risk AI systems – this framework covers these obligations as they relate to human oversight, monitoring, data quality, and incident reporting. The EU also includes requirements to register an AI system before first use, to notify workers, and to retain logs.
- Transparency Obligations – for agents with conversational or interactive interfaces, the transparency disclosure obligations under Article 50 apply independently of high-risk classification and are addressed by this framework’s treatment of end-user responsibility in section on End-User Responsibility and Transparency.
Download the Agentic AI Governance Framework in pdf format.
Working with Agentic Risks on your Agentic AI Governance Framework
This document describes what an effective agentic AI governance framework looks like.
If you would like some help navigating the various factors to consider as you implement what fits best for your organisation, our Agentic AI Governance Design service will create the customised specifications.
The output is a documented, board-ready governance design: a gap analysis against this framework and customised specifications for an integrated governance framework that your risk function can own, your auditors can examine, and your regulators can scrutinise.
It is designed for organisations whose agentic deployments have grown beyond what their existing AI governance structure was designed to manage, as well as for those that want to build governance in from the start rather than retrofit it later.
To start, book a free 30-minute consultation.
Frequently Asked Questions
Effective human oversight of agentic AI means more than monitoring logs and reviewing outputs after the fact. It requires defined checkpoints at which a human can review, override, or halt an agent, calibrated to the stakes involved – with higher-risk and irreversible actions requiring human approval before execution. It also means designing against automation bias: the well-documented tendency to over-trust systems that have previously performed well. In practice, the Agentic AI Governance Framework calls for a clear taxonomy of which agent actions require human review, training overseers on agentic failure modes, and auditing oversight quality – not just oversight volume.
