Table of Contents
Executive Summary
Agentic AI governance is the discipline of controlling agentic AI systems whose risks emerge through actions and decisions over time, not just individual outputs.
In April 2026, Szpruch, et al (University of Edinburgh) published Scalable Runtime Governance for Agentic AI in Financial Services, which demonstrates how traditional model risk management (MRM) and AI governance designed for static models have a structural agentic AI governance gap.
The paper argues that effective governance should shift to runtime governance – continuous monitoring, enforcement, and intervention while the agent is operating – and that relying on traditional MRM for agentic governance will become hard to defend as supervisory expectations develop.
For use cases that demand precision (like many in financial services), the question is no longer whether agentic AI requires a different governance model, but how quickly firms can build one that scales safely.
Why Traditional AI Governance Is Not Enough for Agentic AI
The paper argues that traditional MRM – designed for stable input-output systems – is insufficient for agentic AI governance, because risks materialise over time and can be invisible in a single input-output snapshot.
The authors also show that prompt-based guardrails create “an illusion of control” because they assess semantic reasonableness rather than deterministic precision: “LLM-based verification operates in the same semantic space as the system it checks.”
The reason is architectural: the verifier and verified are both LLM-based, so a check of whether an output seems right is “insufficient by construction” because they may be equally “blind to the errors they are intended to detect.”
For example, a fluently written but numerically wrong answer can pass the check – not because the verifier is careless, but because plausibility and correctness are different questions, and LLMs can only answer the former.
Where correctness is binary – a ratio, a covenant threshold, a policy condition – LLM-based reasoning is the wrong tool for the job.
Their agentic AI governance framework combines five elements:
- Reusable capabilities as the base unit of governance – risk arises from what a system is authorised to do, so validate and monitor the capabilities you need once, reusing them in different workflow combinations.
- Pooled evidence – a new workflow can assemble already-validated capabilities, requiring only incremental validation where risk profiles or capability compositions change.
- A three-layer governance model – system, capability, and trajectory – focusing governance on execution trajectories rather than static outputs.
- Runtime governance – continuous enforcement through semantic telemetry, continuous authorisation, temporal policy conformance checking, and orchestration drift monitoring.
- Human oversight – ‘policy-as-code’ allows conditional and event-driven escalation to humans when runtime controls detect policy violations, threshold breaches, or anomalous behaviour.
Why Agentic AI Governance Matters to Financial Services Firms
Firms that act on this stand to gain four benefits:
- Efficiency – an agentic AI governance architecture built around reusable capabilities will reduce the cost and time of signing off each new agentic deployment.
- Explainability – runtime enforcement with telemetry that records what the system did in governance terms produces the auditable, reconstructable evidence regulators and auditors will expect.
- Proportionality – a tiered control model – four tiers calibrated across agency, authority, impact, exposure, and recoverability – makes governance proportionate to risk.
- Risk avoidance – replacing prompt-based guardrails with deterministic, architecturally-enforced controls will eliminate entire classes of failure that probabilistic compliance cannot prevent.
The paper’s worked example shows the cost of inaction. In a credit memo drafting workflow governed only by prompt-based controls, the paper demonstrates five failure modes:
- Stale data used without flagging.
- A prompt injection attack in a retrieved document that causes the agent to bypass an approval gate.
- A silent numeric error that produces a coverage ratio of 2.82 instead of 1.82 and passes LLM-based checks because the result appears plausible.
- An approval gate reasoned around from conversational signals.
- Gradual orchestration drift in which the fraction of unauthorised releases increases without triggering any per-run violation. None of these failures is exotic – all are consequences of the architecture, not chance.
The paper also positions its framework alongside SR 11-7, PRA SS1/23, NIST AI RMF, and the EU AI Act.
On the paper’s analysis, firms running agentic workflows under MRM programmes designed for static models have a structural agentic AI governance gap – one likely to become harder to defend as supervisory expectations develop.
How the Paper Supports Our Approach to Agentic AI Governance
We have been arguing since we founded Agentic Risks that autonomous systems require an agentic AI governance architecture that goes beyond traditional AI governance, beyond prompt-based controls, and beyond compliance with frameworks not designed for agentic AI.
It is why we established Agentic Risks.
The convergence between the paper and our Governance Framework, Control Framework, and Agentic Risk Appetite and Adoption Strategy is broad, material, and independent – providing formal MRM grounding for our practitioner positions.
On the foundational claim, the paper states that classical MRM becomes “increasingly strained” for agentic systems, and our Governance Framework opens with the assertion that “there is a risk that agentic AI systems may be deployed faster than governance can keep pace.”
The paper’s capability-centric governance model aligns with our emphasis on permissions, autonomy levels, and tool governance.
On compliance versus safety, the paper argues that existing standards were not designed for agentic systems and that compliance with them does not guarantee operational safety. We have made the same argument repeatedly, in our Governance Framework (“compliance [with the EU AI Act] alone will not bring operational safety”), in our webinars, and on our home page: “traditional AI frameworks were not built to govern agentic AI, and regulatory compliance alone will not protect you.”
On prompt-based guardrails, the paper calls industry reliance on them “a fundamental failure in risk mitigation.” Our Governance Framework calls for pre-execution controls (section 2.3) that are “enforced architecturally, not behaviourally” and for operational governance (“the system cannot do X because of control Y”) to replace descriptive governance (“we have a policy that prohibits X”).
On machine-enforceable controls, the paper’s requirement that governance decisions be “expressible as deterministic functions over the governed state” is consistent with our Control Framework, which calls for machine-enforceable constraints (27.04) and pre-completion validation (2.06), and the ‘lethal trifecta’ pre-deployment test (5.08).
On human oversight, the paper argues that it must shift from a runtime safety net to a posture that is “conditional and event driven.” Our Governance Framework’s section 2.7 (Human Oversight Redesigned for Automation Bias) makes the same point: oversight should be concentrated at points where it changes outcomes, not distributed across all decisions.
On runtime governance for agentic AI, the paper’s requirement for continuous enforcement of policy over execution trajectories is consistent with our Control Framework’s Runtime Compliance Monitoring control (24.08), Continuous Monitoring and Telemetry Dashboards control (22.09), and our Governance Framework’s call for dynamic risk assessment so that “one-time validation at deployment should now be the start, not the finish.”
How We Help Firms Build Agentic AI Governance
The question is no longer whether agentic AI requires a different approach to governance. It is whether your organisation is ready for it.
Many firms recognise the governance gap the paper describes but are less sure of its extent or what to do next.
Agentic AI Readiness Assessment
If you are assessing your agentic AI governance readiness, building your roadmap, or unsure whether your governance is ready for autonomous systems, you need a clear view of your current state and what needs to change.
Our Agentic AI Readiness Assessment delivers a complete and systematic view of your firm’s readiness – strategic, technical, and organisational – with a written report within 48 hours: a readiness map and prioritised next steps.
Agentic AI Governance Design
If you have already decided to move forward with agentic AI and now want to ensure your agent estate does not grow faster than your ability to control it, our Agentic AI Governance Design service will set out everything you need to upgrade your governance for autonomous agents.
The outputs are the readiness map (as above), a full governance strategy and design, and a detailed implementation roadmap. If you can assemble everyone on the same day, this service is also available in an executive workshop format.
Pre-Deployment Agentic Risk Assessment
If you want to launch a medium or high-risk agentic workflow but are unfamiliar with the risks and controls you will need, our Pre-Deployment Agentic Risk Assessment identifies the risks in a planned agent, designs the risk treatment plans, and maps the necessary agentic KRIs. The result is a clear view of the controls you will need your engineer to encode, and why.
Frequently Asked Questions
Agentic AI governance is the framework of controls, oversight, and accountability used to manage AI systems that can make decisions, take actions, use tools, and pursue objectives with varying levels of autonomy. Unlike traditional AI governance, which focuses on model outputs, agentic AI governance must also address execution pathways, permissions, tool usage, orchestration, and runtime behaviour.
Traditional model risk management was designed for relatively static input-output systems. Agentic AI introduces risks that emerge across sequences of decisions, interactions, and actions over time. As a result, governance must assess not only what an agent produces, but also what it is authorised to do, how it behaves during execution, and whether it remains within approved boundaries.
Runtime governance is the continuous monitoring and enforcement of policies while an agent is operating. Rather than relying solely on pre-deployment testing, runtime governance detects policy violations, orchestration drift, anomalous behaviour, and unauthorised actions as they occur, enabling intervention before risks escalate.
Prompt-based guardrails rely on an AI model determining whether behaviour appears acceptable. While useful in some situations, they cannot guarantee correctness, enforce permissions, or reliably prevent unsafe actions. For high-risk use cases, firms may need deterministic controls that are enforced architecturally through permissions, policies, workflows, and system constraints.
Financial services firms should focus on governance foundations first: agent inventories, defined ownership, approval processes, runtime monitoring, access controls, audit trails, incident response procedures, and human oversight mechanisms. These controls help organisations understand what agents exist, what they can do, and how to intervene when necessary.
Frameworks such as the EU AI Act, NIST AI RMF, and SR 11-7 provide valuable governance foundations, but were not developed for autonomous agentic systems. Many organisations therefore supplement these frameworks with additional controls that address autonomy, orchestration, runtime monitoring, tool governance, and execution oversight.
The biggest challenge is that risk increasingly emerges from what an agent does over time rather than from a single output. This shifts governance from reviewing isolated decisions to controlling permissions, monitoring execution trajectories, detecting drift, and ensuring that autonomous actions remain within approved limits.
