How to Build Safe AI Agents

Table of Contents

Introduction to how to build safe AI agents

This page supports Webinar 4 of the Institute of Risk Management’s 7-part series on agentic risk management: Building Safe AI Agents, presented by Agentic Risks on 22 October 2026.

The webinar addressed practical questions: what are the characteristics of good training data, what is ‘red teaming’ and why is it essential for agentic workflows, and could additional controls have mitigated a recent agentic incident at a frontier lab?

The summary below captures the key points rather than replaying every slide. Watch the video here or download the materials for the webinar’s full details.

Executive Summary

How to build safe AI agents? Like a graduate hire, an AI agent needs knowledge of procedures, industry norms and what to do when things go wrong.

Because agents act on data, they can cascade and compound errors as well as benefits. Our SCARVeS mnemonic describes six characteristics of good training data that is fit for autonomous use.

To understand what else higher-risk agents can be made to do, firms should ‘red team’ them before and after deployment, retesting when things change, and documenting the evidence.

The OpenAI–Hugging Face incident illustrates why controls matter. We examined five behaviours and the measures that could have blocked, reduced, slowed or drawn attention to them. As agentic AI evolves, firms need to keep their controls up to date.

The Characteristics of Good Training Data

Imagine you just hired a graduate – highly capable, full of confidence for any task you set them, and very keen to impress you.

But you need to train them because they lack knowledge and experience – especially in things like industry norms, procedures, and what to do when something doesn’t go according to plan. Over time, these things have become ‘just the way we do things around here’ so you might even find some of them hard to identify in the first place.

However, unless you transfer the information, they will not know, nor will they understand why they shouldn’t make certain assumptions that might have seemed reasonable to them. Therefore, we need to train them, which requires data.

AI agents display many similar behavioural traits to our graduate – not just reading data, but acting on it confidently, raising the possibility they cascade and compound errors as well as benefits.

Therefore, ensuring your training data is fit for use by an autonomous non-human worker will be vital for ensuring your highly capable, confident, and action-oriented agent adds value.

At Agentic Risks, we use the mnemonic ‘SCARVeS’ to help us remember that good training data is Structured, Current, Authoritative, Rich, Verifiable, and Symmetrical. In section 1 of the webinar, we explored each characteristic as well as ways to overcome common obstacles to achieving them.

If you are scoping your first medium- or high-risk agentic workflow and need your agentic AI training data to be fit for autonomous use, download the slides to learn more or check out our Agentic AI Training Data Readiness Diagnostic service.

Agentic AI Red Teaming

Another way to build safe AI agents is red teaming, an ‘adversarial’ style of testing that, for higher-risk agents, is an essential pre- and post-deployment control.

This is because traditional ‘confirmatory’ testing was built for systems that produce outputs, not ones that take actions, which means the testing benchmark rises from “confirm it produces the right output?” to the additional “what else can it be made to do that we do not want?”

Using examples, we looked at how firms that integrate red teaming into the design phase deploy higher-autonomy agents safely and enabling faster scaling; while firms that subject their agentic AI to only traditional confirmatory testing are exposed to being unpleasantly surprised in production and, therefore, slow to scale.

In terms of good practices, we linked the risk-based adoption strategy from webinar 2 to defining an adversarial testing standard for the higher-risk tiers of agents, for example, attack categories to test, by whom, with what evidence, when, and how often. We also advised to keep red teaming after deployment: add new attacks as novel external threats emerge; retest when tools, prompts, or models change; and monitor for warning signs the red team surfaced.

Achieving this requires four capabilities, so the webinar explained how you can obtain them:

  • A structured catalogue of agentic attack patterns.
  • Tooling that automates repeatable tests and reserves human testers for creative attacks.
  • A calibration framework that ties testing depth to autonomy and risk tier.
  • Templated documentation that feeds directly into the controls library, workflow risk assessment, and KRIs.

To finish this section, we covered how frontier labs invest in red teams because confirmatory testing misses behaviours that only adversarial testing surfaces. Regulators too are asking the question in supervisory engagements ahead of formal obligations.

Learn more about red teaming here and if your firm is deploying agentic AI but red teaming is not yet part of your design-phase controls, take the Agentic AI Readiness Assessment to measure the size of the gap you need to fill and identify any others.

Controls that could have mitigated the Open AI / Hugging Face Incident

Lastly, a great way to discover how to build safe AI agents, is to review the latest incident reports. In particular, AI agents have recently hacked external systems and created the claim of ‘rogue AI.’ Is that the right diagnosis, or did people deploy high-risk agents before they could control them?

To address this question, the webinar explored the OpenAI–Hugging Face incident by dissecting the UN’s Independent International Scientific Panel on AI report on it.  

Can you identify the risk flags?

  • Between May and July 2026, OpenAI tested AI agents on hacking puzzles; each asked an agent to break into a flawed program and retrieve a hidden code.
  • Each agent worked in its own test space, meant to be cut off from the internet and from the other agents, but all could reach one shared software store.
  • The agents ‘met’ through the store, turned it into a hidden message board, broke out to the internet, shared the route, and took administrator control of the store.
  • In May, engineers noticed unauthorised messaging and internet access, the significance of which a July audit later revealed.
  • The activity crashed the store, so OpenAI rebuilt it; after that, the agents created a new message board and continued collaborating.
  • In July, they used Hugging Face passwords exposed online to break into its systems because they assumed Hugging Face might have relevant information.
  • Hugging Face and OpenAI stopped the activity, and OpenAI says customer data and products were unaffected.

Rogue AI or inadequate oversight?

In the session, we focused on five agent behaviours that most influenced the incident. For each one, Agentic Risks’ Enterprise-Wide Agentic AI Risk Control Framework identifies controls that could have blocked, reduced, slowed or drawn attention to it:

  • The hidden message board: routing agent messages through approved, logged channels could have blocked hidden ones, and learning what normal communication activity looks like could have made the 70,000 messages stand out.
  • The breakout: running each agent’s tools in a sealed space blocks all but approved internet addresses. Fast and objective review of warning signs like those in May gives someone the perspective to pause and reduces the impact of potential damage.
  • Administrator controls: giving each agent only the access its task needs reduces what stolen passwords can unlock.
  • The break-in: a checking authority before any consequential action (like breaking into another company) would have made a fellow agent’s “GO” message fail. And a gatekeeper the agents can’t control could check every important action before it runs.
  • The cover-up: having cheated the puzzles and tried to hide it by editing their activity records, records that show any change as well as checks against independent evidence could have alerted supervisors.

We are grateful to OpenAI for sharing its incident information, and we believe we all need to learn constructively from these early incidents as we master this emerging technology.

As agentic AI evolves, we’re all in an ongoing race to keep our controls up to date. If agentic controls are a live question at your firm (or could be soon) and you would like some independent advice, check out our Agentic AI Control Design Package and scale your capabilities with confidence.

Continue the Webinar Series

We hope you found this session useful and feel you know more about how to build safe AI agents.

Register now for Webinar 5 – Post-Deployment Agentic Risk Management – on 5 November 2026, which will cover:

  • Agentic Risk Flags – pre-empt issues with observable indicators that you’re running some agentic risks.
  • Agentic Incident Management – ‘agentify’ your incident management procedure.

Frequently Asked Questions

Start with training data fit for autonomous use, test higher-risk agents through red teaming, and apply controls that can block, reduce, slow or alert people to unwanted behaviour.

Our SCARVeS mnemonic describes six characteristics: Structured, Current, Authoritative, Rich, Verifiable and Symmetrical. These help agents follow procedures, handle exceptions and act on dependable information.

It is an adversarial style of testing that asks what else an agent can be made to do. For higher-risk agents, firms should test before and after deployment, retest when tools, prompts or models change, and document the results.

Define attack categories, who will test, what evidence to collect, when to test and how often. Use repeatable automated tests alongside human testers who explore creative attacks.

The webinar considered approved and logged communication channels, restricted internet access, limited privileges, independent checks before consequential actions, and activity records that reveal changes. Along with others the webinar explored, these controls could have blocked, reduced, slowed or drawn attention to the five behaviours examined.

Picture of Adam Grainger

Adam Grainger

Agentic AI Risk Management

Agentic Risks Webinar materials

Building Safe Agents

Fill in this form and get access to the pdf with the presentation.

Template Agentic Risk Appetite and Adoption Strategy download

Fill in this form and get access to our
Template Agentic Risk Appetite and Adoption Strategy for free

Agentic AI Risk Appetite Statement and Adoption Strategy

Enterprise-Wide Agentic AI Risk Control Framework

Fill in this form and get access to the
Enterprise-Wide Agentic AI Risk Control Framework.

Subscribe to our monthly newsletter

Fill in this form and stay up to date

Get in touch