AI Governance Has a Control Problem, Not a Policy Problem
We’re getting good at writing policies about how AI should be used.
Responsible AI principles. Acceptable-use policies. AI risk frameworks. Approval processes. Governance committees.
All of these have a place.
But there is a harder question that I think organisations need to start asking:
What evidence proves those controls actually work?
Because AI is changing the nature of the control problem.
We are moving from AI that simply provides information to AI that can increasingly access data, make decisions, call tools, trigger workflows and take actions.
And much of our traditional assurance thinking still assumes there is a human sitting somewhere in the process.
That assumption is becoming increasingly uncomfortable.
Responsible AI principles. Acceptable-use policies. AI risk frameworks. Approval processes. Governance committees.
All of these have a place.
But there is a harder question that I think organisations need to start asking:
What evidence proves those controls actually work?
Because AI is changing the nature of the control problem.
We are moving from AI that simply provides information to AI that can increasingly access data, make decisions, call tools, trigger workflows and take actions.
And much of our traditional assurance thinking still assumes there is a human sitting somewhere in the process.
That assumption is becoming increasingly uncomfortable.
Autonomy is scaling faster than assurance
Consider a relatively simple AI agent.
It might be able to:
The question is whether the organisation can demonstrate that they are controlled.
There is an important distinction.
“AI must be used responsibly” is not a control.
A control is something you can test, monitor and prove is working.
That distinction matters enormously when the thing being controlled is capable of taking action autonomously.
The four questions I would start with
For any AI capability that can access information or take action, I think there are four fundamental questions.
1. What can it access?
What data can the AI agent see?
Which systems can it connect to?
What identities and credentials does it use?
Can it access sensitive information?
Does it have access to everything the user has access to, or only what it actually needs?
And perhaps most importantly:
Can we demonstrate that access is restricted to what was intended?
This is where traditional security controls such as identity, least privilege, segregation and access management remain critically important.
AI doesn’t make those controls less relevant.
It makes them more important.
Because an agent with excessive permissions can potentially operate at machine speed.
2. What can it change?
Reading information and changing information are not the same risk.
An agent that retrieves a document is fundamentally different from one that can:
Read fast, write slow.
Let agents move quickly where the risk is low.
But when an AI capability can change state, the control requirements should become progressively stronger.
That might mean additional approval, tighter permissions, transaction limits, validation rules, human oversight or stronger monitoring.
The objective isn’t to put a human in front of every AI action.
That simply doesn’t scale.
The objective is to put the right control around the right level of autonomy.
3. Who approved that capability?
This is an area I think deserves much more attention.
It’s easy to ask whether an AI system has been approved.
It’s harder to ask:
What exactly was approved?
Was the approval for the model?
The application?
The use case?
The data?
The tools it can call?
The actions it can perform?
The level of autonomy?
The specific permissions?
These are not necessarily the same thing.
An organisation might approve an AI assistant for a particular business process and subsequently give it access to additional systems or capabilities.
At that point, the original approval may no longer represent the actual risk.
So governance needs to establish a clear relationship between:
Capability → Permission → Owner → Approval → Risk
If nobody can clearly explain who approved an AI agent having a particular capability, that’s a governance problem.
And if nobody owns the decision, it becomes very difficult to assure.
4. What evidence shows it stayed inside those boundaries?
This is the question I find hardest.
And arguably the most important.
It isn’t enough to know what an AI agent was supposed to do.
We need to know what it actually did.
That means being able to answer questions such as:
Because ultimately:
If you can’t produce evidence of what an autonomous system did, how can you provide assurance that it stayed within its authorised boundaries?
Evidence needs to become part of the design
This is where I think organisations need to change their approach.
Evidence shouldn’t be something we try to collect after deployment.
It should be designed into the AI capability from the beginning.
If an agent is allowed to take an action, we should already know what evidence that action will generate.
For example:
Access
We should be able to demonstrate what the agent could access and what it actually accessed.
Decision
We should be able to understand the decision or rule that resulted in an action, within the limits of what the technology can reliably explain.
Action
We should have an auditable record of what the agent actually did.
Approval
We should know who authorised the capability and under what conditions.
Exception
We should know when the agent moved outside expected behaviour and what happened next.
That turns AI governance into something much more tangible.
Not just:
“Do we have an AI policy?”
But:
“Can we demonstrate that the AI operated within the control boundaries we established?”
Don’t treat every AI action the same
One of the mistakes we could make is applying the same control model to every AI capability.
That won’t work.
There is a significant difference between an agent reading publicly available information and an agent making a change to a critical business system.
The control should reflect the potential impact.
A useful way of thinking about it is:
Low-risk read → higher autonomy
Consider a relatively simple AI agent.
It might be able to:
- Read information from internal systems
- Search documents and databases
- Make decisions based on predefined criteria
- Trigger workflows
- Create or modify records
- Call external services
- Generate and send communications
- Potentially initiate actions without a human reviewing every step
The question is whether the organisation can demonstrate that they are controlled.
There is an important distinction.
“AI must be used responsibly” is not a control.
A control is something you can test, monitor and prove is working.
That distinction matters enormously when the thing being controlled is capable of taking action autonomously.
The four questions I would start with
For any AI capability that can access information or take action, I think there are four fundamental questions.
1. What can it access?
What data can the AI agent see?
Which systems can it connect to?
What identities and credentials does it use?
Can it access sensitive information?
Does it have access to everything the user has access to, or only what it actually needs?
And perhaps most importantly:
Can we demonstrate that access is restricted to what was intended?
This is where traditional security controls such as identity, least privilege, segregation and access management remain critically important.
AI doesn’t make those controls less relevant.
It makes them more important.
Because an agent with excessive permissions can potentially operate at machine speed.
2. What can it change?
Reading information and changing information are not the same risk.
An agent that retrieves a document is fundamentally different from one that can:
- Change a customer record
- Modify a financial transaction
- Alter a configuration
- Create an account
- Approve something
- Delete information
- Deploy code
- Trigger an operational process
Read fast, write slow.
Let agents move quickly where the risk is low.
But when an AI capability can change state, the control requirements should become progressively stronger.
That might mean additional approval, tighter permissions, transaction limits, validation rules, human oversight or stronger monitoring.
The objective isn’t to put a human in front of every AI action.
That simply doesn’t scale.
The objective is to put the right control around the right level of autonomy.
3. Who approved that capability?
This is an area I think deserves much more attention.
It’s easy to ask whether an AI system has been approved.
It’s harder to ask:
What exactly was approved?
Was the approval for the model?
The application?
The use case?
The data?
The tools it can call?
The actions it can perform?
The level of autonomy?
The specific permissions?
These are not necessarily the same thing.
An organisation might approve an AI assistant for a particular business process and subsequently give it access to additional systems or capabilities.
At that point, the original approval may no longer represent the actual risk.
So governance needs to establish a clear relationship between:
Capability → Permission → Owner → Approval → Risk
If nobody can clearly explain who approved an AI agent having a particular capability, that’s a governance problem.
And if nobody owns the decision, it becomes very difficult to assure.
4. What evidence shows it stayed inside those boundaries?
This is the question I find hardest.
And arguably the most important.
It isn’t enough to know what an AI agent was supposed to do.
We need to know what it actually did.
That means being able to answer questions such as:
- What data did it access?
- What decisions did it make?
- What tools did it call?
- What actions did it take?
- What changes did it make?
- What permissions did it use?
- Were any actions outside the expected pattern?
- Were exceptions detected?
- Who intervened?
- What happened when a control failed?
Because ultimately:
If you can’t produce evidence of what an autonomous system did, how can you provide assurance that it stayed within its authorised boundaries?
Evidence needs to become part of the design
This is where I think organisations need to change their approach.
Evidence shouldn’t be something we try to collect after deployment.
It should be designed into the AI capability from the beginning.
If an agent is allowed to take an action, we should already know what evidence that action will generate.
For example:
Access
We should be able to demonstrate what the agent could access and what it actually accessed.
Decision
We should be able to understand the decision or rule that resulted in an action, within the limits of what the technology can reliably explain.
Action
We should have an auditable record of what the agent actually did.
Approval
We should know who authorised the capability and under what conditions.
Exception
We should know when the agent moved outside expected behaviour and what happened next.
That turns AI governance into something much more tangible.
Not just:
“Do we have an AI policy?”
But:
“Can we demonstrate that the AI operated within the control boundaries we established?”
Don’t treat every AI action the same
One of the mistakes we could make is applying the same control model to every AI capability.
That won’t work.
There is a significant difference between an agent reading publicly available information and an agent making a change to a critical business system.
The control should reflect the potential impact.
A useful way of thinking about it is:
Low-risk read → higher autonomy
Sensitive read → stronger access controls
Low-impact write → controlled autonomy
High-impact write → strong validation and approval
Critical action → tightly constrained execution and demonstrable oversight
The exact thresholds will differ between organisations and use cases.
But the principle is straightforward:
The greater the consequence of an autonomous action, the stronger the control and evidence requirements should be.
That is not about stopping AI.
It is about making its use sustainable.
The assurance challenge
This also creates a challenge for cybersecurity and internal assurance teams.
Low-impact write → controlled autonomy
High-impact write → strong validation and approval
Critical action → tightly constrained execution and demonstrable oversight
The exact thresholds will differ between organisations and use cases.
But the principle is straightforward:
The greater the consequence of an autonomous action, the stronger the control and evidence requirements should be.
That is not about stopping AI.
It is about making its use sustainable.
The assurance challenge
This also creates a challenge for cybersecurity and internal assurance teams.
- Traditional control testing often asks questions such as:
- Does the control exist?
- Is it documented?
- Is it implemented?
- Can you provide evidence?
Those questions still matter.
But with autonomous AI, we increasingly need to ask:
Does the control operate effectively when the system is making decisions and taking actions dynamically?
That is a different question.
A policy can tell us what should happen.
A configuration can tell us what the system is capable of doing.
But operational evidence tells us what actually happened.
And that distinction is going to become increasingly important.
Governance shouldn’t mean slowing AI down
There is a temptation to respond to AI risk by adding more approvals, more committees and more paperwork.
I don’t think that is the answer.
If governance becomes an obstacle to every low-risk use of AI, people will find ways around it.
Instead, governance should create clear boundaries for autonomy.
Within those boundaries, let AI move quickly.
Outside them, require stronger controls.
And make the boundaries measurable and auditable.
That’s where I think the conversation needs to move.
From:
“Have we got an AI policy?”
to:
“Have we defined what this AI is allowed to do?”
Then:
“Have we controlled those capabilities?”
And finally:
“Can we prove it stayed within those boundaries?”
That last step is where policy becomes assurance.
Make autonomy auditable
AI governance doesn’t need to be about slowing AI down.
It should be about making autonomy auditable.
The organisations that get this right won’t necessarily be the ones with the longest AI policies.
They will be the ones that can clearly demonstrate:
Because AI autonomy is scaling quickly.
Our assurance needs to scale with it.
Read fast. Write slow. Make autonomy auditable.
But with autonomous AI, we increasingly need to ask:
Does the control operate effectively when the system is making decisions and taking actions dynamically?
That is a different question.
A policy can tell us what should happen.
A configuration can tell us what the system is capable of doing.
But operational evidence tells us what actually happened.
And that distinction is going to become increasingly important.
Governance shouldn’t mean slowing AI down
There is a temptation to respond to AI risk by adding more approvals, more committees and more paperwork.
I don’t think that is the answer.
If governance becomes an obstacle to every low-risk use of AI, people will find ways around it.
Instead, governance should create clear boundaries for autonomy.
Within those boundaries, let AI move quickly.
Outside them, require stronger controls.
And make the boundaries measurable and auditable.
That’s where I think the conversation needs to move.
From:
“Have we got an AI policy?”
to:
“Have we defined what this AI is allowed to do?”
Then:
“Have we controlled those capabilities?”
And finally:
“Can we prove it stayed within those boundaries?”
That last step is where policy becomes assurance.
Make autonomy auditable
AI governance doesn’t need to be about slowing AI down.
It should be about making autonomy auditable.
The organisations that get this right won’t necessarily be the ones with the longest AI policies.
They will be the ones that can clearly demonstrate:
- What an AI system can access.
- What it can change.
- Who approved those capabilities.
- And what evidence proves it stayed inside the boundaries.
Because AI autonomy is scaling quickly.
Our assurance needs to scale with it.
Read fast. Write slow. Make autonomy auditable.
Comments