Set clear expectations for when and how people should be involved in reviewing, validating, or intervening in AI system decisions. Oversight should be proportionate to the system’s purpose and risk, and may vary based on the system’s complexity, speed, and potential impact. Even for third-party or embedded AI, clear oversight roles and processes can help manage risk and maintain trust.
There are different models for applying human oversight depending on the role of AI in a system.
In some cases, a person may need to approve each decision before it is acted on (human-in-the-loop).
In others, humans monitor system behaviour and outputs in real time (human-on-the-loop).
A broader model involves shaping how systems are used and evaluated over time, with a focus on overall goals and societal impact (human-in-command).
Human oversight is also critical for generative AI tools, even when they are not making decisions. This includes reviewing outputs, managing how those outputs are used, and ensuring use remains within defined boundaries.
Why it matters
Human oversight provides a safeguard against automation errors and supports accountability, fairness, and transparency. It is essential in high-risk scenarios and helps build trust with users and the public. Oversight is also increasingly expected by regulators, for example, the EU AI Act explicitly requires it for certain use cases. Head to the Library and check the Understanding EU AI Act resources if you want to know more about it.
Human oversight alone does not guarantee safe or ethical AI use. In some cases, humans may over-rely AI systems (a phenomenon known as automation bias) and fail to question flawed outputs. Oversight needs to be active and empowered, not symbolic.
To be effective, people involved in oversight must have the right knowledge, authority, and support to intervene when needed. Look up the Strengthen Human Oversight and Enable Contestability under the AI Trusted pathway for more details on this.
Implementation tips
- Define human oversight requirements for each AI system individually. This ensures that oversight reflects how the system is actually used, the level of autonomy it has, and the potential impact in its specific context, even when two systems are technically similar.
- Establish clear escalation paths and override procedures for each AI system where critical decisions are involved.
- Ensure systems include features to support oversight (e.g. pausing, explanation, or manual override).
- Train staff on their oversight responsibilities for specific systems they interact with.
- Document oversight requirements in system design or procurement documentation.
- Monitor and evaluate oversight arrangements to ensure they remain effective over time for each system.
Support materials
Digital.govt.nz – Responsible AI Guidance for the Public Service (GenAI)
New Zealand public sector guidance on managing AI responsibly, including expectations for human oversight of generative AI.
CSET – AI Safety and Automation Bias
A brief on how automation bias can undermine effective oversight and what can be done to mitigate it.
SGS – Trustworthy AI: Human Agency and Oversight
Explores oversight as a core pillar of trustworthy AI, with practical considerations for implementation.
World Employment Confederation – Guide on Transparency and Human Oversight
Guidance for embedding transparency and human oversight across the AI lifecycle in employment contexts.
Dialzara – Human Oversight in AI: Best Practices
Practical tips for embedding human review and intervention in AI workflows.
