Gathering Evidence of Competence and Awareness in an AIMS Audit

Why Competence and Awareness Are High Stakes in an AIMS

An AI Management System audit is not like auditing a quality or safety system where most of the evidence lives in familiar places. When you are auditing against ISO 42001, you are dealing with systems that can make consequential decisions, process sensitive data, and affect people in ways that are difficult to trace. That makes the competence and awareness requirements in Clause 7 genuinely critical, not just a box to tick before you move on to the interesting stuff.

Clause 7.2 requires the organisation to determine what competence people need in order to affect the performance of the AI management system, ensure those people are competent, take action where gaps exist, and retain documented evidence of that competence. Clause 7.3 requires people to be aware of the AI policy, their contribution to AIMS effectiveness, the implications of not conforming, and the benefits of improved AI performance.

On the surface, these look identical to the equivalent clauses in ISO 9001 or ISO 14001. In practice, the content of competence in an AI context is far more varied and far harder to verify. This article walks through how to gather evidence that actually holds up, including what to ask, what to look at, and where organisations typically fall short.

Understanding What Competence Means in an AI Context

Before you start pulling records, you need to understand what the organisation has determined competence to mean for its own context. This is the first thing to establish, because it shapes every subsequent evidence request.

In a traditional management system, competence often means holding a qualification, completing a training course, or demonstrating experience in a defined role. In an AIMS, competence may need to cover a much broader and more technically varied set of capabilities.

Roles That Typically Require Defined Competence

Depending on the organisation and its AI activities, you might find competence requirements defined for roles such as:

• Data scientists and machine learning engineers who build or configure AI models
• Data governance specialists responsible for training data quality and bias assessment
• AI system owners who are accountable for a specific AI application in production
• Risk and compliance personnel who assess AI risks and maintain the risk register
• Procurement or vendor management staff who engage third party AI providers
• Operations staff who use AI tools in their day to day work and whose decisions may be influenced or partially delegated to those tools
• Senior leaders and board members who have governance responsibilities over the AIMS

The organisation should have a documented determination of who needs what competence. Your first evidence request is to see that determination. It might exist as a competence matrix, a role description with attached competence criteria, or a training needs analysis. The format matters less than whether it is current, covers the relevant roles, and reflects the actual AI activities in scope.

Technical vs Governance Competence

One of the practical challenges in an AIMS audit is that competence splits across two very different domains. Technical competence covers things like understanding model behaviour, recognising bias indicators, interpreting performance metrics, and knowing when a model is drifting. Governance competence covers things like understanding ethical AI principles, applying the organisation's AI policy, conducting impact assessments, and escalating concerns appropriately.

Many organisations have strong technical competence in their data and engineering teams but weaker governance competence across the broader organisation, including in leadership. Others have invested in ethics and policy frameworks but lack the technical depth to assess whether their AI systems are actually performing as intended. A good audit will probe both dimensions.

Reviewing Documented Evidence of Competence

Once you understand what the organisation has determined competence to mean, you can start reviewing the records that are supposed to demonstrate it.

What to Request

Ask for records that demonstrate competence for a sample of roles. A useful sample includes at least one technical role, one governance or risk role, one operational user role, and one senior leadership role. For each, look for:

• The defined competence criteria for that role
• Evidence that the individual meets those criteria, which might include qualifications, training completion records, certifications, performance assessments, or records of supervised practice
• Any identified gaps and the actions taken to address them
• The date the competence was last assessed or confirmed

Pay attention to the gap between what is required and what is evidenced. An organisation might define a competence requirement for AI ethics training but have no record of anyone completing it. Or it might have training records for a course that was completed three years ago with no evidence of refresh or reassessment, despite significant changes in the AI systems being used.

Common Weaknesses in Competence Records

In practice, the most frequent issues you will find in this area include:

• Competence matrices that list job titles but have not been updated to reflect new AI tools or expanded AI use cases
• Training records that evidence attendance but not effectiveness, meaning there is no assessment or evaluation of whether the training achieved its purpose
• Competence defined only for technical roles, with no corresponding requirements for managers, executives, or operational staff who interact with AI outputs
• Third party AI system operators or contractors whose competence has not been assessed at all
• Records held in multiple disconnected systems with no clear ownership or review cycle

If you find a gap between the defined requirement and the evidence, you need to determine whether the gap is a documentation failure or a genuine competence failure. Both can be nonconformities, but they have different root causes and different corrective actions. Ask the auditee to walk you through how they would demonstrate that a specific person is competent. If they struggle to answer that question, the system is not working.

Interviewing People to Test Real Competence

Documents tell you what the organisation claims. Interviews tell you whether those claims reflect reality. This is where an AIMS audit gets genuinely interesting, and where many auditors underinvest their time.

Interviewing Technical Staff

When you sit down with a data scientist or an AI engineer, you are not trying to test their technical knowledge. You are trying to understand whether they are working within a managed system that has appropriate controls. Useful questions include:

• How do you know when a model you are responsible for is performing within acceptable limits?
• What do you do if you identify a potential bias issue in a model's outputs?
• How does the organisation's AI policy affect the way you do your work?
• When did you last complete any training related to AI governance, ethics, or risk?
• Who do you escalate to if you have a concern about an AI system's behaviour?

Listen carefully to whether the person can connect their technical work to the governance framework. A technically skilled person who has no awareness of the AI policy, cannot name the escalation path, and has never heard of an impact assessment is a competence gap in the governance dimension, even if their technical skills are excellent.

Interviewing Operational Users

Operational users are often the most revealing interview subjects in an AIMS audit. These are the people who use AI tools in their work, whether that is a recruiter using an AI screening tool, a credit analyst using a model to assess applications, or a customer service team using an AI assistant. Ask them:

• What AI tools do you use in your role?
• Were you trained on how to use this tool before you started using it?
• Do you know what the tool is and is not supposed to do?
• What would you do if the tool gave you an output that seemed wrong or unexpected?
• Have you ever overridden or questioned an AI recommendation? What happened?

The answers to these questions reveal whether awareness has actually reached the operational level. It is common to find that governance documentation is thorough at the senior level but has not translated into meaningful awareness for the people who interact with AI outputs every day.

Interviewing Senior Leaders

Leadership interviews in an AIMS audit should probe whether top management understands the AI risks the organisation is carrying and whether they can articulate their responsibilities under the AIMS. Useful questions include:

• What are the highest risk AI applications in your organisation and why?
• How do you receive information about AI system performance and incidents?
• What training have you completed to support your role in the AIMS?
• How do you satisfy yourself that the people responsible for AI governance have the competence they need?

If a senior leader cannot answer these questions with any specificity, you have evidence relevant to both Clause 7.2 and Clause 5.1. Leadership competence in AI governance is not a nice to have. It is a requirement.

Testing Awareness: Going Beyond the Policy Poster

Clause 7.3 of ISO 42001 requires people to be aware of the AI policy, their contribution to AIMS effectiveness, the implications of not conforming, and the benefits of improved AI performance. Awareness is not the same as competence, but it is just as important and often harder to evidence.

What Evidence of Awareness Looks Like

Organisations often try to demonstrate awareness through:

• Records of policy communication, such as email distributions, intranet postings, or team briefings
• Signed acknowledgements confirming that employees have read the AI policy
• Training completion records for awareness sessions
• Induction records showing that AI awareness was covered when people joined the organisation or moved into roles that involve AI

These records are a starting point, but they are not sufficient on their own. An email distribution proves the policy was sent. It does not prove anyone understood it. An interview is the only way to test whether awareness is real.

Awareness Interview Questions

When testing awareness, keep your questions simple and direct. You are not looking for a recitation of the policy. You are looking for genuine understanding. Try:

• Can you tell me what your organisation's AI policy says in plain terms?
• How does what you do every day connect to the AI management system?
• What would happen if you used an AI tool in a way that was not approved?
• What is the benefit to the organisation of having a well managed AI system?

If a person cannot answer these questions with reasonable clarity, and if that person works in a role that interacts with AI systems, you have evidence of an awareness gap. Document what they said and what they could not say. The specificity of your notes will determine whether the finding holds up.

For a broader look at how to gather and record evidence that stands up to scrutiny, the article on gathering audit evidence that stands up to scrutiny covers the underlying principles that apply across any management system audit.

Connecting Competence and Awareness to AI Risk

One of the things that distinguishes a strong AIMS auditor from a competent but mechanical one is the ability to connect competence and awareness findings to actual AI risk. This is not just about finding nonconformities. It is about understanding whether the gaps you are finding create real exposure.

Consider an organisation that operates an AI system used to make decisions about loan applications. If the people operating that system have not been trained to recognise when the model is producing discriminatory outputs, the competence gap is not just a clause conformity issue. It is a risk to the people affected by those decisions and a potential legal exposure for the organisation.

When you raise a finding in this area, your nonconformity report should make the connection explicit. Describe the gap, cite the clause, and where relevant, note the AI system or activity to which the gap relates. This gives the finding context and makes it far more likely that the corrective action will address the real issue rather than just generating another training record.

The article on Clause 7 of ISO 42001: Resources, Competence, Awareness and Communication provides a thorough breakdown of the full support clause and how each element connects to the broader AIMS framework.

Triangulating Evidence Across Documents, Interviews, and Observation

The most reliable evidence in any audit comes from triangulation. When a document says one thing, an interview confirms it, and your observation on the floor is consistent with both, you have strong evidence. When they diverge, you have a finding worth investigating.

In an AIMS audit, triangulation for Clause 7 might look like this:

• The competence matrix says the AI risk manager must hold a recognised AI governance qualification. The training record shows the person completed a two day workshop. The interview reveals the person cannot explain the organisation's impact assessment process. The document and record are present, but the competence outcome is not demonstrated.
• The awareness training records show all staff completed the AI policy induction. Three operational user interviews reveal that none of them can describe what the policy requires of them. The training was delivered but was not effective.
• The role description for the AI system owner includes competence requirements for incident response. The interview reveals the person has never seen the incident response procedure for AI systems and does not know it exists. The documented requirement and the actual state of affairs are disconnected.

Each of these scenarios represents a different type of gap, and each requires a different type of finding. Triangulation is what allows you to see the difference between a documentation gap and a systemic failure. For more on this technique, the article on triangulating evidence: documents, people and observation is worth reading before you go into an AIMS audit.

Documenting Your Findings

When you write up findings from Clause 7.2 and 7.3, be specific. Vague findings produce vague corrective actions. A finding that says

competence records were incomplete

Include the role, the requirement, the evidence reviewed, and the gap. If you tested awareness through interviews, note the questions you asked and summarise the responses. Your audit notes are the foundation of your finding, and they need to be detailed enough to support the conclusion you are drawing.

If you are building your skills in this area and want to understand how auditing Clause 7 requirements applies across ISO standards more broadly, the article on audit techniques for Clause 7: verifying competence, awareness and documents covers the common techniques that apply whether you are auditing ISO 9001, ISO 45001, or ISO 42001.

Preparing for an AIMS Audit: What to Do Before You Arrive

If you are preparing to audit an organisation's AIMS for the first time, your pre audit document review should include a request for the competence framework or matrix, the AI policy and its distribution records, training records for a sample of AI related roles, any awareness assessment or evaluation records, and the organisation's documented determination of competence requirements under Clause 7.2.

Reviewing these documents before the on site phase gives you a baseline. You will know what the organisation claims, and you can use your interviews and observations to test whether reality matches the paper. If the documents are thin or unclear, that itself tells you something about the maturity of the system.

For auditors who want to develop deeper skills in auditing AI management systems, Audit Workshop offers ISO 42001 auditor training at both internal and lead auditor levels. The training is built around practical audit scenarios, not just clause walkthroughs, and is delivered by auditors who have conducted real external audits across complex management systems. You can explore the available courses at auditworkshop.com.