Monitoring and Measuring AIMS Performance Under ISO 42001 Clause 9.1

Why Clause 9.1 Matters in an AI Management System

Most organisations implementing ISO 42001 spend considerable energy on the earlier clauses. They map their AI context, write their AI policy, complete the risk assessment, and build their statement of applicability. By the time they reach Clause 9.1, there is often a temptation to treat it as a formality. It is not.

Clause 9.1 is where your AI Management System (AIMS) proves it is actually working. Without meaningful monitoring and measurement, you have a system built on assumptions. You believe your AI controls are effective. You believe your objectives are being met. You believe your risks are being managed. But without data, belief is all you have, and that will not satisfy a certification auditor or, more importantly, your organisation's leadership.

This article walks through what Clause 9.1 of ISO 42001 requires, how to build a monitoring and measurement framework that holds up in practice, and what auditors look for when they assess this clause during a certification or surveillance audit.

What Clause 9.1 Actually Requires

ISO 42001 Clause 9.1 follows the same general structure as its counterparts in ISO 9001, ISO 14001, and ISO 27001. The high level structure means the intent is consistent across standards. But the content of what you are monitoring is quite different when AI systems are involved.

The clause requires your organisation to determine:

• What needs to be monitored and measured
• The methods for monitoring, measurement, analysis, and evaluation
• When monitoring and measurement shall be performed
• When the results shall be analysed and evaluated
• Who is responsible for carrying out these activities
• When results shall be reported to relevant parties

You must retain documented information as evidence that the results have been produced. That last point is often where organisations fall short. They monitor informally, hold conversations, and make adjustments, but leave no trail that demonstrates the monitoring actually happened in a systematic way.

The clause also requires you to evaluate the performance and effectiveness of your AIMS. This is not just about tracking whether AI systems are running. It is about whether the management system itself is achieving what it was designed to achieve.

The Challenge of Monitoring AI Systems

Monitoring quality or environmental performance is relatively well understood. You measure defect rates, customer satisfaction scores, or energy consumption. The indicators are usually quantifiable and the data sources are fairly obvious.

AI systems introduce a different set of challenges. The outputs of an AI system may be probabilistic rather than deterministic. A model that was accurate six months ago may have drifted. A system that performs well on one dataset may behave differently when the input distribution shifts. These are not traditional quality problems, and your monitoring framework needs to reflect that.

Some practical examples of what you might monitor under an AIMS include:

• Model accuracy or performance metrics against defined thresholds
• Rates of AI system outputs being overridden by human reviewers
• Frequency and nature of AI incidents or near misses
• Completion rates for AI impact assessments at planned intervals
• Staff completion of AI awareness and competence training
• Number of identified AI risks that have been treated versus those still open
• Supplier or third party AI system performance against agreed criteria
• Complaints or concerns raised by affected parties about AI outputs

The right indicators will depend on the nature of your AI systems and the risks you identified during your Clause 6.1 risk assessment. There is no universal list. What matters is that your chosen indicators are genuinely connected to the risks and objectives you have defined.

Connecting Monitoring to Objectives and Risks

One of the most common weaknesses auditors find in Clause 9.1 implementations is a disconnect between what the organisation says it cares about and what it actually measures. The AI objectives established under Clause 6.2 should directly inform the monitoring indicators you choose under Clause 9.1.

If your AI objective is to ensure that your recruitment screening model does not produce biased shortlists, then your monitoring should include a mechanism for detecting and reviewing potential bias in outputs. If your objective is to maintain human oversight of high risk AI decisions, then your monitoring should track the frequency and quality of that oversight.

The same logic applies to your risk register. If you identified a risk around model drift as a significant concern during your risk assessment, then monitoring model performance over time is not optional. It is the mechanism by which you demonstrate the risk is being controlled.

This is also where your AI risk assessment under Clause 6.1.2 becomes practically useful beyond the planning phase. The risks you identified should be traceable through to the monitoring activities you have established. An auditor will follow that thread.

Designing a Monitoring and Measurement Framework

A practical AIMS monitoring framework does not need to be complex, but it does need to be deliberate. The following structure works well in practice.

Step 1: Define Your Indicators

Start with your AI objectives and your risk register. For each objective, ask what data would tell you whether you are on track. For each significant risk, ask what data would tell you whether the control is working. These questions will generate your candidate indicators.

Be specific. An indicator like AI system performance is acceptable is not measurable. An indicator like model precision remains above 85 percent as measured monthly against the validation dataset is measurable. The more specific you are at this stage, the easier the rest of the framework becomes.

Step 2: Define Your Methods

For each indicator, document how it will be measured. This includes the data source, the tool or process used to collect the data, and any calculation or aggregation method. If you are using automated monitoring tools, document what they capture and how frequently. If measurement relies on manual review, document the process for that review.

Methods should be reproducible. If a different person performs the measurement next month, they should get a comparable result. Inconsistent measurement methods undermine the value of your data.

Step 3: Define Frequency and Responsibility

Clause 9.1 requires you to determine when monitoring and measurement shall be performed and when results shall be analysed. These are two separate questions. You might collect data continuously but analyse it monthly. You might conduct a formal review quarterly but report to leadership annually.

Assign clear ownership. Someone needs to be accountable for each monitoring activity. In smaller organisations this might be the AI system owner or the person responsible for the AIMS. In larger organisations with multiple AI systems, responsibility may be distributed across system owners with a central coordination function.

Step 4: Define Reporting and Communication

Monitoring data is only useful if it reaches the people who need to act on it. Define who receives the results of your monitoring activities, in what format, and how often. Management review under Clause 9.3 is one formal channel, but significant performance issues should not wait for the annual management review. Build in escalation triggers for when indicators fall outside acceptable ranges.

What Auditors Look for Under Clause 9.1

When an auditor assesses Clause 9.1 during a certification or surveillance audit, they are looking for evidence of a systematic, documented approach. The questions they ask and the evidence they request will typically cover the following areas.

Is There a Defined List of What Is Being Monitored?

Auditors will ask to see documentation that identifies your monitoring indicators. This might be a performance monitoring plan, a KPI register, or a section of your AIMS procedures. It does not need to be a sophisticated document, but it does need to exist and be current.

Are the Methods Documented and Consistent?

The auditor will want to understand how measurements are taken. For technical AI performance metrics, they may ask about the tools used and who has access to the results. For process indicators like training completion rates, they will ask how this data is collected and verified.

Is There Evidence That Monitoring Actually Happened?

This is where many organisations struggle. Documented information is required as evidence of results. This means records, reports, dashboards, or meeting minutes that demonstrate monitoring activities were carried out at the defined intervals. If you say you monitor model performance monthly but cannot produce twelve months of records, that is a nonconformity.

The article on evaluating monitoring, internal audit and management review in an AIMS covers the auditor perspective on this in more detail.

Are Results Being Analysed and Used?

Collecting data is not the same as using it. Auditors will look for evidence that monitoring results are being analysed, that trends are identified, and that the results feed into decisions. If your monitoring data consistently shows a risk control is underperforming but no action has been taken, that is a problem the auditor will raise.

Does Monitoring Cover the AIMS Itself, Not Just the AI Systems?

This is a distinction that trips up many organisations. Clause 9.1 requires evaluation of the performance and effectiveness of the AIMS, not just the performance of individual AI systems. This means monitoring whether your management system processes are working. Are impact assessments being completed on time? Are corrective actions being closed? Are AI incidents being reported and investigated? These are AIMS performance questions, not just AI system performance questions.

Common Nonconformities Under Clause 9.1

Based on audit practice, the following are the most frequently raised issues against this clause.

• No documented monitoring plan. The organisation monitors informally but has not documented what is being monitored, how, or by whom. The requirement for a systematic approach is not met.
• Indicators disconnected from objectives and risks. The organisation measures things that are easy to measure rather than things that matter. The monitoring framework does not connect to the AIMS objectives or the risk register.
• No records of results. Monitoring happens but no documented information is retained. The auditor cannot verify that monitoring occurred or what the results showed.
• Results not being analysed. Data is collected but sits in spreadsheets or dashboards without being reviewed, interpreted, or acted upon.
• Monitoring covers only AI system performance, not AIMS effectiveness. The organisation focuses entirely on technical metrics for its AI systems and does not monitor whether the management system processes are working.
• Responsibilities not assigned. No one is clearly accountable for monitoring activities, leading to inconsistent or incomplete implementation.

Practical Tips for Internal Auditors Reviewing Clause 9.1

If you are conducting an internal audit of your organisation's AIMS and you are assessing Clause 9.1, the following approach will help you gather meaningful evidence.

Start by asking to see the documented monitoring plan or equivalent. Review it against the AI objectives and risk register. Are the indicators logically connected? Are methods, frequencies, and responsibilities defined?

Then ask for records of the most recent monitoring cycle. Review the actual data. Does it cover the period it claims to cover? Is it complete? Has anyone signed off on it or formally reviewed it?

Ask the person responsible for monitoring to walk you through the process. How do they collect the data? What do they do with it? Who sees the results? What happens when an indicator falls outside the acceptable range?

Finally, trace a monitoring result through to a decision or action. If the monitoring identified a performance issue, what happened next? If everything has been green for twelve months, probe whether the thresholds are set at a level that would actually detect a problem.

This kind of process tracing is consistent with the approach described in resources on internal audit requirements under ISO 42001 Clause 9.2, which covers how the internal audit function itself should be assessing AIMS performance.

Integrating Clause 9.1 with Management Review

The outputs of your Clause 9.1 monitoring activities are a primary input to the management review under Clause 9.3. This is not a coincidence. The standard is designed so that performance data flows from monitoring into management review, and management review decisions flow back into the AIMS through updated objectives, resources, or corrective actions.

If your monitoring framework is weak, your management review will be weak. Senior leaders will be reviewing impressions and anecdotes rather than data. They will not be able to make informed decisions about whether the AIMS needs to change.

Make sure your monitoring outputs are summarised in a format that is useful for management review. Raw data from a model performance dashboard is not suitable for a management review agenda. A summary that shows trends, flags issues, and compares results against objectives is what management needs to see.

The connection between monitoring and management review is one of the areas examined in the article on management review of an AIMS under Clause 9.3, which explores what inputs are expected and how the review should be structured.

Building Monitoring Capability Over Time

Organisations that are new to ISO 42001 often start with a minimal monitoring framework and build from there. That is a reasonable approach, provided the starting point is genuinely systematic and not just a placeholder.

In the first year, focus on getting the basics right. Define your indicators, document your methods, assign responsibilities, and produce records. Even if your indicators are not perfectly calibrated, having a functioning monitoring cycle is more valuable than having an elaborate framework that does not operate in practice.

As you accumulate data, you will start to see patterns. Some indicators will turn out to be more informative than others. Some thresholds will need adjustment. Some monitoring activities will reveal risks that were not on your original register. This is how a monitoring framework matures, and it is exactly the kind of continual improvement that ISO 42001 is designed to drive.

If you are building or reviewing an AIMS and want to develop practical skills in auditing AI management systems, Audit Workshop offers ISO 42001 training at foundation, internal auditor, and lead auditor levels. The courses are built around real audit practice, not just clause recitation, and are delivered by auditors who have worked across multiple industries and management system standards.