Core to the EU AI Act is a risk-based classification system for AI-systems that have specific compliance requirements attached to them. The risk-based classification system for AI technologies introduced in the Act, categorizes AI systems according to their potential impact on health, safety, fundamental rights and emphasizes transparency and human oversight (EP 2024, Article 6), illustrated in Figure 2. It is therefore essential to prohibit certain unacceptable AI practices, establish requirements for high-risk AI systems, and impose obligations on the relevant operators, while also setting transparency requirements for specific AI systems.

Key operators in the AI value chain have been defined by the EU’s regulatory framework, and are important to understand, as they determine compliance requirements based on role (EP 2024, Article 2). Providers are defined as entities that develop an AI system or place a general-purpose model on the market or put the AI system into service under its own name or trademark, whether for payment or free of charge. Deployers are the entities that use an AI system under their authority, except for natural persons using AI systems for personal, non-professional purposes. Finally, an affected person is an individual that uses or is affected by AI systems.

The following three sections of this chapter focuses on transparency requirements, human oversight, and fairness. Transparency requirements within the EU AI Act necessitate that AI systems provide clear and understandable explanations for their decisions, thus calling for the use of an explainability layer to make AI systems’ decision-making processes more transparent. Human oversight ensures that AI systems are monitored based on model performance and explanations and can be corrected when necessary. Fairness principles seek to prevent biases in AI decision-making, and these biases can be identified and addressed using XAI techniques.

In the EU AI Act, AI systems classified as High-risk require providers to supply comprehensive technical documentation. This includes a general description, intended use, and technical details such as system interaction with other hardware or software, and data used for training, including type and relevance (EP 2024, Article 13). The documentation should also be understandable to deployers who may not have specialist knowledge in AI, ensuring they are fully informed of the system’s capabilities and limitations.

AI systems that interact with natural persons must disclose their AI nature unless it is obvious (EP 2024, Article 50). Additionally, non-high-risk systems, including General-Purpose AI, must be clearly marked and labeled, ensuring users can distinguish AI from human interaction in consumer-facing applications.

The EU AI Act strengthens transparency by introducing complaint mechanisms and a right to explanation, allowing individuals affected by AI-driven decisions to seek redress and clarification. This complements the obligation to inform users when they are interacting with an AI system. Under Article 85 (EP 2024), any natural or legal person may file a complaint with a market surveillance authority for suspected violations of the Regulation. Importantly, this mechanism applies to all AI systems, not just high-risk ones. The EU AI act also includes provisions for a right to explanation for decisions made using high-risk AI systems listed in Annex III (with certain exceptions). It states that affected persons have the right to obtain clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken (EP 2024, Article 86).

The complaint mechanism and the right to explanation create a need for explainability in AI decision-making, even though many models are too complex to provide clear justifications. As a result, it can be argued that the EU AI Act implicitly instructs the use of Explainable AI (XAI) techniques to ensure that AI decisions can be understood and communicated.

During the design and development phases of AI systems, features facilitating human oversight must be incorporated into AI systems (EP 2024, Article 14). This involves creating interfaces or tools that allow humans to interpret the system’s outputs effectively. The goal is to maintain human control over the AI system, preventing scenarios where the system operates autonomously without human input or correction.

The Act further mandates that providers must implement measures for continuous monitoring of high-risk AI systems’ operation (EP 2024, Article 14). This monitoring should include mechanisms for detecting and responding to anomalies or unintended behaviors. Humans in oversight roles must have the ability and authority to intervene in real-time to correct or disable the AI system if it behaves unpredictably or deviates from its intended function.

Moreover, the EU AI Act outlines low-risk scenarios, stating that an AI system that does not materially influence the outcome of decision-making should be understood as one that does not affect the substance or result of a decision, whether human or automated (EP 2024, Recital 53). Such systems can be considered exempt from stringent oversight mechanisms as they do not exert a substantial influence on decision-making outcomes.

It is argued that the “human-in-the-loop” model under GDPR and “human-oversight-by-design” under the EU AI Act focuses on ensuring human involvement, but do not define how to ensure that oversight is effective or competent (Laux 2023). Moreover, the obligations placed on AI developers to ensure proper oversight remains underdefined, leaving room for significant interpretation.

LIME uses a simple model to approximate a complex model (Ribeiro et al. 2016). It tries to explain the output of an individual observation by creating a sample with slightly randomly changed inputs and looking at the outputs. A simple interpretable linear model is used to fit this sample and explain the output. A LIME analysis for a specific instance (single loan application) of the credit risk example model is demonstrated in Figure 3. LIME states that this instance has an 82% probability of being a bad loan, seen on the left. In the middle graph it is shown that a low checking account and a low age are the main drivers for a bad loan, whereas the features with the blue coloring are drivers for a good loan. On the right, the value for each feature is shown.

SHAP uses Shapley values that define the importance of an individual explanatory variable (feature), as the relative change in the output, with the specific feature included versus when it is excluded (Lundberg and Lee 2017). A SHAP plot is given for a specific instance, which is demonstrated in Figure 4 (left). A global SHAP explanation of the model is demonstrated in Figure 4 (right), where each dot is an instance of the data and the feature value coloring is the raw relative value of the feature.

The techniques are well available in standard or specific libraries of statistical programming languages such as Python and R, so that with limited effort an explainable layer can be added to an AI system. It may be beneficial in a validation where explainability was not embedded during development (e.g. for low risk systems), to implement XAI to get better understanding about the working of a model and where risks manifest (Zhang et al. 2022). The most applicable packages for each XAI technique are shown in Table 4.

While the EU AI Act does not explicitly assign responsibilities to internal auditors, it imposes clear compliance and documentation obligations on deployers of high-risk AI systems. Deployers are required to conduct a fundamental rights impact assessment (EP 2024, Article 27), maintain up-to-date documentation (EP 2024, Annex C), and cooperate with competent authorities in enforcement actions (EP 2024, Article 27(12)). These responsibilities naturally fall within the scope of internal audit functions, which are typically tasked with providing independent assurance over regulatory compliance, risk management, and control effectiveness. As such, internal auditors are well-positioned to provide assurance that the deployer’s obligations under the EU AI Act are being fulfilled.

Considering internal auditors will be tasked to provide assurance that the outputs of AI systems can be understood and explained , not just for their functionality, but also to verify adherence to fundamental rights, safety, and ethical principles as mandated by the EU AI Act (ECIIA 2024), we must define what assurance means for the internal auditor in this context. According to the IIA’s Global Internal Audit Standards (IIA 2024), assurance is a statement intended to increase the level of stakeholders’ confidence about an organization’s governance, risk management, and control processes over an issue, condition, subject matter, or activity under review (e.g. AI system compliance with the EU AI Act requirements) when compared to established criteria. Internal auditors may provide limited or reasonable assurance, depending on the nature, timing, and extent of procedures performed.

The EU AI Act also does not mandate a specific assurance reporting format for providing assurance on AI systems. In order to have a structured approach for AI assurance and reporting purposes, internal auditors could for example align with the International Standard on Assurance Engagements 3000 (ISAE3000) (IAASB 2024). The ISAE 3000 offers a common methodology for both internal and external auditors such as chartered accountants or IT-auditors to structure their assurance work. Internal auditors may adopt ISAE 3000 principles to guide their evaluations, while external auditors can collaborate with the internal audit function by validating or supplementing the internal audit’s findings.

In the context of AI, internal auditors would apply assurance by evaluating whether the AI system complies with the EU AI Act’s requirements. Assessing the design and effectiveness of internal controls related to AI governance, transparency and oversight, and reviewing documentation such as risk assessments, logs, and human oversight protocols. If an AI system lacks these capabilities and has not been challenged by a second line, internal auditors may need to assess it by implementing an explainability layer themselves. This may be the case when second-line challenge is not available (especially in a non-financial institution environment). In such a case the internal auditor takes up a combined second- and third-line role. If assessing an already implemented and used AI system, it may not be sufficient for the internal auditor to notice a design deficiency in case an explainability layer is missing. If already implemented, the internal auditor may also want to test operational effectiveness by independently adding an explainability layer to the system. It is likely that external expertise would be required to support the internal audit function. Chapter 3 helps to understand the capabilities to look for when hiring external expertise. In case there is a second line function available, it is more likely that the internal auditor feeds back to the first and second line to resolve.

In case explainability is part of the system, internal auditors must evaluate whether this explainability is sufficient for the system’s intended use and matches users’ level of understanding. Moreover, internal auditors should evaluate whether the level of explainability is adequate for the system’s intended use and aligns with users’ ability to interpret it. As highlighted by the European Confederation of Institutes of Internal Auditing (ECIIA), it is considered good practice for organizations to establish policies, procedures, or guidelines that define explainable AI (XAI) requirements and their practical implementation to support compliance efforts (ECIIA 2024).

From an EU AI Act compliance perspective, particularly concerning high-risk systems, as well as for risk management practices for systems with a different classification, outputs need to be understandable and explainable. This means that during the development phase, consideration should be given to either designing interpretability into the system or layering explainability on top of it. If explainability is absent, this indicates a design deficiency, potentially leading to biases and unwanted behavior. Depending on the risk and impact of the system, this may be a blocking issue, in which case there is no added value for the internal auditor for testing operational effectiveness. Alternatively, internal auditors may test operational effectiveness by adding an explainability layer during their review, e.g. also in case the system is already in use (see also Chapter 4). In either case, internal auditors must assess whether the appropriate XAI techniques are employed to satisfy explainability requirements. These requirements must be clearly defined, addressing characteristics such as transparency and user comprehension.

However, it is important to note that XAI does not directly equate to meeting the Transparency and Human Oversight requirements outlined in the EU AI Act. Transparency and Human Oversight entail broader considerations, such as ensuring meaningful human intervention and accountability at critical points in the AI lifecycle, as described in the EU AI Act. While XAI may enhance explainability, internal auditors should carefully evaluate whether the organization’s approach to XAI truly addresses the EU AI Act’s regulatory standards or if additional measures are needed to meet these obligations.

The potential non-compliance and liability risks associated with incorrect decisions made by AI systems, whether direct or indirect, underscores the need for substantiating why certain decisions were made by the system. This is where human oversight becomes essential. High-risk systems must incorporate human interaction within their processes. For systems classified differently, human oversight is a requirement when incorrect outcomes occur, and affected individuals require an explanation. In both scenarios, users need to understand the system’s outputs and have the ability to provide localized explanations. Audit testing should ensure organizations have processes in place to uphold fundamental rights under the EU AI Act. Any model requires explainability, as the ability for customers to file a complaint with a market surveillance authority if they suspect a violation applies to all AI systems covered by the regulation, not just high-risk systems. The EU AI Act also gives individuals the right to an explanation for decisions made by high-risk AI systems listed in Annex III, with some exceptions. Affected individuals must receive clear explanations about the AI system’s role in decision-making and the key factors influencing the outcome. To support this, organizations can use XAI to showcase transparency by providing insights into how AI systems operate. Organizations must also show that users, including those handling complaints, are properly trained to understand the system and its explainability features, ensuring compliance with the Regulation. For the internal auditor it is important to understand that different AI systems may deliver varying levels of accuracy or stability over time. To ensure a thorough understanding of the model, performance-related information should be a part of the explainability process. Well-designed XAI systems incorporate this into the explanations provided to users.

The internal auditor’s responsibilities extend beyond the development phase. Ongoing monitoring and review of the AI system, guided by established policies, must include an assessment of the XAI layer’s effectiveness. A user feedback loop should also be implemented, enabling users to consistently provide input on the system’s performance, particularly regarding any malfunctions or areas for improvement. This feedback is essential for future system improvements, ensuring both functionality and explainability remain robust over time. Below, we break down key areas in greater detail to help internal auditors deliver impactful results.

Internal auditors should also compare the organization’s practices with the guidelines, standards, best practices provided by regulatory bodies on transparency and human oversight in relation to XAI. Industry standards from ISO/IEC, NIST and of course the IIA can serve as a valuable reference for compliance evaluation.

One of the key tools for internal auditors to leverage on, is the IIA updated AI Auditing framework (The IIA 2023). The framework emphasizes the importance of audit trails, logs, and documentation as part of the internal control environment. These records, such as logs of system decisions, human interventions, and model/data updates, are explicitly recognized as critical for demonstrating accountability over time, supporting ToE procedures and enabling traceability of decisions, especially in high-risk or regulated environments.

Through the IIA AI Auditing Framework, internal auditors are encouraged to benchmark organizational AI practices against regulatory guidance (e.g., EU AI Act, U.S. Executive Orders) and industry standards such as ISO/IEC 22989 (AI Concepts and Terminology), ISO/IEC 23894 (AI Risk Management) and the NIST AI Risk Management Framework (ISO 2022; ISO 2023; NIST 2023). Additionally, it recommends that auditors assess whether the organization has adopted transparency-enhancing practices such as clear documentation of model logic and limitations, human-in-the-loop mechanisms and explainability protocols for stakeholders.

While the IIA AI auditing framework does not prescribe a single method for XAI, it acknowledges the growing importance of explainability in AI governance. It suggests that internal auditors should evaluate whether the AI system provides meaningful explanations to users and stakeholders. They should also assess whether an XAI layer is documented, used appropriately and confirm that human oversight mechanisms are in place and effective.