The European Regulation EU 2024/1689 came into effect on June 13, 2024 (EP 2024). It is better known as the European Union Artificial Intelligence Act (EU AI Act) and it ranks as one of the first pieces of legislation attempting to regulate AI technologies. While the Act has already entered law, not all requirements are yet enforced and the roadmap for implementation extends until 2027, the timeline is shown in Figure 1.

Figure 1.

EU AI Act Timeline.

Risk-based approach

Core to the EU AI Act is a risk-based classification system for AI-systems that have specific compliance requirements attached to them. The risk-based classification system for AI technologies introduced in the Act, categorizes AI systems according to their potential impact on health, safety, fundamental rights and emphasizes transparency and human oversight (EP 2024, Article 6), illustrated in Figure 2. It is therefore essential to prohibit certain unacceptable AI practices, establish requirements for high-risk AI systems, and impose obligations on the relevant operators, while also setting transparency requirements for specific AI systems.

Figure 2.

Risk-based classification system for AI systems.

The EU AI act is linked to the European Union’s General Data Protection Regulation (GDPR), which addresses concerns about the opacity of decision-making processes by automated systems. The GDPR includes provisions for automated decision-making (ADM) based on personal data, establishing protections and safeguards for individuals when subjected to decisions based solely on automated processing. (EP 2016, Article 22). GDPR clarifies that data subjects are provided with meaningful information about the logic involved in ADM (EP 2016, Article 13, 14). Yet, the regulation does not offer clear guidelines on how such information should be conveyed in the context of ADM systems, leaving room for interpretation and inconsistency in implementation (Wörsdörfer 2024). In the context of the EU AI Act, transparency is underpinned by guided and structured rules, delineating to the obligations of AI system providers and deployers to disclose relevant information about the functioning, capabilities, and limitations of the AI systems they deploy or provide.

The human oversight mechanism was first established under the GDPR, granting data subjects the right not to be subjected to solely automated decisions involving the processing of personal data that result in legal or similarly significant effects. In cases where such decisions are made, appropriate human supervision and intervention – often referred to as the ‘human-in-the-loop’ effect – are required to safeguard fundamental rights and freedoms (Fügener et al. 2021).

Building on this foundation, the EU AI Act states that high-risk AI systems must be designed to enable effective human oversight. This includes mechanisms that allow humans to understand, monitor, and control the operations of AI systems. The objective is to ensure that AI systems are not autonomous but operate under human governance, enabling necessary interventions and informed decisions (EP 2024, Article 14).

Human oversight also extends to ethical and legal considerations. AI systems must be developed and operated in ways that respect fundamental rights and comply with applicable laws (EP 2024, Article 3). Human overseers play a critical role in ensuring that AI systems do not perpetuate biases or make decisions that could lead to discrimination or other ethical issues.

AI systems are required to comply with the fundamental principles outlined in the EU AI Act. Among these principles, fairness is a cornerstone of trustworthy AI, ensuring that automated decision-making processes do not perpetuate bias or discrimination (EP 2024, Recital 27). However, despite its recognized importance, the EU AI Act does not provide explicit provisions on how fairness should be maintained. There is no clear obligation imposed on AI providers or deployers to assess, mitigate, or rectify model biases, leaving a regulatory gap.

While the EU AI Act does not have a single “Fairness” article, fairness principles are embedded in multiple provisions, particularly those related to bias mitigation (EP 2024, Article 10), human oversight (EP 2024, Article 14), transparency (EP 2024, Article 52), and fundamental rights assessments (EP 2024, Article 28).

The primary legal mechanism ensuring fairness is the requirement that high-risk AI systems must not result in discriminatory, biased, or unfair outcomes. Article 10 of the EU AI Act ensures fairness by requiring high-quality, bias-free, and representative training data for high-risk AI systems. The next chapter, we will explore the implications of XAI techniques on ensuring fairness in AI systems.

AI is flexible in the way that it transforms raw information (input data) into the model’s prediction (output data) by finding the best statistical fit to ensure that the model captures the patterns in the data. Drawback is that it is not always directly clear what the exact relationship is, and why it is how it is. Such application is seen as ‘black box’ to the developer and the user, not knowing what happens inside. Users in this context are defined as IT users within the deployer of the AI system. High-risk AI systems must be sufficiently transparent to enable deployers to interpret their output and provide information that is relevant to explain or interpret their output, as suggested by the EU AI Act (EP 2024, Article 13). As an additional control to use a system appropriately and to ensure accuracy of output, human oversight needs to be established as it provides critical judgement and validation of predictions.

An explainability layer on top of an AI system, referred to as XAI, helps a user performing human oversight in the explanation and interpretation of the output. This is only interesting when the system uses black-box models. This can also be the case when using third-party tooling with proprietary models. Under the EU AI Act, it is not sufficient to solely explain the overall functioning of the AI system (global), the output for a specific input needs to be also explained (local), as the right to complain requires a local explanation. The XAI explainability layer and techniques are not only important for ensuring fairness, but might also be interesting for organizations seeking to comply with both the transparency and human oversight requirements of the EU AI Act.

In addition to a technique giving global or local explanations, and the type of input data that the XAI technique can work with, there are other aspects that characterize specific XAI techniques, which should be considered when designing the aforementioned explainability layer:

• Some XAI techniques rely on an assumption of independence between features (explanatory variable), which leads to the omission of interactions between those features. This assumption can oversimplify real-world scenarios, potentially compromising the accuracy and relevance of the insights provided.
• The implementation difficulty of such techniques encompasses the complexity and time required to create an effective explanation layer. Techniques that require developers to make intricate implementation decisions or finely tune parameters are considered hard to implement. This is a key metric for validators or internal auditors, whose proficiency vary significantly, as programming is not their core skill set.
• Another critical aspect of an XAI technique is its clarity, i.e. user-friendliness (Gerlings et al. 2020; Y and Challa 2023). Techniques that produce multiple subfigures, use non-standard axis or present layered visuals are regarded as having low clarity.
• Closely related is the concept of information density, which covers how much information is conveyed. Techniques that only reveal a single dimension of insight or provide superficial details are considered to have low information density.
• Finally, computational complexity is the amount of resources required to run it (Y and Challa 2023). It illustrates the time it takes to run the XAI technique. Higher computational complexity results in longer processing times, which could impact the feasibility in environments with limited computational resources.

Panigutti et al. (2023) describe that the model can also be transparent by design, leading to interpretable AI (IAI). The decision-making process can then be directly assessed by internal auditors and validators because of the simple understandability of the inner workings of the model. Examples of IAI are relatively simple linear, tree-based, rule-based models, sparse models or models that process information in a way that is interpretable (Panigutti et al. 2023).

The explainability of an outcome cannot be looked at on a standalone basis. It needs to be assessed together with the performance of the model and stability of that performance. Low accuracy and/or stability need to be reflected in the explanation, to get a full understanding of the relationship and how strong it is.

In addition, the way of implementing explainability needs to be suitable to the level of expertise of the users performing oversight, and the users need to be sufficiently trained. In many cases, this may require additional representation tooling on top of the core XAI techniques. Based on a literature study, Haque et al. (2023) describe users’ explanation needs and the effect of explanation on users’ perceptions of an AI system. Dependent on the type of users, different representation formats may be used. The representation needs to be complete and sufficiently accurate for well-informed decision-making. The perception is influenced by the level of communicated versus observed accuracy, transparency of the working of the system, understandability due to sufficient involvement, added value of the explainability, and perceived fairness of outcomes with especially local explainability.

To illustrate the benefits and limitations of XAI, a specific banking application is developed in this section to estimate if a specific loan is likely to default or not, based on a set of features. The section afterwards demonstrates several XAI techniques given this credit risk model. Credit risk models are high-risk models under the EU AI Act, and as such the risk classification must be explainable due to the transparency requirements of the EU AI Act.

Ferreira (2018) selected some important features from a dataset published by Hofmann (1994). The variable to predict is the risk, where ‘good’ and ‘bad’ signify no default and default, respectively. The table of (Ferreira 2018) is shown in Table 1.

After encoding the ordinal (ordered variables: ‘Saving account’, ‘Checking account’) and nominal (unordered variables: ‘Sex’, ‘Housing’, ‘Purpose’) variables, the data is randomly split into 80% training and 20% test data. A Random Forest Classifier (introduced by Breiman (2001)) has been used to predict the risk variable, this could have also been any other classification model. This model results in an accuracy of 0.74. The results from the predictions are shown in Table 2, where on the columns the prediction of the model is placed and on the rows the ground-truth. For example, out of the 200 credit loans of which the risk level was predicted, the model classified 38 credit loans as having a ‘bad’ risk, while they were actually loans with a good risk profile.

Two of the most widely used XAI techniques are LIME and SHAP. LIME provides a local explanation and SHAP can give a local and a global explanation.

Local Interpretable Model-agnostic Explanations (LIME)

LIME uses a simple model to approximate a complex model (Ribeiro et al. 2016). It tries to explain the output of an individual observation by creating a sample with slightly randomly changed inputs and looking at the outputs. A simple interpretable linear model is used to fit this sample and explain the output. A LIME analysis for a specific instance (single loan application) of the credit risk example model is demonstrated in Figure 3. LIME states that this instance has an 82% probability of being a bad loan, seen on the left. In the middle graph it is shown that a low checking account and a low age are the main drivers for a bad loan, whereas the features with the blue coloring are drivers for a good loan. On the right, the value for each feature is shown.

Figure 3.

Local explanation from LIME.

SHapley Additive exPlanations (SHAP)

SHAP uses Shapley values that define the importance of an individual explanatory variable (feature), as the relative change in the output, with the specific feature included versus when it is excluded (Lundberg and Lee 2017). A SHAP plot is given for a specific instance, which is demonstrated in Figure 4 (left). A global SHAP explanation of the model is demonstrated in Figure 4 (right), where each dot is an instance of the data and the feature value coloring is the raw relative value of the feature.

Figure 4.

Local (left) and global (right) explanation from SHAP technique.

Table 3 presents the mentioned LIME and SHAP but also several other XAI techniques and their characteristics. These techniques are selected in their ability for assessing transparency, human oversight and fairness (Molnar 2019; Zhang et al. 2022). The characteristics (implementation difficulty, clarity, information density and computational complexity) are ranked relative to each other.

All these techniques are model agnostic, although there are variants that are model-specific. This means that XAI is quite flexible. Different techniques, or a combination of techniques may be used, dependent on the type of input data, sophistication of the XAI developer, sophistication of the user of XAI, and accuracy and consistency required.

While XAI offers significant benefits when tailored to stakeholders’ needs, it also comes with notable limitations. Therefore, just like with AI itself, XAI cannot be implemented as a tool that will automatically resolve all transparency, human oversight, and fairness issues. Expert involvement is essential in choosing how to apply XAI, what method(s) to use, how to interpret results, how to communicate these, and to opine on the AI system in areas where XAI was not applied. Most important limitations are:

• Judicial. XAI does inherently not guarantee AI decisions are legitimate, reliable, or unbiased.
• Inconsistency and irreproducibility. Different XAI methods can yield varying results for the same model. Some, like LIME and Anchors, introduce randomness, making explanations unstable.
• Automation bias. XAI can create a false sense of reliability, known as automation bias, leading to reduced human oversight and errors in decision-making going unnoticed (Schemmer et al. 2022). Underlined by the credit risk models, which might make significant errors if the operators rely too much on the decision making of the risk model without understanding the underlying model, showing the crucial importance of human judgement.
• Fairness concerns. As mentioned in the previous section, the EU AI Act emphasizes fairness as a key principle (EP 2024, Recital 27). While XAI can help reveal biases (McDermid et al. 2021; Hofeditz et al. 2022; Chuan et al. 2024), it does not inherently ensure fair decision-making. Analysis highlights that biases can persist through proxy variables; simply removing the ‘Sex’ feature from the credit risk example will not directly make the model fair. Ensuring fairness requires careful design, stakeholder involvement, and transparency behind XAI techniques (Longo et al. 2024). In this context, Dwork et al. (2012) introduced the concept of Fairness Through Awareness, which emphasizes that fairness should be understood and addressed relative to the specific context of the decision-making process.
• Selection of XAI techniques. The effectiveness of XAI depends on the chosen methods, which vary by stakeholder needs. A one-size-fits-all approach can cause misinterpretation. XAI should adapt to user needs and include educational tools. Developers may also manipulate representations to hide biases (Deck et al. 2023).

Internal auditors will play a critical role in evaluating AI systems in the context of meeting the transparency, human-oversight and fairness requirements of the EU AI Act. See also our previous article (Sandu et al. 2022) including a proposed framework for auditing algorithms in general, and the life cycle approach for continuous involvement of the internal audit department. As defined in Chapter 1, while the involvement of Internal Audit (IA) professionals in overseeing AI systems is not mandated by the EU AI Act, the IIA’s publication on the AI Act (IIA 2023) underscores their vital role in assessing AI risks, promoting transparency, and ensuring that governance frameworks align with regulatory expectations. The publication particularly highlights two critical contributions of the IA function:

1. Advisory Capacity – Internal auditors support management by providing guidance on how AI should be effectively managed, developed, and governed.
2. Assurance Function – Internal auditors independently assess whether AI-related controls and processes are properly designed, implemented, and functioning as intended.

As AI systems become more integral to organizational decision-making, Explainable AI (XAI) serves as a key mechanism for internal auditors to evaluate whether these systems comply with the EU AI Act. Specifically, XAI supports critical assessments of transparency, human oversight and fairness, which are central obligations under Articles 10, 13, and 14 of the EU AI Act (EP 2024) that were addressed previously in Chapter 2.

XAI enhances transparency by making AI decision-making processes understandable to human stakeholders. For internal auditors, this involves ensuring that the decision logic behind AI outputs can be clearly articulated, verifying whether input features, processing steps, and model decisions are documented and interpretable, assessing whether users can understand how outcomes are generated.

XAI also plays a role in evaluating fairness, particularly in ensuring that AI systems do not produce discriminatory outcomes. Internal auditors should apply XAI to detect bias patterns in both training data and model logic.

By enabling transparent inspection of model behavior, XAI also supports compliance with Article 10 on data quality and governance and helps uphold broader EU human rights and non-discrimination principles. According to Article 14, AI systems must allow for meaningful human oversight, enabling human intervention where ne­cessary. XAI contributes to this by providing actionable explanations that support human operators in overriding or correcting AI outputs.

Internal auditors can use these insights to evaluate whether oversight mechanisms are not just formally present but also functionally effective. It is thus considered good practice for an organization to implement policies, procedures, or guidelines outlining XAI requirements and their application to enable this.

Internal auditors evaluating an organization’s ability to leverage XAI to meet the transparency, human oversight and fairness requirements outlined in the EU AI Act will require a structured approach to ensure compliance.

The first step would be to understand the organization’s AI governance framework (see also our previous article, Sandu et al. (2022)). This involves examining policies, procedures, and controls established to ensure responsible AI practices. Internal auditors should evaluate whether these frameworks address transparency and oversight, including the existence of XAI principles. Attention should be paid to documented processes for risk assessment, decision-making accountability, and alignment with the EU AI Act.

The internal auditor should also assess the technical capabilities of the AI system. This includes determining if the system provides understandable and accurate explanations for its decisions or outputs (the transparency). It needs to be assessed to what extent the explainability or interpretability of the model meets the standards required for transparency under the EU AI Act. Internal auditors should also assess the technical documentation provided with the system.

As we have seen, an important aspect of compliance is ensuring that human oversight mechanisms are in place. Internal auditors should assess whether human reviewers have the necessary tools, authority, and expertise to oversee AI decisions. This includes checking for procedures, workflows and tollgates that allow humans to intervene or override decisions made by the AI system in case of errors or ethical concerns.

In addition to transparency and human oversight, internal auditors should assess the fairness by reviewing records regarding the functioning of AI system, particularly those employing XAI. This includes logs of system decisions, interventions, and updates to the model or data. These records, in essence, provide evidence of compliant operations over a period of time, very useful when performing any sort of Test of Effectiveness (ToE) on the AI system.