In the Cambridge Dictionary, an algorithm is defined as “a set of mathematical instructions or rules that, especially if given to a computer, will help to calculate an answer to a problem” (Cambridge Dictionary 2022). A simple and often used algorithm is sorting. Much more complex algorithms are used in robotics, where a mixture of physicality, linear algebra and statistics are used to map out actions and movement (Alvi 2022). Ismail (2018) describes AI algorithms as “a group of algorithms that can modify its algorithms and create new algorithms in response to learned inputs and data as opposed to relying solely on the inputs it was designed to recognize as triggers. This ability to change, adapt and grow based on new data, is described as intelligence”. These definitions give us insight into how difficult it is to control and audit algorithms.

Three main aspects make AI algorithms challenging to control: the input data used to train AI algorithms, the way the algorithm operates, and the autonomous learning performed by an algorithm. It is problematic if the data used to train the algorithm is unrepresentative for the group where the algorithm will be applied. If the algorithm is trained on predominantly European individuals but the intention is to use it on a diverse population, containing other countries, then the algorithm might not function properly as it was trained on the wrong data. Additionally, many AI algorithms are applied without a thorough understanding of how they work and if they can answer the problem for which they are used. It is inaccurate to use algorithms aimed at predicting quantitative variables, such as amount of sales, to predict qualitative variables, such as impairment or no impairment. Lastly, it is often not clear what the algorithm learns from the data, whether it learns to spot the “right” aspect of a problem. For example, a failure of AI was documented when, instead of learning to identify cancerous lesions, AI learned instead to identify images coming from a specific piece of equipment (Barnett et al. 2021). So, all imagines coming from a specific equipment were classified wrongly as “cancerous”. All three aspects can culminate in AI algorithms which lead to discrimination against protected groups of citizens, spread disinformation and even cause bodily harm.

Another challenge specific to AI algorithms is that they often use a large volume of structured (for example, tabular data) and unstructured data (for example, video data) which makes it difficult to ensure data integrity and representativeness. Additionally, many AI algorithms are self‑learning, continuously improving the algorithm, or adapting to changing circumstances. This introduces difficulties for the validation of the algorithm which changes from a point‑in‑time validation to a frequent or continuous (automated) monitoring situation. It also requires storing all algorithm changes and historical data used to train the algorithm. AI algorithms are also sensitive to the selection of hyperparameters. Usually simple algorithms, with less or no hyperparameters, can lead to underfitting (missing the underlying patterns in the data) while complex algorithms, with multiple hyperparameters that need tuning, tend to overfit (give a lower training error than the actual test error). In both cases this leads to poor performance of the algorithm in out‑of‑sample testing or production (Burnham and Anderson 2002). Finally, vendor algorithms tend to be proprietary, and this adds to the lack of transparency inherent in learning algorithms.

Considering the challenges introduced by algorithms, and especially AI algorithms, the question arises how an audit on algorithms should be executed and who should audit algorithms. A domain with experience in providing auditing services is the internal audit domain. The main purpose of internal audit is to add value and improve an organization’s operations and it does so by performing independent assessments on the effectiveness of governance, risk management, and control processes in an organization (IIA 2022). As organizations increasingly use algorithms (Makridakis 2017), they are confronted with significant challenges as algorithms can have an impact on the governance, risks and processes of an organization and, ultimately, on whether the organization achieves its goals or not. Then, the internal auditor becomes also responsible for auditing algorithms.

If auditing algorithms is expected from internal audit, then the internal auditor should have sufficient skills and experience to perform such an audit. This again makes the role of the internal auditor with respect to algorithms unclear. Various stakeholders still consider that internal auditors should work within the apparent current scope of their main activities and focus on operational and financial risks, on the governance and process rather than the correctness of the algorithms themselves (van Eck and Middelkoop 2020). Differently, in the article “Closing the AI Accountability Gap” Raji et al (2020) emphasize the added value of internal audit compared to external audit, because of the internal audit’s direct access to internal systems. This privileged access to internal systems such as the algorithms used in an organization, creates the obligation for the internal auditor to have knowledge about algorithms themselves. Skills and experience in algorithms and in auditing algorithms can be acquired by internal auditors by expanding their current knowledge as suggested by Arnold (2021), similar to how they acquired knowledge in performing other tasks such as controlling the risks of models.

According to Carawan et al. (2018) internal auditors now have an important role in the Model Risk Management domain, as internal auditors can be tasked with assessing the effectiveness of the Model Risk Management Framework used by financial institutions, including the governance, policies, procedures, and activities conducted to address the risk of model error. Internal auditors are also responsible for understanding when and how the model is used and if it is in line with the model’s stated purpose. The currently undocumented practice into model risk management also reveals the fact that internal auditors manage to perform a rigorous audit of the model risks only if they have knowledge on the model, as well. One indication of this can be the trend of including professionals with model knowledge (e.g., statisticians) in audit teams. Similar to the Model Risk Management domain, where the prevalent use of models by financial institutions pushed the internal auditor into assessing the risks of models, the internal auditor working for organizations who use algorithms will have to audit AI algorithms as well. For this, the internal auditor needs a framework as a guide to auditing algorithms.

Kazim and Koshiyama (2020) distinguish between three phases of AI ethics maturity: the principles and guidelines phase, the processes or the ethics‑by‑design phase, and the AI assurance phase. The first phase involves AI ethics principles and guidelines issued by diverse organizations, ranging from NGOs and governmental bodies to private entities (for example, the Ethics Guidelines for Trustworthy AI issued by EU’s High‑Level Expert Group on Artificial Intelligence). Despite serving as an important step in the process of AI ethics maturation, merely spelling out AI ethical principles such as transparency, non‑maleficence and responsibility mostly fails to converge into implementable instruments (Jobin et al. 2019).

The second phase involves the inclusion of AI ethics principles in professional codes of ethics such that AI has ethics‑by‑design. For example, the Association for Computing Machinery (ACM) revised in 2018 its 1992 Code of Ethics to include ethics principles for software developers. Such codes of ethics should guide AI developers to the right choices when faced with ethical dilemmas (for example, how much and what user data to collect). But placing the responsibility of developing ethical AI on the sole shoulders of AI developers can turn out to be ineffective when codes of ethics do not actually influence the choices of AI developers (McNamara et al. 2018). The irrefutable conclusion of these approaches was that the responsibility to develop ethical AI was an interdisciplinary problem (Vanhée and Borit 2022) that should be shared by engineers, philosophers, psychologists and many more. Nevertheless, putting an interdisciplinary stamp does not alleviate the elusiveness of getting AI in control in a way that fulfills organizational goals without causing societal or business harm.

The third and current phase of AI ethical approaches concerns the standardization and operationalization of AI ethics. In this stage of AI maturity, the focus is placed on risk‑assessments of AI systems. In a risk‑based approach to auditing AI, teams of people close to the problem, highly familiar to the environment where the AI system operates, would sit together with the auditors, and generate a risk registry with everything that can go wrong with the algorithm. But in the case of AI, where data combines with algorithms and user behavior in unknown ways, teams of AI developers and auditors might not be able to anticipate important risks. In this case, the diversity of the team, not only in terms of expertise but also in terms of other aspects such as race, gender, education, cultural background, might be key to implementing a good risk registry. For example, while a team of Dutch developers of a globally deployed image captioning AI system might not identify the poor performance of AI as a risk, when tasked to identify celebrities from Taiwan, an international AI review team might.

SMACTR (Scoping, Mapping, Artefact Collection, Testing and Reflection), the internal audit framework proposed by Raji et al. (2020), aims to give an opinion on the compliance of AI systems with the ethical values of the organization. Most importantly, every stage in SMACTR leads to the creation of artefacts, or documentation, useful for the conclusion of the audit. The documentation design is borrowed from other fields such as aerospace (e.g., the Failure Modes and Effects Analysis systematic risk management approach used in safety engineering) but it also mentions specific documentation aimed at AI (e.g., datasheets and model cards).

Guided by commonly recognized ethical principles (Jobin et al. 2019) such as transparency, justice, fairness & non-discrimination, safety & non-maleficence, responsibility & accountability, SMACTR starts off with a Scoping stage where potential risks to ethical principles are identified and analyzed for their potential impact. In one hypothetical example in Raji et al. (2020), a smile detection algorithm used to automatically trigger cameras in physical photo booths can lead to threats to the principle of fairness & non-discrimination if the algorithm disproportionately impacts people with disabilities or different cultural norms on the formality of smiling. SMACTR continues with a Mapping of stakeholders stage and an Artefact Collection stage where the documentation needed to understand the algorithm is collected. Once risks are identified and ranked, testing can be done to check the compliance of the AI systems with the ethical guidelines of the company that are most at risk at the time of the audit. Testing includes mostly technical solutions such as adversarial testing where auditors simulate what an enemy might do to confuse a system, or review of metrics for specific user profiles. The testing stage of SMACTR should confirm the existence of the anticipated risks. SMACTR ends with a Reflection stage that includes a risk appetite analysis and a decision on the action plans recommended to mitigate the most important risks. At this point, depending on the risk thresholds of the company, the risks that are considered acceptable by the company are proposed to be mitigated with specific action plans. In the case of the smile detection algorithm, a risk mitigation action could be to add a button for users to opt-in to the algorithmic solution.

A more encompassing framework which tries to operationalize the audit of algorithms, is the framework published in 2021 by an extensive team of researchers gathered from different universities and institutes, including the University College London and the London Stock Exchange (Koshiyama et al. 2021). In the context of this recent publication, the audit of an algorithm is defined as “the research and practice of assessing, mitigating, and assuring an algorithm’s safety, legality, and ethics” (Koshiyama et al. 2021, p. 2). According to this framework, there are four dimensions in the audit of an algorithm: Development, Assessment, Mitigation, and Assurance (DAMA, henceforth). While the Development dimension refers to the stages of an algorithm including data setup, feature engineering, model selection, post‑processing, and deployment, the Assessment dimension comprises different pressures that act upon algorithms such as fairness, robustness, explainability and privacy concerns (or verticals, as they are called in the paper of Koshiyama et al. (2021)). These two dimensions interact to give rise to specific auditing activities. For example, the data setup stage interacts with the fairness vertical to inspire the focus of the audit on how balanced the data is. But the situation gets even more complicated as there is an additional interaction. The Development and Assessment dimension interact with a third element, the level of access of the algorithm. Consequently, the audit or inspection activities are much more limited when the level of access to the algorithm is restricted to only an indirect observation of the system. Mitigation strategies aim to address the issues discovered in auditing activities. For example, if explainability issues were identified, mitigation strategies like the application of model agnostic local explanations (for example, LIME approach) can be suggested. The Assurance stage contains possible certification opportunities that narrow the scope of the assurance service, such as certification of a system or sector specific certifications.

The DAMA framework presents an interesting view on how a low number of dimensions interacting can lead to numerous auditing activities and that these activities and the subsequent certification of the algorithmic system are impacted by the level of access to the algorithm. It is also valuable to see the focus of the Assessment dimension on four main verticals inspired by the most recent development in Fair AI: fairness, robustness, explainability and privacy. Still, the operationalization of the DAMA framework is not included in the paper and so the concrete steps to take or items to check in an audit are unclear.

On September 22, 2020, the Netherlands Court of Audit (NCA) organized a thinking session with more than 30 experts on 5 themes related to AI algorithm use: data driven working, data quality, AI and algorithms, AI at the government, and transparency. One important conclusion of the thinking session was that due to the way algorithms are developed (for example, using historical data which might be biased or inserting bias through development choices), we can never have certainty that algorithms do not discriminate. As such, necessary controls need to be placed on algorithms. In its 2021 publication ‘Aandacht voor Algoritmes’ (Algemene Rekenkamer 2021), the Netherlands Court of Audit proposes a risk assessment framework aimed to control algorithms. The framework is based on a combination of existing frameworks and is targeted at five areas: governance, model and data, privacy, IT general controls, and ethics. To showcase the usefulness of the framework, the Netherlands Court of Audit tested it on three algorithms and concluded that the current use of algorithms within Dutch municipalities is limited and the algorithms used can be controlled. Although not limited to, the main intended users of the framework developed by the Netherlands Court of Audit are governmental organizations.

In 2021 NOREA, the Dutch Association of chartered IT‑auditors, released the Guiding Principles for Trustworthy AI Investigations (De Boer and Van Geijn 2021). The NOREA principles are aimed at guiding Dutch chartered IT‑auditors in investigations into AI algorithms. The guidelines are developed using leading practices for trustworthy AI (for example, General Data Protection Regulation, Information Commissioner’s Office Guidance on the AI Auditing Framework). Nevertheless, the NOREA principles do not intend to present a comprehensive framework for algorithmic system scrutiny and the advice presented in the guidelines is that, for algorithm control frameworks, existing risk management methodologies such as COSO and COBIT are encouraged to be followed. It is not clear though if such methodologies are a good match with the challenges introduced by AI algorithms.

The DAMA, SMACTR, the NCA and NOREA conceptual frameworks highlighted in this section are aimed at the ethical risks of AI algorithms. In this article we express an opinion that there is a need for a more comprehensive framework, which goes beyond ethical risks to consider other business risks of AI algorithms and of algorithms, in general. We further attempt to step beyond purely conceptual frameworks to design a framework which can be more readily implemented by an auditor in an audit of an algorithm. As such, we develop normative statements (or positions) against which an algorithm can be checked. These normative statements, which describe how things should work and are audit objectives, are aimed to resonate with auditors who use audit objectives in other audit tasks. For example, in an environmental audit, an audit objective (or normative statement) is that “Environmental policies exist during the reporting period as described in the notes to the environmental data” (Kamp 2002). With a set of normative statements aimed at AI spelled out, we believe that an auditor can perform a rigorous and comprehensive audit of an AI algorithm.

All previous work on the audit of algorithms highlighted in the previous section focuses mainly on the ethical risks introduced by AI algorithms, for example, the risk of discriminating against protected groups of people. The upcoming AI Act also addresses ethical issues around AI. Although they are very important risks, it is salient to regard ethical risks in the bigger picture of the overall risk that the use of AI algorithms carries: the risk that the algorithm produces erroneous results or is used in the wrong way. One area which can serve as a starting point for getting a more comprehensive picture of risks of algorithms, beyond ethical risks, is the area of model risk management. Conceptually, AI algorithms and models are related in that AI “algorithms operate by learning models from existing data and generalizing them to unseen data” (Suresh and Guttag 2021, p. 1). As algorithms are fundamentally the engines of a model, we can borrow from Model Risk Management best practices to identify other risks in using algorithms, besides ethical risks.

The risk management around algorithms can be approached similarly to the Model Risk Management approach, with a “three lines” system (IIA 2020). The first line (for example, the management of the organization) develops and uses an algorithm, being primarily responsible for all relevant risks. The second line (for example, a special function within an organization responsible for risk management, compliance, or internal control) assists in risk management and often also advises management on accepting or rejecting a specific algorithm and under what conditions. As a third line, internal audit gives an independent and objective assessment to a governing body (for example, the management of the organization) regarding the sufficiency and effectiveness of governance and risk management over AI algorithms.

Inspired from the Model Risk Management best practice (Garla and Dhillon 2016), the Life cycle framework for auditing AI algorithms (figure 1) recognizes seven phases during which the first line must manage the risks introduced by AI algorithms: Initiation, Development, Implementation, Use, Monitoring, Review and Retirement. Three of those phases take place before the algorithm is put to use (the Initiation, Development and Implementation phases) while three phases are distinguished during the use of the algorithm (Use, Monitoring and Review). During the last phase, the model is retired (Retirement). The second line in the accountability structure over AI algorithms (for example, a special function within an organization responsible for risk management, compliance, or internal control) is responsible for the Validation of the AI algorithm(s) and plays an important role during most of the phases of the algorithm’s lifecycle. Importantly, the use of the algorithm needs to be approved by a governing body, advised by the second line’s validation function. While the life cycle stages are common to the Model Risk Management approach, the contribution of this article refers to specific aspects in each stage, aimed particularly at AI algorithms.

Figure 1.

The Life cycle framework for the AI algorithm audit.

To offer more granularity to the Life cycle framework for AI algorithm audit, we present in Appendix 1: Table A1 an overview of risks per Life cycle phase (including the Validation phase) and normative statements (or positions) against which an algorithm can be checked. While some normative statements in Table A1 are common to the Model Risk Management framework, the highlighted aspects (in blue italic font) are aimed specifically at AI algorithms (Dil et al. 2019). One current limitation of the framework is that it does not spell out the evidence gathering techniques applicable to AI algorithms for every normative statement. It does not do so because the choice of technique for evidence gathering is a creative one and evidence gathering techniques for AI algorithms can range from highly technical approaches (for example, technical toolkits aimed at understanding the workings of an algorithm such as the LIME approach (Ribeiro et. al. 2016)) to less technical or more traditional approaches (for example, interviews, checking documentation). Nevertheless, the NOREA Guiding Principles for Trustworthy AI Investigations (2021) offers valuable guidance on types of evidence to look for when engaged in investigating the trustworthiness of AI.

Returning to Table A1, the Initiation phase of the Life cycle refers mainly to stakeholders (for example, involving the right stakeholders, making roles and responsibilities clear) and the scope of the algorithm (for example, delineating an area of investigation depending on the purpose and the regulations covering the algorithm). The Development phase is focused on how sound the process of developing the algorithm is (for example, “are the chosen algorithms applied correctly?”) and how sound the data used is (for example, “is the data valid?”). The Implementation phase aims to check if the implementation of the algorithms is in line with its design (for example, “when implemented, are deviations from the developed algorithms documented?”). The audit activities in the Use phase concentrate on making sure that the algorithm is used in line with its intended purpose and that the users know how to use the algorithm and can provide feedback on its use. The Monitoring phase delineates what to measure with respect to the algorithm (for example, what performance metrics to monitor) during the life of an algorithm and when to signal an issue (for example, setting up acceptable thresholds for performance metrics). The Review phase follows up on the discoveries made in the Monitoring phase, looks at the continued functioning of the entire algorithm, and aims to assess if necessary improvements are made to the algorithms (for example, reparameterization is performed). In the Retirement phase, the organization’s inventory of algorithms is updated such that algorithms no longer in use are not still available for use or misuse. The Validation of the algorithm aims to timely challenge the development, implementation, use, monitoring, and review of the algorithm (for example, check if all relevant risks are considered). Although it is difficult to claim that a framework is comprehensive enough to support an AI algorithm audit, it is possible that, by focusing on the processes around an AI algorithm, on its life cycle within the organization, such an audit can be performed. Thus, we envisage that this framework can be used for the audit of AI algorithms.

To illustrate the application of the Life cycle audit framework, the hypothetical example of an algorithm supporting an online hotel rating system is used. Such a rating system can be attached to the website of an online travel company who lists hotels, and their ratings, based on the search terms of the customers (for example, location, period, amenities). The rating of hotels can, and does, play a role in the decision of customers to reserve a hotel room (Eslami et al. 2017). The hypothetical example considers that most of the normative positions in the framework are met, with three exceptions where risks are identified

One risk of the rating algorithm is that its purpose is not clearly described and shared with the users (normative statement 2.1 is not met), leading to users being confused by the rating system and eventually not using the rating provided. As an example, users of the rating system might be under the impression that only user reviews matter for the rating, while the rating system has the overall purpose of “rating hotels based on user input and our own domain knowledge”. The introduction of domain knowledge in the algorithm can lead to a different rating (for example, a higher rating) than the one obtained from using only user input (for example, averaging user ratings). The auditor of such an algorithm can check the website of the online travel company to verify if the purpose of the rating algorithm is clearly described.

Another risk of the rating algorithm is that its performance is not checked per sub‑group (normative statement 4.6 is not met). If the algorithm has a lower performance for the low‑to‑medium quality hotels sub‑group, then users can lose trust in the rating and stop using it. In this situation, the algorithm developer can compare the rating of hotels which are rated on different rating platforms (benchmarking). If the results show a significant difference between the ratings for different hotel categories (for example, low‑to‑medium quality) on different platforms (for example, Booking versus Expedia), then this is an indication that the algorithm might not perform as expected per sub‑groups. The auditor might check if the developer has done the benchmarking and if the validator has challenged the benchmarking.

Rating algorithms which use external user input suffer from a risk of invalid data (normative statement 5.1 is not met). Fake user reviews used in the rating algorithm can materialize into incorrect ratings which lead to a public mistrust in the rating system. To get protection against fake user reviews, the algorithm developer can prompt users to provide a valid email address when making an account on the rating platform or can use other algorithms to check for the existence of fake user accounts. Here too, the auditor would verify the work done by the developer and whether the validation function sufficiently challenged the developer.