ALEX Institute / AI / AI Ethics · Module 5

Fairness, bias, accountability and governance

Track C · AI Ethics, module 5. Turing Commons · about 1.5 h reading (18,147 words).

OPEN Source and license. “AI Ethics and Governance” by The Alan Turing Institute, Turing Commons (Christopher Burr and contributors). Source: Turing Commons, AI Ethics and Governance skills track. License: CC BY 4.0. Turing Commons states its material is available under CC BY 4.0 (README and the track's welcome page). Changes: images and interactive elements removed; headings and links adapted for this page.

← Track C · AI Ethics: all modules

Fairness & Bias Mitigation, Transparency, Explainability, and Governance

Chapter Outline

Chapter Summary

In this chapter we explore various issues that arise within AI systems. We will look at the elements of AI fairness, and the strategies to conduct bias mitigation. Finally, we will touch upon the concepts of accountability and governance, and how they can be integrated into the AI project lifecycle.

Learning Objectives

In this chapter, you will:

  • Learn about the landscape of meanings the concept of fairness has, as well as how to apply it in the context of AI.
  • Understand the different elements of AI fairness and how they relate to one another.
  • Look at the concept of bias in AI systems, the different kinds of biases which may arise, as well as how to mitigate them.
  • Familiarise yourself with AI accountability, its main components, and how it relates to AI governance.

Introduction to the principle of fairness

In this section, we will explore the complicated landscape of AI fairness definition as a preliminary step towards understanding the ways in which existing biases manifest in the design, development, and deployment of AI systems. Understanding how concepts of fairness are used in the field of AI ethics and governance is a crucial prerequisite to understanding where and how unfair biases arise across the AI project lifecycle, because relevant notions of fairness operate both as ethical and legal criteria based upon which biases can be identified across the AI project workflow and as normative yardsticks against which they can be measured and then appropriately mitigated.

When thinking about AI fairness, it is important to keep in mind that these technologies, no matter how neutral they may seem, are designed and produced by human beings, who are bound by the limitations of their own given contexts and by biases that can arise both in their cognitive processes and in the social environments that influence their actions and interactions. Pre-existing or historical configurations of discrimination and social injustice—as well as the prejudices and biased attitudes that are shaped by such configurations—can be drawn into the AI innovation lifecycle and create unfair biases at any point in the project workflow. This is the case from the earliest stages of agenda setting, problem selection, project planning, problem formulation, and data extraction to later phases of model development and system deployment. Additionally, the datasets used to train, test, and validate AI/ML models can encode socially and historically crystallised forms of inequity and discrimination, thereby embedding biases in an algorithmic model’s variables, inferences, and architecture.

This wide range of entry points for bias and discrimination across the AI/ML innovation lifecycle has complicated the notion of fairness, since the inception of fairness-centred approaches like ‘discrimination-aware data mining’ and ‘fair machine learning’ more than a decade ago.---.

To be sure, possibilities for reaching consensus on a commonly accepted definition for AI/ML fairness and on how to put such a definition into practice have been hampered by the myriad technical and sociotechnical contexts in which fairness issues arise. Such prospects for consensus have also been challenged by the broad spectrum of views in society on what the concept of fairness means and how it should best be operationalised. For this reason, in this practical guidance, we take a context-based and society-centred approach to understanding AI/ML fairness that is anchored in two pillars.

First, to understand how concepts of fairness are defined and applied in AI innovation contexts, we must begin by acknowledging that there is a plurality of views in the social world on the meaning of fairness—myriad interpretations of its sense and significance within and across cultures, societies, and legal systems. For instance, the meaning of the term ‘fairness’, in its contemporary English language usage, has dozens of interpretations which include a range of related but distinctive ideas such as equity, consistency, non-discrimination, impartiality, justice, equality, honesty, and reasonableness.-

(Audard, 2014; Carr, 2017)

Likewise, the translation of the word ‘fairness’ into other languages has proven to be notoriously difficult, with some researchers claiming that it cannot be consistently understood across different linguistic groups.- It is clear, from this vantage point, that fairness (and adjacent notions like equity, impartiality, justice, equality, and non-discrimination) must be approached with an appropriately nuanced responsiveness both to the many ways in which these concepts can be interpreted and to the many contexts in which they can be applied.

Second, despite this pluralism in the understanding and application of the concept of fairness, there has been a considerable convergence around how the interrelated priorities of non-discrimination and equality constitute the justificatory nucleus of fairness concerns. Though some claim that fairness is ultimately a subjective value that varies according to individual preferences and cultural outlooks, general ethical and legal concepts of fairness are predicated on core beliefs in the equal moral status of all human beings and in the corollary right of all human beings to equal respect, concern, protection, and regard before the law.

On this view, it is because each person possesses an intrinsic and equal moral worth that everyone deserves equal respect and concern—respect and concern that is grounded in the common dignity and humanity of every person.--

Carr 2000

The normative core of fairness, in this respect, has to do with the moral duty to treat others as moral equals and to secure the membership of all in a ‘moral community’ where every person can regard themselves as having equal value. Wrongful discrimination, along these lines, occurs when decisions, actions, institutional dynamics, or social structures do not respect the equal moral standing of individual persons.--

Convergence around this centrality of equality and non-discrimination as an indispensable cornerstone of fairness concerns has also led to their widespread acceptance as normative anchors of both international human rights law and anti-discrimination and equality statutes. In human rights law, interlocking principles of equality and non-discrimination are taken to be essential preconditions for the realisation of all human rights insofar as equality and non-discrimination are implied in the guarantee of the equal enjoyment and protection of fundamental rights and freedoms to all human beings per se. For this reason, principles of equality and non-discrimination are treated as jus cogens in human rights law—i.e., they are treated as peremptory or foundational norms that permeate all human rights provisions and from which no derogation is permitted in any case.-

check clifford 2013

In anti-discrimination and equality statutes in the UK and beyond, dovetailing priorities of equality and non-discrimination likewise form principal aims and essential underpinnings of fairness concerns. In this case, equality before the law manifests as equal protection from discriminatory harassment and from both direct and indirect kinds of discrimination (UK Equality Act, 2010). In discriminatory harassment, unwanted or abusive behaviour linked to a protected characteristic violates someone’s dignity, degrades their identity, or creates an offensive environment for them. For example, an employer who makes a racist remark about a protected group in the presence of an employee from that racial background would be considered to have harassed that employee based on the protected characteristic of race.

What are protected characteristics?

In the 2010 UK Equality Act, protected classes include age, gender reassignment, being married or in a civil partnership, being pregnant or on maternity leave, disability, race including colour, nationality, ethnic or national origin, religion or belief, sex, and sexual orientation. The European Convention on Human Rights, which forms the basis of the UK’s 1998 Human Rights Act, includes as protected characteristics ‘sex, race, colour, language, religion, political or other opinion, national or social origin, association with a national minority, property, birth or other status'.

Direct discrimination occurs when individuals are treated adversely based on their membership in some protected class. This type of discrimination is also known as ‘disparate treatment’ because it involves instances where otherwise similarly positioned individuals receive different and more-or-less favourable treatment on the basis of differences between their respective protected characteristics. For instance, direct discrimination would occur if an otherwise well-qualified job applicant were intentionally denied an opportunity for employment because of their age, disability, or sexual orientation.

By contrast, indirect discrimination occurs when existing provisions, criteria, policies, arrangements, or practices—which could appear on their face to be neutral—disparately harm or unfairly disadvantage members of some protected class in comparison with others who are not members of that group. This type of discrimination is also known as ‘disparate impact’ because what matters here is not directly unfavourable treatment in individual cases but rather the broader disproportionate adverse effects of provisions, criteria, policies, arrangements, or practices that may subtly or implicitly disfavour members of some protected group while appearing to treat everyone equally. Indirect discrimination can therefore involve the impacts of tacitly unjust or unfair social structures, underlying inequalities, or systemic patterns of implicit historical bias that manifest unintentionally through prevailing norms, rules, policies, and behaviours. For instance, indirect discrimination would occur if a job advertisement specified that applicants needed to be native English speakers, for this would automatically disadvantage candidates of different nationalities regardless of their levels of fluency or language training.

These three facets of anti-discrimination and equality law (harassment, direct discrimination, and indirect discrimination) have significantly shaped contemporary approaches to AI fairness.----

Liu and Lamb, 2018; Ntoutsi et al., 2019; Pessach and Shmueli, 2020; Wachter et al., 2021).

Indeed, attempts to put the principle of AI fairness into practice have largely converged around the priority to do no discriminatory harm to affected people along each of these three vectors of potential injury. It is thus helpful to think about basic AI fairness considerations as tracking three corresponding questions (An example of an associated discriminatory harm is provided alongside each.):

  1. How could the use of the AI system we are planning to build or acquire—or the policies, decisions, and processes behind its design, development, and deployment—lead to the discriminatory harassment of impacted individuals (i.e., unwanted or abusive treatment of them which is linked to a protected characteristic and which violates their dignity, degrades their identity, or creates a humiliating or offensive environment for them)?

    Example

    An AI-enabled customer support chat bot is built from a large language model that has been pre-trained on billions of data points scraped from the internet and then customised to provide tailored responses to customer questions about the provision of a public service. After the system goes live, it is soon discovered that, when certain customer names (which are indicative of protected classes) are entered, the chatbot generates racist and sexist text responses to customer inquiries.

  2. How could the use of the AI system we are planning to build or acquire—or the policies, decisions, and processes behind its design, development, and deployment—lead to the disproportionate adverse treatment of impacted individuals from protected groups on the basis of their protected characteristics?

    Example

    An AI system used to filter job applications in a recruitment process is trained on historic data that contains details about the characteristics of successful candidates over the past several years. Because white male applicants were predominantly hired over this time, the system learns to infer the likelihood of success based on proxy features connected to the protected characteristics of race and sex. It consequently filters out non-white and non-male job candidates from the applicant pool.

  3. How could the use of the AI system we are planning to build or acquire—or the policies, choices, and processes behind its design, development, and deployment—lead to indirect discrimination against impacted individuals from protected groups?

    Example

    An AI-enabled medical diagnosis tool is built as a smartphone application and made available to all participants in a national health system without sufficient considerations of the barriers to access faced by some citizens. It becomes clear after the app is launched that the device disproportionately favours younger, more digitally literate, and more affluent community members, while disadvantaging both the elderly, less digitally literate population and digitally deprived people who do not have access to smartphone technologies and internet connections.

It is important to note, regarding this final question on indirect discrimination, that the Public Sector Equality Duty mandates considerations both of how to ‘reduce the inequalities of outcome which result from socio-economic disadvantage’ and of how to advance equality of opportunity and other substantive forms of equality. This means that our approach to putting the principle of AI fairness into practice must include social justice considerations that concentrate on how the production and use of AI technologies can address and rectify structural inequalities and institutionalised patterns of inequity and discrimination rather than reinforce or exacerbate them. We must consequently take a multi-pronged approach to AI fairness that integrates formal approaches to non-discrimination and equality (which focus primarily on consistent and impartial application of rules and equal treatment before the law) with more demanding substantive and transformative approaches(which focus on equalizing the distribution of opportunities and outcomes and on the fundamental importance of addressing the material pre-conditions and structural changes needed for the universal realisation of equitable social arrangements).

What is social justice?

Social justice is a commitment to the achievement of a society that is equitable, fair, and capable of confronting the root causes of injustice. In an equitable and fair society, all individuals are recognised as worthy of equal moral standing and are able to realise the full assemblage of fundamental rights, opportunities, and positions.

In a socially just world, every person has access to the material means needed to participate fully in work life, social life, and creative life through the provision of proper education, adequate living and working conditions, general safety, social security, and other means of realising maximal health and well-being.

Social justice also entails the advancement of diversity and participatory parity and a pluralistically informed recognition of identity and cultural difference. Struggles for social justice typically include accounting for historical and structural injustice coupled to demands for reparations and other means of restoring rights, opportunities, and resources to those who have been denied them or otherwise harmed.]

Discriminatory non-harm

While there are different ways to characterise or define fairness in the design and use of AI systems, you should consider the principle of discriminatory non-harm as a minimum required threshold of fairness. This principle directs us to do no harm to others through direct or indirect discrimination or through discriminatory harassment linked to a protected characteristic that violates the dignity of impacted individuals, degrades their identity, or creates a humiliating or offensive environment for them:

Key Concept: Principle of Discriminatory Non-Harm (Do No Discriminatory Harm)

The producers and users of AI systems should prioritise the identification and mitigation of biases and discriminatory influences, which could lead to direct or indirect discrimination or discriminatory harassment. This entails an end-to-end focus on how unfair biases and discriminatory influences could arise (1) in the processes behind the design, development, and deployment of these systems, (2) in the outcomes produced by their implementation, and (3) in the wider economic, legal, cultural, and political structures or institutions in which the AI project lifecycle is embedded—and in the policies, norms, and procedures through which these structures and institution influence actions and decisions throughout the broader AI innovation ecosystem. Developers and users of AI systems should, in this respect, acknowledge and address discriminatory patterns that may originate in the data used to train, test, and validate the system and in the model architectures (i.e., the variables, parameters, inferences, etc.) that generate system outputs.

Beyond this, the principle of discriminatory non-harm implies that producers and users of AI systems should ensure that their research, innovation, and implementation practices are undertaken in an optimally responsible and ethical manner, more broadly, in keeping with the historical tendency that deficiencies in the deployment and operation of faulty systems often disparately impact protected, under-represented, or disadvantaged groups.

The principle of discriminatory non-harm applies to any AI system that processes social or demographic data (i.e., data pertaining to features of human subjects, population- and group-level traits and characteristics, or patterns of human activity and behaviour). However, the principle applies equally to AI systems that process bio-physical or biomedical data. In this case, imbalanced datasets, selection biases, or measurement errors could have discriminatory effects on impacted individuals and communities—for instance, where a demographic group’s lack of representation in a biomedical dataset (e.g., one used to train a diagnostic prediction model) means that the trained system performs poorly for that group relative to others that are better represented in the data.

Prioritising discriminatory non-harm implies that the producers and users of AI systems ensure that the decisions and behaviours of their models neither treat impacted individuals adversely based on their membership in some protected class or socioeconomic group nor, intentionally or unintentionally, generate discriminatory or inequitable impacts on affected individuals and communities that unfairly disadvantage members of some protected class or socioeconomic group in comparison with others who are not members of that class or group. It can also be seen as a proportional approach to bias mitigation because it sets a baseline for fair AI systems, while, nevertheless, creating conditions for developers and users to strive towards an ideal for fair and equitable outcomes for all people as moral equals and as members of a just community where every person can regard themselves as having intrinsic dignity and equal moral standing.

Finally, the scope of the principle means that, beyond designers and users, any individuals, organisations, or departments who are procuring AI systems must ensure that the vendors of such systems can demonstrate the mitigation of potential biases and discriminatory influences in the processes behind their production and in their outputs.


AI Fairness as a contextual and multi-valent concept

AI Fairness must be treated as a contextual and multi-valent concept that manifests in a variety of ways and in a variety of social, technical, and sociotechnical environments. It must, accordingly, be understood in and differentiated by the specific settings in which it arises. This means that our operating notion of AI fairness should distinguish between the kinds of fairness concerns that surface in

  1. the context of the social world that precedes and informs approaches to fairness in AI/ML innovation activities—i.e., in general normative and ethical notions of fairness, equity, and social justice, in human rights laws related to equality, non-discrimination, and inclusion, and in anti-discrimination laws and equality statutes;
  2. the data context—i.e. in criteria of fairness and equity that are applied to responsibly collected and maintained datasets;
  3. the design, development, and deployment context—i.e. in criteria of fairness and equity that are applied (a) in setting the research agendas and policy objectives that guide decisions made about where, when, and how to use AI systems and (b) in actual model design and development and system implementation environments as well as in the technical instrumentalisation of formal fairness metrics that allocate error rates and the distribution of outcomes through the retooling of model architectures and parameters; and
  4. the ecosystem context—i.e. in criteria of fairness and equity that are applied to the wider economic, legal, cultural, and political structures or institutions in which the AI project lifecycle is embedded—and to the policies, norms, and procedures through which these structures and institution influence actions and decisions throughout the AI innovation ecosystem.

Each of these contexts will generate different sets of fairness concerns. In applying the principle of discriminatory non-harm to the AI project lifecycle we will accordingly break down the principle of fairness into six subcategories that correspond to their relevant practical contexts:

Data Fairness: The AI system is trained and tested on properly representative, fit-for-purpose, relevant, accurately measured, and generalisable datasets.

Application Fairness: The policy objectives and agenda-setting priorities that steer the design, development, and deployment of an AI system and the decisions made about where, when, and how to use it do not create or exacerbate inequity, structural discrimination, or systemic injustices and are acceptable to and line up with the aims, expectations, and sense of justice possessed by impacted people.

Model Design and Development Fairness: The AI system has a model architecture that does not include target variables, features, processes, or analytical structures (correlations, interactions, and inferences) which are discriminatory, unreasonable, morally objectionable, or unjustifiable or that encode social and historical patterns of discrimination.

Metric-Based Fairness: Lawful, clearly defined, and justifiable formal metrics of fairness have been operationalised in the AI system have been made transparently accessible to relevant stakeholders and impacted people.

System Implementation Fairness: The AI system is deployed by users sufficiently trained to implement it with an appropriate understanding of its limitations and strengths and in a bias-aware manner that gives due regard to the unique circumstances of affected individuals.

Ecosystem Fairness: The wider economic, legal, cultural, and political structures or institutions in which the AI project lifecycle is embedded—and the policies, norms, and procedures through which these structures and institution influence actions and decisions throughout the AI innovation ecosystem —do not steer AI research and innovation agendas in ways that entrench or amplify asymmetrical and discriminatory power dynamics or that generate inequitable outcomes for protected, marginalised, vulnerable, or disadvantaged social groups.

Data fairness

Responsible data acquisition, handling, and management is a necessary component of algorithmic fairness. If the results of an AI project are generated by a model that has been trained and tested on biased, compromised, or skewed datasets, affected stakeholders will not be adequately protected from discriminatory harm. Data fairness therefore involves the following key elements:

  • Representativeness Depending on the context, either under representation or over representation of demographic, marginalised or legally protected groups in the data sample may lead to the systematic disadvantaging of vulnerable or marginalised stakeholders in the outcomes of the trained model. To avoid such kinds of sampling and selection biases, domain expertise is crucial, for it enables the assessment of the fit between the data collected or procured and the underlying population to be modelled.

  • Fit-for-Purpose and Sufficiency An important question to consider in the data collection and procurement process is: Will the amount of data collected be sufficient for the intended purpose of the project? The quantity of data collected or procured has a significant impact on the accuracy and reasonableness of the outputs of a trained model. A data sample not large enough to represent with sufficient richness the significant or qualifying attributes of the members of a population to be classified may lead to unfair outcomes. Insufficient datasets may not equitably reflect the qualities that should rationally be weighed in producing a justified outcome that is consistent with the desired purpose of the AI system. Members of the project team with technical and policy competences should collaborate to determine if the data quantity is, in this respect, sufficient and fit-for-purpose.

  • Source Integrity and Measurement Accuracy Effective bias mitigation begins at the very commencement of data extraction and collection processes. Both the sources and instruments of measurement may introduce discriminatory factors into a dataset. When incorporated as inputs in the training data, biased prior human decisions and judgments—such as prejudiced scoring, ranking, interview-data or evaluation—will become the ‘ground truth’ of the model and replicate the bias in the outputs of the system. In order to secure discriminatory non-harm, you must do your best to make sure your data sample has optimal source integrity. This involves securing or confirming that the data gathering processes involved suitable, reliable, and impartial sources of measurement and sound methods of collection.

  • Timeliness and Recency If your datasets include outdated data then changes in the underlying data distribution may adversely affect the generalisability of your trained model. Provided these distributional drifts reflect changing social relationship or group dynamics, this loss of accuracy with regard to the actual characteristics of the underlying population may introduce bias into your AI system. In preventing discriminatory outcomes, you should scrutinise the timeliness and recency of all elements of the data that constitute your datasets.

  • Relevance, Appropriateness and Domain Knowledge The understanding and utilisation of the most appropriate sources and types of data are crucial for building a robust and bias-mitigating AI system. Solid domain knowledge of the underlying population distribution and of the predictive or classificatory goal of the project is instrumental for choosing optimally relevant measurement inputs that contribute to the reasonable determination of the defined solution. You should make sure that domain experts collaborate closely with your technical team to assist in the determination of the optimally appropriate categories and sources of measurement.

To ensure the uptake of best practices for responsible data acquisition, handling, and management across your AI project delivery workflow, you should initiate the creation of a Dataset Factsheet at the alpha stage of your project. This factsheet should be maintained diligently throughout the design and implementation lifecycle in order to secure optimal data quality, deliberate bias-mitigation aware practices, and optimal auditability. It should include a comprehensive record of data provenance, procurement, pre-processing, lineage, storage, and security as well as qualitative input from team members about determinations made with regard to data representativeness, data sufficiency, source integrity, data timeliness, data relevance, training/ testing/validating splits, and unforeseen data issues encountered across the workflow.

Application Fairness

Fairness considerations should enter into your AI project at the earliest possible stage of horizon scanning, problem selection, and project planning. This is because, the overall fairness and equity of an AI application is significantly determined by the objectives, goals, and policy choices that lie behind initial decisions to dedicate time and resources to its design, development, and deployment. For example, the choice made to build a biometric identification system, which uses live facial recognition technology to identify criminal suspects at public events, may be motivated by the objective to increase public safety and security. However, many members of the public may find this use of AI technology unreasonable, disproportionate, and potentially discriminatory. In particular, members of communities historically targeted by disproportionate levels of surveillance in law enforcement contexts may be especially concerned about the potential for abuse and harm. Appropriate fairness and equity considerations should, in this case, occur at the horizon scanning and project planning stage (e.g., as part of a stakeholder impact assessment process that includes engagement with potentially affected individuals and communities). Aligning the policy goals of a project team with the reasonable expectations and potential equity concerns of those affected is a key component of the fairness of the application.

Application fairness, therefore, entails that the policy objectives of an AI project are non-discriminatory and are acceptable to and line up with the aims, expectations, and sense of justice possessed by those affected.-----

(Dobbe et el., 2018; Eckhouse et al. 2018; Green 2018; Green &Hu 2018; Mitchell et al., 2021; Passi & Barocas, 2019).

As such, whether the decision to engage in the production and use of an AI technology can be described as “fairness-aware” depends upon ethical and policy considerations that are external and prior to considerations of the technical feasibility of building an accurate and optimally performing system or the practical feasibility of accessing, collecting, or acquiring enough and the right kind of data.

Beyond this priority of assuring the equity and ethical permissibility of policy goals, application fairness requires additional considerations in framing decisions made at the horizon scanning and project scoping or planning stage:

1

Equity considerations surrounding real-world context of   the policy issue to be solved: When your project team is assessing the fairness of using an AI solution to address a particular policy issue, it is important to consider how equity considerations extend beyond the statistical and sociotechnical contexts of designing, developing, and deploying the system. Applied concepts of AI fairness and equity should not be treated, in a technology-centred way, as originating exclusively from the design and use of any particular AI system. Nor should they exclusively be treated as abstractions that can be engineered into an AI application through technical or mathematical retooling (e.g., by operationalising formal fairness criteria).--- When designers of algorithmic systems limit their understanding of the scope of ‘fairness’ or ‘equity’ to these two dimensions, it can artificially constrain their perspectives in such a way that they erroneously treat only the patterns of bias and discrimination that arise in AI innovation practices or that can be measured, formalised, or statistically digested as actionable indicators of inequity.

Rather, equity considerations should be grounded in a human-centred approach, which includes reflection on and critical examination of the wider social and economic patterns of disadvantage, injustice, and discrimination that arise in the real-world contexts surrounding the policy issue in question. Such considerations should include an exploration of how such patterns of inequity may lead, on the one hand, to the disparate distribution of the risks and adverse impacts of the AI system or, on the other, to a lack of equitable access to its potential benefits. For instance, while the development of an AI chatbot to replace a human serviced medical helpline may provide effective healthcare guidance for some, it could have disparate adverse impacts on others, such as vulnerable elderly populations or socioeconomically deprived groups who may face barriers to accessing and using the app. Here, reflection on the real-world contexts surrounding the provision of this type of healthcare support yields a more informed and compassionate awareness of social and economic conditions that could impair the fairness of the application.
(Fazelpour & Lipton, 2020; Green & Viljoen, 2020; Leslie 2020; Selbst et al., 2019).

2

Equity considerations surrounding the group targeted by the AI innovation intervention: Each AI application that makes predictions about or classifies people, targets a specific subset of the wider population to which they belong. For instance, a résumé filtering system that is used to select desirable candidates in a recruitment process will draw from a pool of job applicants that constitute a subgroup within the broader population. Potential equity issues may arise here because the selection of subpopulations sampled by AI applications is non-random. Instead, the sample selection may reflect particular social patterns, structures, and path dependencies that are unfair or discriminatory. In the case of the résumé filtering system, the sample may reflect long-term hiring patterns where a disproportionate number of male job candidates from elite universities (or those with similarly privileged educational backgrounds) have been actively recruited. Such practices have historically excluded people from other gender identities and socioeconomic and educational backgrounds. As a result, the pattern of inequity surfaces, in this instance, not within the sampled subset of the population but rather in the way that discriminatory social structures have led to the selection of a certain group of individuals into that subset.-
<!-- (Mitchell et al., 2021). -->
<!-- (Mehrabi et al., 2021; Olteanu et al., 2019) -->

3

Equity considerations surrounding the way that the model’s output shapes the range of possible decision-outcomes: AI applications that assist human decision-making shape and delimit the range of possible outcomes for the problem areas they address. For example, a predictive risk model used in children’s social care may generate an output that directly influences the space of choices available to a social worker. Because the model’s target is the identification of at-risk children, it may lead to social care decisions that focus narrowly on whether a child needs to be taken into care. This centring of negative outcomes could restrict the range of viable choices open to the social worker insofar as it de-emphasises the potential availability of other strengths-based approaches (e.g., stewarding positive family functioning through social supports and identifying and promoting protective factors). These alternative decision-making paths could be closed off in the social care environment given how the predictive risk model’s outputs restrictively shape the range of actions that can be taken to address the problem it is being used to inform.

Model design and development fairness

Because human beings have a hand in all stages of the construction of AI systems, fairness-aware design must take precautions across the AI project workflow to prevent bias from having a discriminatory influence:

Design phase

Problem formulation At the initial stage of problem formulation and outcome definition, technical and non-technical members of your team should work together to translate project goals into measurable targets.---- This will involve the use of both domain knowledge and technical understanding to define what is being optimised in a formalisable way and to translate the project’s objective into a target variable or measurable proxy, which operates as a statistically actionable rendering of the defined outcome.

At each of these points, choices must be made about the design of the algorithmic system that may introduce structural biases which ultimately lead to discriminatory harm. Special care must be taken here to identify affected stakeholders and to consider how vulnerable groups might be negatively impacted by the specification of outcome variables and proxies. Attention must also be paid to the question of whether these specifications are reasonable and justifiable given the general purpose of the project and the potential impacts that the outcomes of the system’s use will have on the individuals and communities involved.

These challenges of fairness aware design at the problem formulation stage show the need for making diversity and inclusive participation a priority from the start of the AI project lifecycle. This involves both the collaboration of the entire team and the attainment of stakeholder input about the acceptability of the project plan. This also entails collaborative deliberation across the project team and beyond about the ethical impacts of the design choices made.

Development phase

Data pre-processing Human judgement enters into the process of algorithmic system construction at the stage of labelling, annotating, and organising the training data to be utilised in building the model. Choices made about how to classify and structure raw inputs must be taken in a fairness-aware manner with due consideration given to the sensitive social contexts that may introduce bias into such acts of classification. Similar fairness aware processes should be put in place to review automated or outsourced classifications. Likewise, efforts should be made to attach solid contextual information and ample metadata to the datasets, so that downstream analyses of data processing have access to properties of concern in bias mitigation.

The constructive task of selecting the attributes or features that will serve as input variables for a model requires human decisions to be made about the sorts of information that may or may not be relevant or rationally required to yield an accurate and unbiased classification or prediction. Moreover, the feature engineering tasks of aggregating, extracting, or decomposing attributes from datasets may introduce human appraisals that have biasing effects.

At this stage, human decisions about how to group or disaggregate input features (e.g., how to carve up categories of gender or ethnic groups) or about which input features to exclude altogether (e.g., leaving out deprivation indicators in a predictive model for clinical diagnostics) can have significant downstream influences on the fairness and equity of an AI system. This applies even when algorithmic techniques are employed to extract and engineer features, or support the selection of features (e.g., to optimise predictive power).

For this reason, discrimination awareness should play a large role at this stage of the AI model-building workflow as should domain knowledge and policy expertise. Your team should proceed in the model development stage aware that choices made about grouping or separating and including or excluding features as well as more general judgements about the comprehensiveness or coarseness of the total set of features may have significant consequences for historically marginalised, vulnerable, or protected groups.

Model selection The model selection stage determines the model type and structure that will be produced in the next stages. In some projects, this will involve the selection of multiple prospective models for the purpose of comparison based on some performance metric, such as accuracy or sensitivity. The set of relevant models is likely to have been highly constrained by many of the issues dealt with in previous stages (e.g., available resources and skills, problem formulation), for instance, where the problem demands a supervised learning algorithm instead of an unsupervised learning algorithm.

Fairness and equity issues can surface in model selection processes in at least two ways: First, they can arise when the choice between algorithms has implications for explainability. For instance, it may be the case that there are better performing models in the pool of available options but which are less interpretable than others. This difference becomes significant when the processing of social or demographic data increases the risk that biases or discriminatory proxies lurk in the algorithmic architecture. Model interpretability can increase the likelihood of detecting and redressing such discriminatory elements.

Second, fairness and equity issues can arise when the choice between algorithms has implications for the differential performance of the final model for subgroups of the population. For instance, where several different learning algorithms are simultaneously trained and tested, one of the resulting models could have the highest overall level of accuracy while, at the same time, being less accurate than others in the way it performs for one or more marginalised sub-groups. In cases like this, technical members of your project team should proceed with attentiveness to mitigating any possible discriminatory effects of choosing one model over another and should consult members of the wider team—and impacted stakeholders, where possible—about the acceptability of any trade-offs between overall model accuracy and differential performance.

Model training, testing, and validation

The process of tuning hyperparameters, setting metrics, and resampling data at the training, validation, and testing stages also involves human choices that may have fairness and equity consequences in the trained model. For instance, the way your project team determines the training-testing split of the dataset can have a considerable impact on the need for external validation to ensure that the model’s performance “in the wild” meets reasonable expectations. Therefore, your technical team should proceed with an attentiveness to bias risks, and continual iterations of peer review and project team consultation should be encouraged to ensure that choices made in adjusting the dials, parameters, and metrics of the model are in line with bias mitigation and discriminatory non-harm goals.

Evaluating and validating model structures

Design fairness also demands close assessment of the existence in the trained model of lurking or hidden proxies for discriminatory features that may act as significant factors in its output. Including such hidden proxies in the structure of the model may lead to implicit ‘redlining’ (the unfair treatment of a sensitive group on the basis of an unprotected attribute or interaction of attributes that ‘stands in’ for a protected or sensitive one).

Designers must additionally scrutinise the moral justifiability of the significant correlations and inferences that are determined by the model’s learning mechanisms themselves, for these correlations and inferences could encode social and historical patterns of discrimination where these are baked into the dataset. In cases of the processing of social or demographic data related to human features, where the complexity and high dimensionality of machine learning models preclude the confirmation of the discriminatory non-harm of these inferences (for reason of their opaqueness by human assessors), these models should be avoided.-- In cases where this is not possible, a different, more transparent and explainable model or portfolio of models should be chosen.

(ICO/Turing, 2020; Mitchell et al., 2021; Rudin, 2019)

Model structures must also be confirmed to be procedurally fair, in a strict technical sense. This means that any rule or procedure employed in the processing of data by an algorithmic system should be consistently and uniformly applied to every decision subject whose information is being processed by that system. AI project teams should be able to certify that any relevant rule or procedure is applied universally and uniformly to all relevant individuals.

Implementers of the system, in this respect, should be able to show that any model output is replicable when the same rules and procedures are applied to the same inputs. Such a uniformity of the application of rules and procedures secures the equal procedural treatment of decision subjects and precludes any rule-changes in the algorithmic processing targeted at a specific person that may disadvantage that individual vis-à-vis any other. It is important to note that the consistent and uniform application of rules over time apply to deterministic algorithms, whose parameters are static and fixed after model development. Close attention should be paid, in this sense, to the procedural fairness issues that may arise with the use of dynamic learning algorithms. This is because the parameters and inferences of such systems evolve over time and can yield different outputs for the same inputs at different points in system’s lifespan.

Metric-based fairness

As part of the safeguarding of discriminatory non-harm and diligent fairness and equity considerations, well-informed consideration must be put into how you are going to define and measure the formal metrics of fairness that can be operationalised into the AI system you are developing.

Metric-based fairness involves the mathematical mechanisms that can be incorporated into an AI model to allocate the distribution of outcomes and error rates for relevant subpopulations (e.g., groups with protected or sensitive characteristics). In formulating your approach to metric-based fairness, your project team will be confronting challenging issues like the justifiability of differential treatment based on sensitive or protected attributes, where differential treatment can indicate differences in the distribution of model outputs or the distribution of error rates and performance indicators like precision or sensitivity.

There is a great diversity of beliefs in this area as to what makes the consequences of an algorithmically supported decision allocatively equitable, fair, and just. Different approaches to metric-based fairness—detailed below—stress different principles: some focus on demographic parity, some on individual fairness, others on error rates equitably distributed across subpopulations. Regardless of this diversity of views, your project team must ensure that the choices made regarding formal fairness metrics are lawful and conform to governing equality, non-discrimination, and human rights laws. Where appropriate, relevant experts should be consulted to confirm the legality of such choices.

Your determination of metric-based fairness should heavily depend both on the specific use case being considered and the technical feasibility of incorporating your chosen criteria into the construction of the AI system. (Note that different fairness-aware methods involve different types of technical interventions at the pre-processing, modelling, or post-processing stages of production). Again, this means that determining your fairness definition should be a cooperative and multidisciplinary effort across the project team.

You will find below a summary table of several of the main definitions of metric-based fairness that have been integrated by researchers into formal models as well as a list of current articles and technical resources, which should be consulted to orient your team to the relevant knowledge base. (Note that this is a rapidly developing field, so your technical team should keep updated about further advances.) The first four fairness types fall under the category of group fairness and allow for comparative criteria of non-discrimination to be considered in model construction and evaluation. The final two fairness types focus instead on cases of individual fairness, where context-specific issues of effective bias are considered and assessed at the level of the individual agent.

Take note, though, that these technical approaches have limited scope in terms of the bigger picture issues of application and design fairness that we have already stressed. Moreover, metric-based approaches face other practical and technical barriers. For instance, to carry out group comparisons, formal approaches to fairness require access to data about sensitive/protected attributes as well as accurate demographic information about the underlying population distribution (both of which may often be unavailable or unreliable, and furthermore, the work of identifying sensitive/protected attributes may pose additional risks of bias). Metric-based approaches also face challenges in the way they handle combinations of protected or sensitive characteristics that may amplify discriminatory treatment. These have been referred to as intersectional attributes (e.g. the combination gender and race characteristics), and they must also be integrated into fairness and equity considerations. Lastly, there are unavoidable trade-offs, inconsistencies, or even incompatibilities, between mathematical definitions of metric-based fairness that must be weighed in determining which of them are best fit for a particular use case. For instance, the desideratum for equalised odds (error rate balance) across subgroups can clash with the desideratum for equalised model calibration (correct predictions of gradated outcomes) or parity of positive predictive values across subgroups.----

Selecting fairness metrics

Demographic/statistical parity (group fairness)

An outcome is fair if each group in the selected set receives benefit in equal or similar proportions, i.e. if there is no correlation between a sensitive or protected attribute and the allocative result.

---- This approach is intended to prevent disparate impact, which occurs when the outcome of an algorithmic process disproportionately harms members of disadvantaged or protected groups.

(Calders et al., 2009; Denis, 2021; Dwork et al., 2011; Feldman et al., 2015; Zemel et al., 2013).

True positive rate parity (group fairness)

An outcome is fair if the ‘true positive’ rates of an algorithmic prediction or classification are equal across groups. This approach is intended to align the goals of bias mitigation and accuracy by ensuring that the accuracy of the model is equivalent between relevant population subgroups. This method is also referred to as ‘equal opportunity’ fairness because it aims to secure equalised odds of an advantageous outcome for qualified individuals in a given population regardless of the protected or disadvantaged groups of which they are members.

(Zafar et al., 2017).

Equalised odds (group fairness)

An outcome is fair if false positive and true positive rates are equal across groups. In other words, both the probability of incorrect positive predictions and the probability of correct positive predictions should be the same across protected and privileged groups.- This approach is motivated by the position that sensitive groups and advantaged groups should have similar error rates in outcomes of algorithmic decisions.

(Hardt et al., 2016; Verma & Rubin, 2018).

Positive predictive value parity (group fairness)

An outcome is fair if the rates of positive predictive value (the fraction of correctly predicted positive cases out of all predicted positive cases) are equal across sensitive and advantaged groups. Outcome fairness is defined here in terms of a parity of precision, where the probability of members from different groups actually having the quality they are predicted to have is the same across groups.

(Chouldechova, 2017).

Individual fairness (individual fairness)

An outcome is fair if it treats individuals with similar relevant qualifications similarly. This approach relies on the establishment of a similarity metric that shows the degree to which pairs of individuals are alike with regard to a specific task.-

(Dwork et al., 2012; Kusner et al., 2017).

Counterfactual fairness (individual fairness)

An outcome is fair if an automated decision made about an individual belonging to a sensitive group would have been the same were that individual a member of a different group in a closest possible alternative (or counterfactual) world. Like the individual fairness approach, this method of defining fairness focuses on the specific circumstances of an affected decision subject, but, by using the tools of contrastive explanation, it moves beyond individual fairness insofar as it brings out the causal influences behind the algorithmic output. It also presents the possibility of offering the subject of an automated decision knowledge of what factors if changed, could have influenced a different outcome. This could provide them with actionable recourse to change an unfavourable decision.

System implementation fairness

When your project team is approaching the beta stage, you should begin to build out your plan for implementation training and support. This plan should include adequate preparation for the responsible and unbiased deployment of the AI system by its on-the-ground users. Automated decision-support systems present novel risks of bias and misapplication at the point of delivery, so special attention should be paid to preventing harmful or discriminatory outcomes at this critical juncture of the AI project lifecycle. In order to design an optimal regime of implementer training and support, you should pay special attention to the unique pitfalls of bias-in-use to which the deployment of AI technologies give rise. These can be loosely classified as decision-automation bias (more commonly just ‘automation bias’) and automation-distrust bias:

Decision-Automation Bias I (over-reliance)

Users of automated decision-support systems may tend to become hampered in their critical judgement, rational agency, and situational awareness as a result of their faith in the perceived objectivity, neutrality, certainty, or superiority of the AI system (Gaube et al., 2021). This may lead to over-reliance or errors of omission, where implementers lose the capacity to identify and respond to the faults, errors, or deficiencies, which might arise over the course of the use of an automated system, because they become complacent and overly deferent to its directions and cues.

(Bussone et al., 2015).

Decision-Automation Bias II (over-reliance)

Decision-automation bias may also lead to over-compliance or errors of commission where implementers defer to the perceived infallibility of the system and thereby become unable to detect problems emerging from its use for reason of a failure to hold the results against available information. Both over-reliance and over-compliance may lead to what is known as out-of-loop syndrome where the degradation of the role of human reason and the de-skilling of critical thinking hampers the user’s ability to complete the tasks that have been automated. This condition may bring about a loss of the ability to respond to system failure and may lead both to safety hazards and to dangers of discriminatory harm. To combat risks of decision-automation bias, you should operationalise strong regimes of accountability at the site of user deployment to steer human decision-agents to act on the basis of good reasons, solid inferences, and critical judgment.

Automation-Distrust Bias

At the other extreme, users of an automated decision-support system may tend to disregard its salient contributions to evidence-based reasoning either as a result of their distrust or scepticism about AI technologies in general or as a result of their over-prioritisation of the importance of prudence, common sense, and human expertise. An aversion to the non-human and amoral character of automated systems may also influence decision subjects’ hesitation to consult these technologies in high impact contexts such as healthcare, transportation, and law. ‘Research shows that people often prefer humans’ forecasts to algorithms’ forecasts,- more strongly weigh human input than algorithmic input,- and more harshly judge professionals who seek out advice from an algorithm rather than from a human.-

(Longoni et al., 2019). (Dietvorst et al., 2015). (Diab, Pui, Yankelevich, & Highhouse, 2011; Eastwood, Snook, & Luther, 2012), (Önkal, Goodwin, Thomson, Gönül, & Pollock, 2009; Promberger & Baron, 2006) (Shaffer, Probst, Merkle, Arkes, & Medow, 2013)’ (Dietvorst et al., 2015).

Taking account of the context of impacted individuals in system implementation

In cases where you are utilising a decision-support AI system that draws on statistical inferences to determine outcomes which affect individual persons (e.g., a predictive risk model that helps an adult social care work determine an optimal path to caring for an elderly patient), fair implementation processes should include considerations of the specific context of each impacted individual.--- Any application of this kind of system’s recommendation will be based on statistical generalisations, which pick up relationships between the decision recipient’s input data and patterns or trends that the AI model has extracted from the underlying distribution of that model’s original dataset. Such generalisations will be predicated on inferences about a decision subject’s future behaviours (or outcomes) that are based on populational-level correlations with the historical characteristics and attributes of the members of groups to which that person belongs rather than on the specific qualities of the person themself.

(Binns, 2107, 2018, 2019; Eiselson, 2013).

Fair and equitable treatment of decision subjects entails that their unique life circumstances and individual contexts be taken into account in decision-making processes that are supported by AI-enabled statistical generalisations. For this reason, you should train your implementers to think contextually and holistically about how these statistical generalisations apply to the specific situation of the decision recipient. This training should involve preparing implementers to work with an active awareness of the socio-technical aspect of implementing AI decision-assistance technologies from an integrative and human- centred point of view. You should train implementers to apply the statistical results to each particular case with appropriate context-sensitivity and ‘big picture’ sensibility. This means that the dignity they show to decision subjects can be supported by interpretive understanding, reasonableness, and empathy.

Ecosystem fairness

The AI project lifecycle does not exist in isolation from, or independent of, the wider social system of economic, legal, cultural, and political structures or institutions in which the production and use of AI systems take place. Rather, because it is embedded in these structures and institutions, the policies, norms, and procedures through which such structures and institutions influence human action also influence the AI project lifecycle itself. Inequities and biases at this ecosystem level can steer or shape AI research and innovation agendas in ways that can generate inequitable outcomes for protected, marginalised, vulnerable, or disadvantaged social groups. Such ecosystem-level inequities and biases may originate in and further reinforce asymmetrical power structures, unfair market dynamics, and skewed research funding schemes that favour or bring disproportionate benefit to those in the majority, or those who wield disproportionate power in society, at the cost of those who are disparately impacted by the discriminatory outcomes of the design, development, and use of AI technologies.

Ecosystem-level inequities can occur across a wide range of AI research and innovation contexts. For instance, when AI-enabled health interventions such as mobile-phone-based symptom checker apps or remote AI-assisted medical triaging or monitoring are designed without regard for the barriers to access faced by protected, vulnerable, or disadvantaged groups, they will disproportionately benefit users from other, more advantaged groups. Likewise, where funding of the development of AI technologies, which significantly affect the public interest, is concentrated in the hands of firms or vendors who singly pursue commercial or financial gain, this may result in exclusionary research and innovation environments, AI systems that are built without due regard for broad fairness and equity impacts, and limitations on the development and deployment of wide-scale, publicly beneficial technology infrastructure. Moreover, widespread structural and institutional barriers to diversity and inclusion can create homogeneity on AI project teams (and among organisational policy owners) that has consequential fairness impacts on application decisions, resource allocation choices, and system deployment strategies.

The concept of ecosystem fairness highlights the importance of mitigating the range of inequity-generating path dependencies that originate at the ecosystem level and that are often neglected by or omitted from analyses of AI project lifecycles. Ecosystem fairness therefore focuses both on rectifying the social structures and institutions that engender indirect discrimination and on addressing the structural and institutional changes needed for the corrective modification of social patterns that engender discriminatory impacts and socio-economic disadvantage. In this way, ecosystem fairness involves the transformation of unjust economic, legal, cultural, and political structures or institutions with the aim of the universal realisation of equitable social arrangements.


Bias Mitigation

Bias Self-assessment and Risk Management

By pinpointing risks of bias or downstream discriminations, project teams can streamline possible solutions in a proactive, pre-emptive, and anticipatory way. This is what fairness-aware design and implementation will enable the team to do.

At each stage of the AI project lifecycle, a collaborative Bias Self-assessment should be carried out with regard to the applicable dimension of fairness. This self-assessment consists of three steps:

Step 1: Familiarising with biases and fairness types that are relevant to each project stage.

Step 2: Reflecting and identifying how a particular AI project might be vulnerable to biases that may arise at each stage and pose risks to each relevant fairness type.

Step 3: Determine and document bias risk mitigation actions that will be implemented to correct any existing problems that have been identified, strengthen areas of weakness that have possible discriminatory consequences, and take proactive bias-prevention measures in areas that have been identified as potential sources of risk.

The Fairness Self-Assessment and Risk Mitigation template will help you go through this process. It locates a set of social, statistical, and cognitive biases within specific steps of an AI project lifecycle. These biases require ongoing reflection and deliberation to minimise the possible negative impact upon downstream activities or the risk of discriminatory outcomes.

List of Biases1

### Bias categories **Social bias:** Social bias has to do with the way that pre-existing or historical patterns of discrimination and social injustice – and the prejudices and discriminatory attitudes that correspond to such patterns – can be drawn into the Al lifecycle. In particular, it relates to how these patterns and attitudes can be perpetuated, reinforced, or exacerbated through the development and deployment of data-driven technologies. **Statistical bias:** Statistical bias refers to a systematic deviation from an expected statistical result that arises due to the influence of some additional factor. This understanding is common in observational studies where bias can arise in the process of sampling or measurement. Statistical biases can involve errors (deviations from a true state) or differences between measured or calculated values and true values. **Cognitive bias:** Cognitive bias refers to a systematic deviation from a norm of rationality that can occur in processes of thinking or judgement and that can lead to mental errors, misinterpretations of information, or flawed patterns of response to decision problems. ## Social Bias

World biases

Historical bias: Historical bias concerns pre-existing societal patterns of discrimination and social injustice—and the prejudices and discriminatory attitudes that correspond to such patterns. These patterns, prejudices, and attitudes can be drawn into every stage of the AI innovation lifecycle and be perpetuated, reinforced, or exacerbated through inequitable innovation ecosystem dynamics and the pursuit of biased application choices and research agendas. They can also arise in AI innovation contexts when historical patterns of inequity or discrimination are inadvertently or unintentionally reproduced, or even augmented, in the development and use of an AI system—even when the system is functioning to a high standard of accuracy and reliability.- For instance, even with scientifically sound sampling and feature selection, a project will exhibit historical bias where it perpetuates (or exacerbates) socioeconomic inequalities through the outcomes it generates.

(Mehrabi et al., 2019; Suresh & Guttag, 2019).

Structural racism: Structural racism (also sometimes called systemic racism) is a form of racial discrimination that is ‘not simply the result of private prejudices held by individuals, but is also produced and reproduced by laws, rules, and practices, sanctioned and even implemented by various levels of government, and embedded in the economic system as well as in cultural and societal norms’. Other forms of discrimination such as sexism, classism, ableism, ageism, antisemitism, and transphobia can also similarly have structural or systemic aspects.

(Bailey et al., 2021, p. 768).

Institutional bias: Institutional bias is ‘a tendency for the procedures and practices of particular institutions to operate in ways which result in certain social groups being advantaged or favoured and others being disadvantaged or devalued. This need not be the result of any conscious prejudice or discrimination but rather of the majority simply following existing rules or norms’.

(Chandler & Munday, 2011, p. 118).

Data biases

Representation bias: When a population is either inappropriately represented (e.g. not allowing sufficient self-representation in demographic variables) or a sub-group is underrepresented in the dataset, the model may subsequently fail to generalise and under-perform for a sub-group (or sub-groups).-[mehrabi2019]- For example, representation bias could arise in a symptom-checking application that has been trained on a data collected exclusively through smartphone use or online interaction as this dataset would likely underrepresent groups within the general population like elderly people who may lack access to smartphones or connectivity.

(Feng et al., 2022; Mehrabi et al., 2019; Suresh & Guttag, 2019).

Selection bias: Selection bias is a term used for a range of biases that affect the selection or inclusion of data points within a dataset. In general, this bias arises when an association is present between the variables being studied and additional factors that make it more likely that some data will be present in a dataset when compared to other possible data points in the space. For instance, where individuals differ in their geographic or socioeconomic access to an activity or service that is the site of data collection, this variation may result in their exclusion from the corresponding dataset. Likewise, where certain socioeconomically deprived or marginalised social groups are disproportionately dependent on a social service to fulfil basic needs, it may be over sampled if data is collected from the provision of that service.

Admission bias: Admission rate bias is a subtype of selection bias. It occurs when the exposure to a risk and the occurrence of a disease increases the likelihood of admission or referral (hence, this is also known as ‘referral bias’). As a result, the cases included in any subsequent study or dataset will systematically differ from the population at large.

(Spencer & Heneghan, 2017).

Diagnostic decision bias: Diagnostic decision bias occurs in medical datasets that are based on implicit biases in clinical judgement about patient diagnosis (for instance, where biased medical training and unrepresentative or discriminatory clinical knowledge lead to the under-diagnosis of minority or under-served groups). This leads to ‘differential diagnosis of patients with the same pathology, on the basis of demographic features’ (Straw & Callison-Burch, 2020, p. 2) and, ultimately, datasets that reflect biased diagnoses rather than the true rate of disease.

Chronological bias: Chronological bias arises when individuals in the dataset are added at different times, and where this chronological difference results in individuals being subjected to different methods or criteria of data extraction based on the time their data were recorded. For instance, if the dataset used to build a predictive risk model in children’s social care spans over several years, large-scale care reforms, policy changes, adjustments in relevant statutes (such as changes to legal thresholds or definitions), and changes in data recording methods may create major inconsistencies in the data points extracted from person to person.

Diagnostic access bias: Where barriers to patient access exist for diagnostic tests, there can be systematic exclusion of sub-groups from a dataset, resulting in the under- or over-estimation of the true prevalence or incidence of the disease. These barriers can exist for myriad reasons, including cultural (e.g. distrust of formal healthcare institutions), geographic (e.g. remote living), socioeconomic (e.g. unable to get time off from childcare or work), among other reasons.

Prevalence-incidence bias: Prevalence-incidence bias occurs as a result of the timing in which cases are included in the dataset for a study. For example, excluding patients who have died could result in the appearance of decreased severity for a disease. A longer time frame between exposure and investigation could increase the likelihood of additional patients dying or recovering and being excluded from an analysis.

Data coding bias: Data coding bias occurs when the misrepresentation or erasure of demographic characteristics such as gender or ethnicity by biased coding systems obscures patient needs, adversely impacts patient’s access to appropriate screenings, diagnosis, and treatments, and, subsequently, prejudices the datasets in which biased coding is embedded.

Missing data bias: Missing data can cause a wide variety of issues within an AI project, and these data may be missing for a variety of reasons related to broader social factors. Missing data bias can lead to inaccurate inferences and affect the validity of the model where it is the result of non-random but statistically informative events.- For instance, missing data bias may arise in predictive risk models used in social care to detect potentially harmful behaviour in adolescents where interview responses and longitudinal data collected over extended periods of time are used as part of the dataset. This can be seen in cases where interview questions about socially stigmatised behaviours or traits like drug use or sexual orientation trigger fears of punishment, humiliation, or reproach and thus prompt non-responses, and in cases where data collection over time leads to the inconsistent involvement and drop-out of study participants.

Wrong sample-size bias: Using the wrong sample size for the study can lead to chance findings that fail to represent adequately the variability of the underlying data distribution, in the case of small samples, or findings that are statistically significant but not relevant or actionable, in the case of larger samples. Wrong sample size bias may occur in cases where model designers have included too many features in a machine learning algorithm. This is often referred to as the ‘curse of dimensionality’, a mathematical phenomenon wherein increases in the number of features or ‘data dimensions’ included in an algorithm means that exponentially more data points need to be sampled to enable good predictive or classificatory performance.

Measurement quality bias: Measurement quality bias arises when under-resourced clinical environments lack the personnel, expertise, training capacity, and digital maturity to consistently collect high quality & complete data. They can also occur during deployment when measurement tools & devices are designed for dominant groups and consequently mismeasure minoritised, disadvantaged, or underserved groups.

Design, development, and deployment biases

Hardware bias: Hardware bias arises where physically instantiated algorithmic systems or measurement devices are not designed to consider the diverse physiological needs of minoritised, marginalised, disadvantaged, or other non-majority stakeholders. When deployed, such systems will therefore perform less effectively for members of these groups due to their design.

Annotation bias: Annotation bias occurs when annotators incorporate their subjective perceptions into their annotations.-[kuwatly2020] Data is often annotated by trained expert annotators or crowdsourced annotators. In its simplest form, annotators can choose an inaccurate label due to fatigue or lack of focus but annotation bias can also result from positionality limitations that derive from demographic features, such as age, education, or first language,[kuwatly2020] and other systemic cultural or societal biases that influence annotators. For instance, annotators may label differently facial expressions of different ethnic, age, or gender groups, have different levels of familiarity with communication norms, or different understandings of what should be annotated as harmful content. When the role of annotator subjectivity is unacknowledged or annotators are not specifically trained to mitigate biases, there are greater chances that the model will incorporate annotation biases and be unfair.-

(Chen & Joo, 2021; Kuwatly et al., 2020). (Hovy & Prabhumoye, 2021) (Röttger et al., 2022)

Label or label choice bias: A label (or target variable) used within an algorithmic model may not have the same meaning for all data subjects. There may be a discrepancy between what sense the designers are seeking to capture in a label, or what they are trying to measure in it, and the way that affected individuals understand its meaning.--- Where there is this kind of variation in meaning for different groups within a population, adverse consequences and discriminatory impact could follow. For example, designers of a predictive model in public health may choose ‘patient wellbeing’ as their label, defining it in terms of disease prevalence and hospitalisation. However, subpopulations who suffer from health disparities and socioeconomic deprivation may understanding wellbeing more in terms of basic functionings, the food security needed for health promotion, and the absence of the social environmental stressors that contribute to the development of chronic medical conditions. Were this predictive model to be used to develop public health policy, members of this latter group could suffer from a further entrenchment of poor health outcomes.

(Corbett‐Davies & Goel, 2018; Feng et al., 2022; Obermeyer et al., 2019, 2021; Rajkomar et al., 2018)

Measurement bias: Measurement bias addresses the choice of how to measure the labels or features being used. It arises when the measurement scale or criteria being applied fails to capture data pertaining to the concepts or constructs that are being measured in a fair and equitable manner.---[olteanu2019]- For example, a recidivism risk model that uses prior arrests or arrested relatives as proxies to measure criminality may surface measurement bias insofar as patterns of arrest can reflect discriminatory tendencies to over-police certain protected social groups or biased assessments on the part of arresting officers rather than true criminality.

(Jacobs & Wallach, 2021; Mehrabi et al., 2019; Mitchell et al., 2021; Olteanu et al., 2019; Suresh & Guttag, 2019).

Cohort bias: Cohort bias is a subtype of measurement bias where model designers use categories that default ‘to traditional or easily measured groups without considering other potentially protected groups or levels of granularity (e.g. whether sex is recorded as male, female, or other or more granular categories)’.

(Rajkomar et al., 2018, p. 8).

Model selection bias: Model selection bias occurs when AI designers and developers choose a model that does not sufficiently respond to the needs and requirements of the research question or problem and the domain context or use-case. This may result not only in a lack of appropriate transparency and explainability (where a complex model is chosen and the context demands interpretable results) but also in outcomes based upon model inferences that do not reflect the level of nuance needed to confront the question or problem itself.

Evaluation bias: Evaluation bias occurs during model iteration and evaluation as a result (1) of the application of evaluative metrics that mask the differential performance of an AI model for subgroups or (2) of the use of benchmarking datasets that do not accurately represent the composition of the target population. For example, an evaluation bias may occur where performance metrics that measure only overall accuracy are applied to a trained computer vision system that performs differentially for subgroups that have different skin tones. Likewise, evaluation biases arise where the external benchmark datasets that are used to evaluate the performance of trained models are insufficiently representative of the populations to which they will be applied. In the case of computer vision, this may occur where established benchmarks overly represent a segment of the populations (such as adult light-skinned males) and thus reinforce the biased criteria for optimal performance.

(Suresh & Guttag, 2019)

Semantic bias: Semantic bias occurs when discriminatory inferences are allowed to arise in the architecture of a trained AI model and to remain an element of the productionalised system. When historical biases are baked into datasets in the form of discriminatory proxies or embedded prejudices (e.g. word embeddings that pick up on racial or gender biases), these biases can be semantically encoded in the model’s co-variates and parameters. Semantic biases occur when model design and evaluation processes fail to detect and mitigate such discriminatory aspects.

(Hovy & Prabhumoye, 2021)

Confounding: Confounding is a well-known causal concept in statistics, and commonly arises in observational studies. It refers to a distortion that arises when a (confounding) variable independently influences both the dependant and independent variables (e.g. exposure and outcome), leading to a spurious association and a skewed output. Clear examples of confounding can be found in the use of electronic health records (EHRs) that arise in clinical environments and healthcare processes. EHRs are observational data and often reflect not only the health status of patients, but also patients’ interactions with the healthcare system. This can introduce confounders such as the frequency of inpatient medical testing reflecting the busyness or labour shortages of medical staff rather than the progression of a disease during hospitalisation, differences between onset of a disease and the date of diagnosis, and health conditions that are missing from the EHRs of a patient due to a non-random lack of testing. Contextual awareness and domain knowledge are crucial elements for identifying and redressing confounders.

Social determinant blindness: Social determinant blindness occurs where clinical researchers and/or AI model designers act with a lack of awareness of the environmental, occupational, and life-course exposures of patients who face precarity, historical marginalisation, and discrimination. This leads to ‘a lack of data on the complete etiologic context and exposures of a patient’s health, including lack of opportunities and resources, such as broadband connectivity and social factors that can trigger adverse health outcomes’.

(Dankwa-Mullan & Weeraratne, 2022, p. 1425).

Aggregation bias: Aggregation bias arises when a ‘one-size-fits-all’ approach is taken to the outputs of a trained algorithmic model (i.e. one where model results apply evenly to all members of the impacted population) even where variations in subgroup characteristics mean that mapping functions from inputs to outputs are not consistent across these subgroups.- In other words, in a model where aggregation bias is present, even when combinations of features affect members of different subgroups differently, the output of the system disregards the relevant variations in conditional distributions for the subgroups. This results in the loss of information, lowered performance, and, in cases where data from one subgroup is more prevalent than those of others, the development of a model that is more effective for that sub-group. Good examples of aggregation bias come up in clinical decision-support systems in medicine, where clinically significant variations between sexes and ethnicities—in terms of disease aetiology, expression, complications, and treatment—mean that systems which aggregate results by treating all data points similarly will not perform optimally for any subgroup.

(Mehrabi et al., 2019; Suresh & Guttag, 2019).

Reporting bias: Reporting bias arises when systems are produced without transparently reported evidence of effectiveness across demographic categories or testing for differential performance across sensitive, protected, or intersectional subgroups. This deficit in equity-aware testing and reporting can lead system developers and users to disregard or to attempt to bypass public accountability for equity, fairness, and bias mitigation. This means that transparency and accountability measures that should be in place to ensure that there are no discriminatory harms resulting from the use of the system are obscured or deprioritised.

Population bias and training-serving skew: Population bias occurs when the demographics or characteristics of the cohort that comprises the training dataset differ from those of the original target population to whom the model is applied.- For instance, a polygenic risk model that was trained primarily on data from a cohort of people of European ancestry in a country that has significant ethnic diversity is applied to people of non-European descent in that country and consequently performs worse for them. Another example arises in the social media research context where the demographic composition of internet platform users is skewed towards certain population subgroups and hence can differ from the studied target population. Training-serving skew occurs, in a similar manner, when a model is deployed for individuals whose data is dissimilar to the dataset used to train, test, and validate the model. This can occur, for instance, where a trained model is applied to a population in a geographical area different from where the original data was collected, or to the same population but at a time much later than when the training data was collected. In both cases, the trained model may fail to generalise because the new, out-of-sample inputs are being drawn from populations with different underlying distributions.

(Mehrabi et al., 2019; Olteanu et al., 2019) (Feng et al., 2022)

Race correction bias: Race correction bias arises when algorithmic systems, which are adjusted or corrected for race or ethnicity, guide clinical decisions in ways that may direct more attention, care, or resources to Caucasian patients or to patients from other advantaged ethnic groups than to members of racialised and minoritised groups. They can also lead to informed mistrust, reluctance, and care avoidance on the part of racialised and minoritised groups and thus to harmful health outcomes for them. It has been argued that race correction is not only an unreliable proxy for genetic and biological difference but also fails to surface the underlying social causes of health disparities.

(Vyas et al., 2020)

Cause-effect bias: Cause-effect bias arises when users or implementers of decision-support systems, which generate inferences based upon statistical correlations, mistakenly assume that correlation implies causation without examining the validity of such an attribution.- For instance, a model for predicting pneumonia risk and hospital re-admission shows that patients with asthma have a lower risk of death than non-asthmatics, leading users to erroneously conclusion that having asthma is a protective factor in the risk scenario. However, a closer look at the model’s inferences demonstrates that the correlation of asthma to lower risk was attributable to the fact that patients in the cohort known to have asthma were, as a common practice, automatically triaged to the Intensive Care Unit as a precautionary measure, and this meant that they had a 50% reduction in mortality risk.

(Fazelpour & Danks, 2021; Mehrabi et al., 2019) (Caruana et al., 2015).

Implementation bias: Implementation bias refers to any bias that arises when a system is implemented or used in ways that were not intended by the designers or developers but, nevertheless, made more likely due to affordances of the system or its deployment. For example, a biometric identification system that is used by a public authority to assist in the detection of potential terrorist activity could be repurposed to target and monitor activists or political opponents.

Decision-automation bias: Decision-automation bias arises when users of automated decision-support systems become hampered in their critical judgment, rational agency, and situational awareness as a result of their faith in the efficacy of the system. This may lead to over-reliance or errors of omission, where implementers lose the capacity to identify and respond to the faults, errors, or deficiencies, which might arise over the course of the use of an automated system, because they become complacent and overly deferent to its directions and cues. Decision-automation bias may also lead to over-compliance or errors of commission where implementers defer to the perceived infallibility of the system and thereby become unable to detect problems emerging from its use for reason of a failure to hold the results against available information.

Automation-distrust bias: Automation-distrust bias arises when users of an automated decision-support system disregard its salient contributions to evidence-based reasoning either as a result of their distrust or scepticism about AI technologies in general or as a result of their over-prioritisation of the importance of prudence, common sense, and human expertise. An aversion to the non-human and amoral character of automated systems may also influence decision subjects’ hesitation to consult these technologies in high impact contexts such as healthcare, transportation, and law.

Dismissal bias: Dismissal bias is a ‘conscious or unconscious desensitisation to alerts that are systematically incorrect for a protected group (e.g. an early warning score for patients with sepsis). Alert fatigue is a form of this’.

(Rajkomar et al., 2018, p. 9).

Status quo bias: An affectively motivated preference for “the way things are currently”, which can prevent more innovative or effective processes or services being implemented. This bias is most acutely felt during the transition between projects, such as the choice to deprovision a system and begin a new project, in spite of deteriorating performance from the existing solution. Although this bias is often treated as a cognitive bias, we highlight it here as a social bias to draw attention to the broader social or institutional factors that in part determine the status quo.

Ecosystem biases

Ecosystem bias: Ecosystem bias occurs when economic, legal, cultural, and political structures or institutions—and the policies, norms, and procedures through which these structures and institution influence human action—steer AI research and innovation agendas in ways that generate inequitable outcomes for minoritised, marginalised, vulnerable, historically discriminated against, or disadvantaged social groups. Ecosystem biases, which exist in the wider social system wherein AI technologies are designed and used, may originate in and further entrench asymmetrical power structures, unfair market dynamics, and skewed research funding schemes that favour or bring disproportionate benefit to those in the majority, or those who wield disproportionate power in society, at the cost of those who are disparately impacted by the discriminatory outcomes of the design, development, and use of AI technologies.

(Schwartz et al., 2022)

Privilege bias: Privilege bias occurs when health policies, institutions, and infrastructures skew the benefits of healthcare technologies and medical devices disproportionately towards privileged social groups. ‘Models may be unavailable in settings where protected groups receive care or require technology/sensors disproportionately available to the nonprotected class’.

(Rajkomar et al., 2018, p. 9).

Research bias: Research bias occurs where there is a deficit in social and health equity standards to guide how AI research and innovation is funded, conducted, reviewed, published, and disseminated. It can also manifest in a lack of inclusion and diversity on research teams and in clinical trials, and limited studies incorporating representative real-world data for health insights. Research bias additionally includes inequitable manifestations of funding structures or in incentives set by investors or funding institutions.

(Dankwa-Mullan & Weeraratne, 2022)

Allocation discrepancy: Allocation discrepancy occurs when resources (such as extra clinical attention or social services) are withheld from a protected group because it is associated with fewer positive predictions.

(Rajkomar et al., 2018)

Optimism bias: Also known as the planning fallacy, optimism bias can lead project teams to under-estimate the amount of time required to implement a new system or plan adequately. In the context of the project lifecycle, this bias may arise during project planning, but can create downstream issues when implementing a model during the model productionalisation stage, due to a failure to recognise possible system engineering barriers.

Law of the instrument (Maslow’s hammer): This bias is best captured by the popular phrase ‘If all you have is a hammer, everything looks like a nail’. The phrase cautions against over-reliance on a particular tool or method, often one that is familiar to members of the project team. For example, a project team that is composed of experts in a specific ML technique may over-use that technique and mis-apply it in a context where a different technique would be better suited, or where it would be better not to use ML/AI technology at all.

McNamara fallacy: McNamara fallacy describes the belief that quantitative information is more valuable than other information. This can lead to scientistic reductivism, technochauvinism, or technological solutionism. The McNamara fallacy plays a significant role in biasing AI innovation processes when AI researchers, designers, and developers view algorithmic techniques and statistical insights as the only inputs capable of solving societal problems, thereby actively disregarding interdisciplinary understandings of the subtle historical and sociocultural contexts of inequity and discrimination.

(Schwartz et al., 2022) (Leslie, 2022) (Broussard, 2018 (Morozov, 2013)

Informed mistrust: Informed mistrust occurs where, ‘given historical exploitation and unethical practices, protected groups may believe that a model is biased against them. These patients may avoid seeking care from clinicians or systems that use the model or deliberately omit information. The protected group may be harmed by not receiving appropriate care’.

(Rajkomar et al., 2018, p. 9)

De-agentification bias: De-agentification bias occurs when social structures and innovation practices systemically exclude minoritised, marginalised, vulnerable, historically discriminated against, or disadvantaged social groups from participating or providing input in AI innovation ecosystems. ‘Protected groups may not have input into the development, use, and evaluation of models. They may not have the resources, education, or political influence to detect biases, protest, and force correction’.

(Rajkomar et al., 2018, p. 9)

Feedback loops: Feedback loops occur where ‘the clinician accepts the recommendation of a model even when it is incorrect to do so, [and therefore] the model's recommended versus administered treatments will always match. The next time the model is trained, it will learn to continue these mistakes’.

(Rajkomar et al., 2018, p. 9)

Positive results bias: Positive results bias, also known as publication bias, refers to the system-wide or social phenomenon of observing a skewed level of positive results published in journals, because negative or null results tend to go unpublished. The consequence of this can be the overestimation of efficacy for specific techniques or methods, as well as research duplication since other research teams might attempt to repeat studies that have already been performed but not published. An example of this was observed in the well-known 'reproducibility crisis' that affected the social psychology literature.

Plüddemann et al., 2017)

Biases of rhetoric or spin: Biases of rhetoric or spin occur during the communication of research or development (e.g. model performance) and refer to the use of unjustified or illegitimate forms of persuasive language that lacks meaningful content or substantive evidence. These biases relate to overemphasis of the performance or efficacy of a technique or intervention (e.g. showing comparative preference for the favoured technique to the detriment of alternatives).

(Heneghan et al., 2017)

Cognitive biases

Status quo bias: Status quo bias is an affectively motivated preference for “the way things are currently”, which can prevent more innovative or effective processes or services being implemented. This bias is most acutely felt during the transition period between projects, such as the choice not to deprovision a system and begin a new project, despite deteriorating performance from the existing solution. Although this bias is often treated as a cognitive bias, we also highlight it here as a social bias to draw attention to the broader social or institutional factors that in part determine the status quo.

Confirmation bias: Confirmation bias arises from the tendency to search for, gather, or use information that confirms pre-existing ideas and beliefs, and to dismiss or downplay the significance of information that disconfirms one’s favoured hypothesis. This can be the result of motivated reasoning or sub-conscious attitudes, which in turn may lead to prejudicial judgements that are not based on reasoned evidence. For example, confirmation biases could surface in the judgment of the user of an AI decision-support application, who believes in following common sense intuitions acquired through professional experience rather than the outputs of an algorithmic model and, for this reason, dismisses its recommendations regardless of their rational persuasiveness or veracity.

Self-assessment bias: A tendency to evaluate one’s abilities in more favourable terms than others, or to be more critical of others than oneself. In the context of a project team, this could include the overly positive assessment the group’s abilities (e.g. through reinforcing groupthink). For instance, during project planning, a project team may believe that their resources and capabilities are sufficient for the objective of the project, but in fact be forced to either cut corners or deliver a sub-par product.

Availability bias: The tendency to make judgements or decisions based on the information that is most readily available (e.g. more easily recalled). When this information is recalled on multiple occasions, the bias can be reinforced through repetition—known as a 'cascade'. This bias can cause issues for project teams throughout the project lifecycle where decisions are influenced by available or oft-repeated information (e.g. hypothesis testing during data analysis).

Naïve realism: A disposition to perceive the world in objective terms that can inhibit recognition of socially constructed categories. For instance, treating ‘employability’ something that is objectively measurable and, therefore, able to be predicted by a machine learning algorithm based on objective factors (e.g. exam grades, educational attainment).

Bias Self-Reflection and Risk Mitigation Across the Workflow The Bias self-reflection and risk mitigation template is to be initially completed during the Project Planning step of the AI lifecycle, informed by the PS Report and the list of biases provided. It is to be iteratively revisited and updated as a part of completing your Stakeholder Impact Assessment, each time identifying and responding to emerging biases through mitigation activities, and providing up-to-date documentation on the project’s management of non-discrimination. Fairness Position Statement During the model Training, Testing, and Validation step, you and your project team are to establish a fairness metric for your system by thoroughly considering the use case appropriateness as well as technical feasibility of the fairness metrics most relevant to your system. You should prepare a Fairness Position Statement (FPS), in which the metric-based fairness criteria being employed in the model is made explicit and explained in plain and nontechnical language. This FPS should then be made publicly available for review by all affected stakeholders.

  1. Some of the biases in this list are specific to health care systems, bust most are widely applicable. They are taken from CITE. ↩


Accountability in AI

Introduction to accountability

What does accountability mean? It means that humans are answerable for the parts they play across the entire AI design, development, and deployment. It also demands that the results of this work are traceable from start to finish.

According to the principle of fairness, designers and implementers are held accountable for being equitable and for not harming anyone through bias or discrimination. According to the principle of sustainability, designers and implementers are held accountable for producing AI innovation that is safe and ethical in its outcomes and wider impacts. And according to the principle of explainability, designers and implementers are held accountable for making sure that any decision the AI system makes, or provides support for humans to make, can be adequately explained to relevant stakeholders. Therefore, the principle of accountability is an end-to-end governing principle.

Responsible AI project delivery requires confronting two relevant challenges to accountability.

Accountability gap

Automated decisions are not self-justifiable. Whereas human agents can be called to account for their judgments and decisions in instances where those judgments and decisions affect the interests of others, the statistical models and underlying hardware that compose AI systems are not responsible in the same morally relevant sense. This creates an accountability gap that must be addressed so that clear and imputable sources of human answerability can be attached to decisions assisted or produced by an AI system.

Complexity of AI production processes

Establishing human answerability is not a simple matter when it comes to the design, development, and deployment of AI systems. This is due to the complexity and multi-agent character of the production and use of these systems. Typically, AI project delivery workflows include department and delivery leads, technical experts, data procurement and preparation personnel, policy and domain experts, implementers, and others. Due to this production complexity, it may become difficult to answer the question of who among these parties involved in the production of AI systems should bear responsibility if these systems’ uses have negative consequences and impacts.

Meeting the special requirements of accountability, which are born out of these two challenges, calls for a sufficiently fine-grained concept of what would make an AI project properly accountable. This concept can be broken down into two sub-components of accountability, answerability and auditability:

Answerability

The principle of accountability demands that the onus of justifying algorithmically supported decisions be placed on the shoulders of the human creators and users of those AI systems. This means that it is essential to establish a continuous chain of human responsibility across the whole AI project delivery workflow. Making sure that accountability is effective from end to end necessitates that no gaps be permitted in the answerability of responsible human authorities from first steps of the design of an AI system to its algorithmically steered outcomes.

Answerability also demands that explanations and justifications of both the rationale underlying the results of an AI system and the processes behind their production and use be offered by competent human authorities in plain, understandable, and coherent language. These explanations and justifications should be based upon sincere, consistent, sound, and impartial reasons that are accessible to non-technical hearers.

Auditability

Whereas the notion of answerability responds to the question of who is accountable for an automation supported outcome, the notion of auditability answers the question of how the designers and implementers of AI systems are to be held accountable. This aspect of accountability has to do with demonstrating both the responsibility of design, development, and deployment practices and the justifiability of outcomes.

Auditability also has to do with traceability; It refers to the process by which all stages of the AI innovation lifecycle from data collection and model selection to system deployment, updating, and deprovisioning are documented in a way that is accessible and easily understood.

The project team must ensure that every step of the process of designing and implementing an AI project is accessible for audit, oversight, and review by appropriate parties. Successful auditability requires builders and implementers of algorithmic systems to:

  • keep records and to make available information that enables monitoring of the soundness and diligence of the innovation processes that produced these systems

  • keep track of the accountable parties within an organisation’s project team and others involved in the supply chain (where system components are procured)

  • keep track of the governance actions taken across the entire AI innovation workflow

  • keep records and make accessible information that enables monitoring of data provenance and analysis from the stages of collection, pre-processing, and modelling to training, testing, and deploying. This is the purpose of the Dataset Factsheet.

Moreover, auditability requires the team to enable peers and overseers to probe and to critically review the dynamic operation of the system in order to ensure that the procedures and operations which are producing the model’s behaviour are safe, ethical, and fair. Practically transparent algorithmic models must be built for auditability, reproducible, and equipped for end-to-end recording and monitoring of their data processing.The deliberate incorporation of both of these elements of accountability (answerability and auditability) into the AI project lifecycle may be called accountability-by-design.

Key Concept: Accountability-by-design

AI systems must be designed to facilitate end-to-end answerability and auditability. This requires both responsible humans-in-the-loop across the entire design and implementation chain as well as activity monitoring protocols that enable end-to-end oversight and review.

Types of accountability

Accountability deserves consideration across the entire design and implementation workflow. As a best practice, the team should actively consider the different demands that accountability-by-design requires before and after the roll out of the AI project. We will refer to the process of ensuring accountability during the design and development stages of the AI project as ‘anticipatory accountability’. This is because the team is anticipating the project’s accountability needs prior to it being completed.

Following a similar logic, we will refer to the process of addressing accountability after the start of the deployment of your AI project as ‘remedial accountability’. This is because after the initial implementation of a system, the team is remedying any of the issues that may be raised by its effects and potential externalities. These two subtypes of accountability are sometimes referred to as ex-ante (or before-the-event) accountability and ex-post (after-the-event) accountability respectively.

Anticipatory accountability

Treating accountability as an anticipatory principle entails that the project team takes as of primary importance the decisions made and actions taken by them prior to the outcome of an algorithmically supported decision process. This kind of ex ante accountability should be prioritised over remedial accountability, which focuses instead on the corrective or justificatory measures that can be taken after that automation supported process had been completed.

Ensuring that the AI project delivery processes are accountable prior to the actual application of the system in the world will bolster the soundness of design and implementation processes and thereby more effectively preempt possible harms to individual wellbeing and public welfare or other adverse impacts.

Likewise, by establishing strong regimes of anticipatory accountability and by making the design and delivery process as open and publicly accessible as possible, the project team will put affected stakeholders in a position to make better informed and more knowledgeable decisions about their involvement with these systems in advance of potentially harmful impacts. Doing so will also strengthen the public narrative and help to safeguard the project from reputational harm.

Example

During the preprocessing phase of an AI production lifecycle, technical members of a project team are deciding which features to include, and which to leave out. To safeguard sufficient anticipatory accountability, they make sure to log which team members are involved in making these decisions and record the rationale behind the choices made.

Remedial Accountability

While remedial accountability should be seen, along these lines, as a necessary fallback rather than as a first resort for imputing responsibility in the design, development and deployment of AI systems, strong regimes of remedial accountability are no less important in providing necessary justifications for the bearing these systems have on the lives of affected stakeholders.

Putting in place comprehensive auditability regimes as part of your accountability framework and establishing transparent design and use practices, which are methodically logged throughout the AI project delivery lifecycle, are essential components for this sort of remedial accountability.

One aspect of remedial accountability that you must pay close attention to is the need to provide explanations to affected stakeholders for algorithmically supported decisions.

Offering explanations for the results of algorithmically supported decision-making involves furnishing decision subjects and other interested parties with an understandable account of the rationale behind the specific outcome of interest. It also involves furnishing the decision subject and other interested parties with an explanation of the ethical permissibility, the fairness, and the safety of the use of the AI system.

Example

After receiving an unfavourable decision in their recruitment process, a job applicant seeks assurance and justification that the resume filtering AI system used in their process was not biased or discriminatory. To safeguard sufficient remedial accountability, the implementers of the system draw on the logged records of bias-mitigation measures and fairness-aware design practices to demonstrate the fair and non-discriminatory practices behind the production of the system. They also provide an explanation of the rationale behind the applicant’s negative result, showing that the determinative factors behind the system’s output were fair and reasonable.


Governance

Putting accountability into practice

Now that we have explored some of the main aspects of the concept of accountability, we are ready to examine–in greater detail–how elements of answerability and auditability can be put into practice.

The central importance of the end-to-end operability of good governance practices should guide your strategy to embed accountability across the project workflow. Three components are essential to creating a such a workflow:

  1. Maintaining strong regimes of professional and institutional transparency.

  2. Establishing and maintaining a clear and accessible Process-Based Governance Framework (PBG Framework).

  3. Establishing a well-defined auditability trail for your PBG Framework through robust activity logging protocols that are consolidated digitally in a Process Log.

Maintaining professional and institutional transparency

At every stage of the design and implementation of your AI project, team members should be held to rigorous standards of conduct that secure and maintain professionalism and institutional transparency. These standards should include the core values of selflessness, integrity, honesty, accountability, openness, sincerity, neutrality, objectivity, impartiality, and leadership.

All professionals involved in the research, development, production, and implementation of AI technologies are, first and foremost, acting as fiduciaries of the public interest and must, in keeping with these core values of the Civil Service, put the obligations to serve that interest above any other concerns. For helpful guidance on professionalism and good conduct for public office holders, see the Seven Principles of Public Life (also known as the Nolan Principles).

Furthermore, from start to finish of the AI project lifecycle, the design, development, and deployment process should be as transparent and as open to public scrutiny as possible with restrictions on accessibility to relevant information limited to the reasonable protection of justified confidentiality and of analytics that may tip off bad actors to methods of gaming the system of service provision or otherwise taking advantage of their insight to the detriment of the system's performance or the rest of the users.

Process-based governance framework

We have looked at some of the most important values and principles necessary for establishing responsible innovation practices in the AI project lifecycle.

Perhaps the most vital of these measures is the effective operationalisation of these practices.The recently-adopted standard, ISO 37000, defines governance as ‘the system by which the whole organisation is directed, controlled, and held accountable to achieve its core purpose in the long run’.

Establishing a diligent and well-conceived governance framework that covers the entire design, development, and deployment process will provide the foundation for effectively establishing needed practical actions and controls, exhaustively distributing roles and responsibilities, and operationalising answerability and auditability throughout the AI lifecycle. Organising all of the governance actions into a PBG Framework is a way to better accomplish this task.

The purpose of a PBG Framework is to provide a template for the integrations of the norms, values, and principles, which motivate and steer responsible innovation, with the actual processes that characterise the AI design and development pipeline. Establishing a PBG framework creates the baseline conditions for ensuring that the goal of instituting an AI innovation process that is accountable-by-design is achieved.

A PBG Framework should give the team a landscape view of the governance actions that are organising the control structures of the project workflow. Constructing a good PBG Framework will provide the team with a big picture of:

• The relevant stages of the workflow in which actions are necessary to meet governance goals

• The relevant team members and roles involved in each governance action

• Explicit time frames for any necessary follow-up actions, re-assessments, and continual monitoring

• Clear and well-defined protocols for logging activity and for instituting mechanisms to assure end-to-end auditability and appropriate documentation

The PBG framework asks that teams not only outline the governance actions established for individual projects, but also roles involved in each action, time frames for follow-up actions, and logging protocols.

Establishing proportional governance actions

Just as with the determination of proportionate stakeholder involvement, the establishment of proportionate governance protocols should involve a preliminary assessment of the potential risks and hazards of the model or tool under consideration. Low-stakes AI applications that are not safety critical, do not directly impact the lives of people, and do not process potentially sensitive social and demographic data may need less proactive governance controls and processes than high-stakes projects.

By completing the Project Summary Report and Stakeholder Impact Assessments, the project team will need to carry out evaluations of the scope of the possible risks that could arise from the project and of the potential hazards it poses to affected individuals and groups. These assessments of the dangers posed to individual wellbeing and public welfare will help formulate proportionate governance actions to be outlined in the PBG framework.

Notwithstanding the importance of the need for this reasonable application of proportionate governance actions, a strong regime of accountability-by-design across the project lifecycle should nonetheless be established. It may be the case that the assessment of potential risks and adverse impacts does not sufficiently anticipate the full range of possible harms. In instances where such unforeseen harms do arise, proper mechanisms of anticipatory accountability and corresponding documentation protocols should already be in place, so that the best practices of the project team are demonstrable.

Here is a summary picture of where all possible governance actions fit across the project workflow:

Each principle can be operationalised through specific processes:

Sustainability

Stakeholder engagement process (SEP)

Process facilitating a contextually informed understanding of the social environment and human factors that may be impacted by, or may impact, individual AI projects, and the uptake of proportionate stakeholder engagement and input throughout the AI lifecycle.

Stakeholder impact assessment (SIA)

Process facilitating the iterative evaluation of the social impact and sustainability of individual AI projects, as well as the corroboration of these potential impacts in dialogue with stakeholders, when appropriate.

Safety

SSA & RA (Safety Self-Assessment and Risk Management)

Process facilitating the evaluation of how AI projects align with safety objectives through the iterative identification and documentation of risks of potential safety risks across the lifecycle, and assurance actions implemented to address these.

Responsible data management

Data Factsheet

Live document facilitating the uptake of best practices for Responsible Data Management and Stewardship across the AI project workflow by facilitating the documentation of a comprehensive record of the data lineage iterative assessments of data integrity, quality, protection, and privacy.

Fairness

BSA & RM (Bias self-assessment & risk management)

Process facilitating the evaluation of how AI projects align with the principle of fairness through the iterative identification and documentation of risks of bias across the lifecycle, and assurance actions implemented to address these.

Fairness position statement

Document establishing the metric-based fairness criteria for individual AI projects, providing an explanation in plain and nontechnical language.

Accountability

PBG Framework

Live document outlining governance actions, relevant team members and roles involved in each action, time frames for follow-up actions, and logging protocols, for individual AI projects.

Explainability

Transparency & EAM (explainability assurance management)

Iterative process aimed to facilitate the implementation and evaluation of transparency and explainability assurance activities across the project lifecycle and assist in providing clarification of AI system outputs to a range of impacted stakeholders.

Accountability across the workflow

The task of establishing a PBG framework for the project should be initially undertaken in the project planning step of the project alongside the Project Summary Report The results of the Stakeholder Analysis (particularly, the scoping of potential stakeholder impacts) should inform a proportional selection of governance actions within the PBG framework. At this stage, the PBG framework will provide a prospective and provisional plotting of governance actions, roles, and responsibilities for the project. This preliminary outline of governance structures will provide the necessary information for answering the Governance Framework Reflection questions within the PS Report.

In the PS Report, the task of reflecting on your governance framework on your involves answering the following questions:

  1. Do established governance actions proportionally mitigate possible harms to stakeholders posed by this project? If not, how can your PBG framework be rectified to address these potential harms?

  2. Does this distribution of responsibilities outlined in the PBG Framework establish a continuous chain of human accountability throughout the design, development, and deployment of this project? If not, how can any identified gaps or breaks in the chain be rectified in the PBG Framework?

  3. How will you ensure that all team members, who are assigned roles/responsibilities understand the roles/responsibilities that have been assigned to them?

  4. If you are procuring parts or elements of the system from third-party vendors, suppliers, sub-contractors, or external developers, how are you instituting appropriate governance controls that will establish end-to-end accountability, traceability, and auditability for these procured parts or elements?

  5. If any data being used in the production of the AI system will be acquired from a vendor, supplier, or third party, how are you instituting appropriate governance controls that will establish end-to-end accountability, answerability, and auditability across the data lifecycle?

These questions (alongside the rest of the PS Report) are to be revisited and updated as part of completing each iteration of the Stakeholder Impact Assessment, at each point informing any necessary updates to the project’s governance structure (and PBG framework). The PBG Framework is therefore a live document reflecting a governance structure that responds to the emerging needs across the design, development, and deployment lifecycle. It is to be updated after each revisitation of the PS report to reflect the project’s current governance structure.

The process by which these questions are answered should be as collaborative and inclusive as possible. The aim is to involve all relevant members of the project team (and any other relevant managers, operators, or vendors), so that all people involved in the workflow can share input and come to understand expectations about their roles and responsibilities. Any future revisions or updates of this part of the PS Report should likewise include all affected parties.

← Track C · AI Ethics: all modules


ALEX Institute