ALEX Institute / AI / AI Ethics · Module 7
Track C · AI Ethics, module 7. Turing Commons · about 1.1 h reading (13,359 words).
Chapter Summary
The last chapter of the course is divided into two sections. First, we will delve into the concepts of transparency and explainability in AI, looking at the difference between the two, and the various types of AI explanations.
In the second half of the chapter we will bring everything back together with a review of the CARE & ACT principles. These principles serve as a practical tool to ensure AI systems are developed in an ethical and responsible manner. They are the following:
Learning Objectives
Anticipating impacts of an AI system involves reflecting on and assessing the potential short-term and long-term effects the system may have on impacted individuals and on affected communities and social groups, more broadly.
Why is this kind of anticipatory reflection important? Its purpose is to safeguard the sustainability of AI projects across the entire project lifecycle instead of taking an approach of dealing with issues as they appear. There is no guarantee that a team will be able to anticipate all potential impacts, but dealing with the most relevant ones before they become a problem ensures more sustainable systems overall (it is also a much more efficient use of resources over time).
How does one ensure that the activities and outputs of the AI system are socially and environmentally sustainable? Project team members must proceed with a continuous responsiveness to the real-world impacts that their system could have.
The way to translate into practice as we have seen, is through concerted and stakeholder-involving exploration of the possible adverse and beneficial effects that could otherwise remain hidden from view if deliberate and structured processes for anticipating downstream impacts were not in place.
Attending to sustainability, along these lines, also entails the iterative re-visitation and re-evaluation of impact assessments. To be sure, in its general usage, the word “sustainability” refers to the maintenance of and care for an object or endeavour over time. In the context of AI, this implies that building sustainability into a project is not a “one-off” affair.
Rather, carrying out an initial impact assessment at the inception of a project is only a first, albeit critical, step in a much longer, end-to-end process of responsive re-evaluation and re-assessment. Such an iterative approach ensures that continuous attention is payed both to the dynamic and changing character of the project lifecycle and to the shifting conditions of the real-world environments in which studies are embedded.
Methodical impact evaluation should involve an initial adoption of normative criteria that function as metrics for scoping and assessing the possible harms and benefits of the research and its outputs. Taking GPAI’s “12 Principles and Priorities of Responsible Data Innovation” as an example, relevant impact assessment questions could include:
12 Principles and Priorities of Responsible Data Innovation
- Respect for and protection of human dignity
- Interconnectivity, solidarity, and intergenerational reciprocity
- Environmental flourishing, sustainability, and the rights of the biosphere
- Protection of human freedom and autonomy
- Prevention of harm and protection of the right to life and physical, psychological, and moral integrity
- Non-discrimination, fairness, and equality
- Rights of Indigenous peoples and Indigenous data sovereignty
- Data protection and the right to respect of private and family life
- Economic and social rights
- Accountability and effective remedy
- Democracy
- Rule of law
How could our research and its outputs advance each of these twelve principles and priorities or hinder their realisation?
Are there particular stakeholder groups who could disproportionately enjoy the benefits of the research and its outputs, or suffer from the potential harms they generate, as these harms and benefits relate to each of the twelve principles and priorities?
If things go wrong in our research or if its outputs (especially tools produced or capacities enabled) are used out-of-the-scope of their intended purpose and function, what harms could be done to stakeholders in relation to each of the twelve principles and priorities?
It is important to note here that stakeholder involvement in the impact assessment process can be a critical safeguard against evaluative blind spots and omissions.
Methodical impact evaluation should also involve an assessment of the severity of potential adverse impacts. This brings clarity to the prioritisation of impact mitigation actions by allowing the severity levels of potential negative effects to be differentiated, elucidated, and refined. As explained in the United Nations Guiding Principles on Business and Human Rights (UNGP), assessing the severity of potential negative impacts on fundamental rights and freedoms involves consideration of their scale, scope, and remediability, where scale is defined as “the gravity or seriousness of the impact,” scope as “how widespread the impact is, or the numbers of people impacted,” and remediability, as the “ability to restore those affected to a situation at least the same as, or equivalent to, their situation before the impact” (UNGP, 2011, Principle 14).
One notable challenge faced by researchers who are assessing the severity of potential adverse impacts is identifying cumulative or aggregate downstream impacts, which can be much more difficult than identifying harms directly or proximately caused by a project.
Discerning these impacts may require additional research and consultation with domain experts and other relevant stakeholders. This difficulty results from the fact that cumulative impacts are often incremental and more difficult to perceive, and they frequently involve complex contexts of multiple actors or projects operating in the same area or sector or affecting the same populations. Some “big picture” questions to reflect on when assessing cumulative or aggregate impacts include:
(Götzmann et al., 2020)Could the project contribute to wider scale adverse impacts when its deployment is coordinated with (or occurs in tandem with) other projects or innovation activity that serve similar functions or purposes? For example, if the impacts of a project that aims to discover an effective method of behavioural nudging at scale are considered in combination with the proliferation of many other similar projects or computational systems in a given sector, concerns about wider cumulative effects like mass manipulation, objectification, and infringement on autonomy and human dignity become relevant.
Could the project replicate, reinforce, or augment socio-historically entrenched legacy harms that create knock-on effects in impacted individuals and groups? For example, if a project analyses sensitive personal information contained in databases scraped from social media websites without gaining the proper consent of research subject in accordance with their reasonable expectations, it could add to the legacy harms of companies that have used data recklessly and eroded public trust regarding the respect of privacy and data protection rights in the digital sphere. This can create wider chilling effects on elements of open communication, information sharing, and interpersonal connection that are essential components for the sustainability of democratic forms of life.
Could the production and use of the system be understood to contribute to wider aggregate adverse impacts on the biosphere and on planetary health when its deployment is considered in combination with other systems that may have similar environmental impacts? For example, a project that involves moderate levels of energy consumption in model training or data storage may be seen to contribute to significant environmental impact when considered alongside the energy consumption of similar projects across research ecosystems.
Once impacts have been evaluated and the severity of any potential harms assessed, impact prevention and mitigation planning should commence. Diligent impact mitigation planning begins with a scoping and prioritization stage. Team members (and engaged stakeholders, where appropriate) should go through all the identified potential adverse impacts and map out the interrelations and interdependencies between them as well as surrounding social factors (such as contextually specific stakeholder vulnerabilities and precariousness) that could make impact mitigation more challenging. Where prioritization of prevention and mitigation actions is necessary (for instance, where delays in addressing a potential harm could reduce its remediability), decision-making should be steered by the relative severity of the impacts under consideration. As a general rule, while impact prevention and mitigation planning may involve prioritization of actions, all potential adverse impacts must be addressed. When potential adverse impacts have been mapped out and organised, and mitigation actions have been considered, the research team (and engaged stakeholders, where appropriate) should begin co-designing an impact mitigation plan (IMP). The IMP will become the part of your transparent reporting methodology that specifies the actions and processes needed to address the adverse impacts which have been identified and that assigns responsibility for the completions of these tasks and processes. As such, the IMP will serve a crucial documenting function.
Establishment of protocols for re-visitation and re-evaluation of the research impact assessment:
Impact assessments must pay continuous attention both to the dynamic and changing character of the project lifecycle and to the shifting conditions of the real-world environments in which research practices, results, and outputs are embedded. There are two sets of factors that should inform when and how often initial impact assessments are re-visited to ensure that they remain adequately responsive to factors that could present new potential harms or significantly influence impacts that have been previously identified:
Lifecycle and production factors: Choices made at any point along the workflow may affect the veracity of prior impact assessments—leading to a need for re-assessment, reconsideration, and amendment. For instance, design choices could be made that were not anticipated in the initial impact assessment (such choices might include adjusting the variables that are included in the model, choosing more complex algorithms, or grouping variables in ways that may impact specific groups). These changes may influence how a computational model performs, how it is explained, or how it impacts affected individuals and groups. Processes are also iterative and frequently bi-directional, and this often results in the need for revision and update. For these reasons, impact assessments must remain agile, attentive to change, and at-the-ready to evaluatively move back and forth across the decision-making pipeline as downstream actions affect upstream choices and evaluations.
Environmental factors: Changes in project-relevant social, regulatory, policy or legal environments (occurring during the time in which the research is taking place) may have a bearing on how well the resulting computational model works and on how the research outputs impact affected individuals and groups. Likewise, domain-level reforms, policy changes, or changes in data recording methods may take place in the population of concern in ways that affect whether the data used to train the model accurately portrays phenomena, populations, or related factors in an accurate manner. In the same vein, cultural or behavioral shifts may occur within affected populations that alter the underlying data distribution and hamper the predictive and explanatory efficacy of a model, which has been trained on data collected prior to such shifts. All of these alterations of environmental conditions can have a significant effect on how research practices, outputs, and results impact affected individual and communities.
Another crucial element of responsible research and innovation in AI supposes that the people who design, develop, and deploy a system engage in reflexive practices that scrutinise the way potential perspectival limitations and power imbalances can exercise influence on the equity and integrity projects and on the motivations, interests, and aims that steer them.
The imperative of reflecting on purposes, positionality, power makes explicit the importance of this dimension of inward-facing reflection. This is a complement to the more outward-facing activities of stakeholder analysis, engagement, and impact assessment.
As we have already discussed, all individual human beings come from unique places, experiences, and life contexts that shape their perspectives, motivations, and purposes. Reflecting on these contextual attributes is important insofar as it can help team members understand how their viewpoints might differ from those around them and, more importantly, from those who have diverging cultural and socioeconomic backgrounds and life experiences.
Identifying and probing these differences enables individual team members to better understand how their own backgrounds, for better or worse, frame the way they see others, the way they approach and solve problems, and the way they carry out research and engage in innovation. By undertaking such efforts to recognise social position and differential privilege, they may gain a greater awareness of their own personal biases and unconscious assumptions. This then can enable them to better discern the origins of these biases and assumptions and to confront and challenge them in turn.
Social scientists have long referred to this site of self-locating reflection as “positionality”.-- When people take their own positionalities into account, and make this explicit, they can better grasp how the influence of their respective social and cultural positions potentially creates research strengths and limitations. On the one hand, one’s positionality—with respect to characteristics like ethnicity, race, age, gender, socioeconomic status, education and training levels, values, geographical background, etc.—can have a positive effect on an individual’s contributions to a project; the uniqueness of each person’s lived experience and standpoint can play a constructive role in introducing insights and understandings that other team members do not have. On the other hand, one’s positionality can assume a harmful role when hidden biases and prejudices that derive from a person’s background, and from differential privileges and power imbalances, creep into decision-making processes undetected and subconsciously sway the purposes, trajectories, and approaches of projects.
When taking positionality into account, team members should reflect on their own positionality matrix. They should ask: To what extent do my personal characteristics, group identifications, socioeconomic status, educational, training, & work background, team composition, & institutional frame represent sources of power and advantage or sources of marginalisation and disadvantage? How does this positionality influence the team's ability to identify & understand affected stakeholders and the potential impacts of the project? Answering these questions involves probing several other areas of self-ascription related to each researcher’s contextual attributes:
A solid grasp on positionality allows team members to better interrogate and reflect on the power dynamics that could unduly influence research purposes and trajectories. Such reflections on power should involve an investigation of how power operates, and where it manifests, both across the project lifecycle and in the real-world environments the project is situated. Leslie et al., propose a series of guiding questions that can be used as a reflective tool to help make potentially noxious power dynamics explicit:
What, if any, power imbalances exist between me (or my team) and the communities impacted by our project?
Do the projects I currently pursue reinforce or challenge these imbalances?
How, if at all, do these imbalances result in unjust exercises of power?
Are my current activities entrenching or combating such exercises of power?
What are my interests (or my team’s interest) in collecting or procuring data and in using these to build models and answer questions?
How, if at all, are these interests similar to or different from the interests of those in the communities that research impacts?
How, if at all, do any power imbalances that exist between me (or my team) and impacted communities influence the pursuit of these interests in my (or my team’s) agendas?
How, if at all, do I (or my team) exploit power imbalances to pursue these interests?
What other actors hold power and influence over the agendas I pursue and the ways I collect or procure data and build and implement models and data applications?
How reliant am I on the data, tools, models, and digital infrastructure (connectivity, computing resources, and data assets) provided by other actors?
What are the interests of these actors? How are they similar to or different from my interests and from those of the members of the communities impacted by my data work?
What, if any, power imbalances exist between these actors and me (and my firm or organisation)?
What is the history of these power imbalances? Are current policies and available resources reinforcing or contesting these imbalances?
How, if at all, do these imbalances result in unjust exercises of power? Are current policies and available resources enabling or combating such exercises of power?
What does the institutional context of my team look like (taking into account the authority structure within my team(s), wider policy-ownership and power hierarchies in my organisation, levels of decision-making autonomy, and opportunities to voice concerns)?
Does this institutional context enable my practices to safeguard the public interest and to ensure that standards and governance regimes in the research ecosystem are working towards just and societally beneficial outcomes?
If reflection on power dynamics, positionality, and the purpose of the project being developed is an inwards-facing process for the AI project team, stakeholder engagement and community involvement represents the outwards-facing side of the coin.
As we have seen, engagement and involvement with the community can bolster a project’s legitimacy, social license, and democratic governance as well as ensure that its outputs will possess an appropriate degree of public accountability and transparency.
A diligent stakeholder engagement process can help teams to:
This process can also safeguard the equity and the contextual accuracy of impact assessments and facilitate appropriate end-to-end processes of transparent project governance by supporting their iterative revisitation and re-evaluation.
"While practices of inward-facing reflection on purposes, positionality, and power can strengthen the reflexivity, objectivity, and reasonableness of CSS research activities (D’Ignazio and Klein, 2020; Haraway, 1988; Harding, 1992, 1995, 2008, 2015), practices of outward-facing stakeholder engagement and community involvement can bolster a research project’s legitimacy, social license, and democratic governance as well as ensure that its outputs will possess an appropriate degree of public accountability and transparency. A diligent stakeholder engagement process can help research teams to identify stakeholder salience, undertake team positionality reflection, and facilitate proportionate community involvement and input throughout the research project workflow. This process can also safeguard the equity and the contextual accuracy of impact assessments and facilitate appropriate end-to-end processes of transparent project governance by supporting their iterative revisitation and re-evaluation. Moreover, community-involving engagement processes can empower the public and the CSS community alike by introducing the transformative agency of “citizen science” into research processes (Albert et al., 2021; Sagarra et al., 2016; Tauginienė et al., 2020).It is important to note, however, that all stakeholder engagement processes can run the risk either of being cosmetic or tokenistic. They can be employed to grant legitimacy to projects without substantially and meaningfully engaging with the impacted communities (i.e., being one-way information flows or nudging exercises that serve as public relations instruments).-
To avoid such hazards of superficiality, team members should shore up a proportionate approach to stakeholder engagement through deliberate and precise goal setting.
Factors that affect stakeholder engagement objective
Assessment of risks of adverse impacts: As we have stressed throughout this course, stakeholder involvement in projects should be proportionate to the scope of their potential risks and hazards.
Assessment of positionality: Stakeholder involvement should address positionality limitations. For instance, in cases where the identity characteristics of team members do not sufficiently reflect or represent significantly impacted groups, stakeholder participation can “fill gaps” in knowledge, domain expertise, and lived experience.
Assessment of project needs: Stakeholder involvement should help team members strengthen their ability to frame questions and to tackle problems. After all, those impacted by the project are the most likely to know what their problems are, and thus, what issues the project should tackle. Team members should explore the optimal means for community members to actively contribute to their practices.
Practical challenges will be encountered when trying to operationalise a stakeholder engagement process. For example, limits on available resources and tight timelines could be at cross-purposes with the degree of stakeholder involvement that is recommended by team-based assessments of research needs, potential hazards, and positionality limitations. Likewise, the chosen degree of appropriate public participation may be unrealistic or out-of-reach given the engagement barriers that arise from constraints on the capacity of vulnerable stakeholder groups to participate, difficulties in reaching marginalised, isolated, or socially excluded groups, and challenges to participation that are presented by digital divides. In these instances, project teams should take a deliberate and reflective approach to deciding on how to balance engagement goals with practical considerations and should, at all events, make explicit the rationale behind their choices and document this.
Regardless of any potential trade-offs, the establishment of clear and explicit stakeholder engagement goals should be prioritised. Relevant questions to pose in establishing these goals include:
The imperative of acting transparently and responsibly is a call to all AI and data science researchers, developers, deployers and users to marshal the habits of responsible research and innovation cultivated in the CARE processes to produce systems that prioritise data stewardship and that are robust, accountable, fair, non-discriminatory, explainable, reproducible, and replicable.
While the mechanisms and procedures which are put in place to ensure that these normative goals are achieved will differ from project to project we can summarise the following priorities that should be incorporated into a team's governance, self-assessment, and reporting practices:
Full documentation of data provenance, lineage, linkage, and sourcing:
This involves keeping track of and documenting both responsible data management practices across the entire project lifecycle, from data extraction or procurement and data analysis, cleaning, and pre-processing to data use, retention, deletion, and updating.--
It also involves demonstrating that the data is ethically sourced, responsibly linked, and legally available for the project's purposes and making explicit measures taken to ensure data quality (source integrity and measurement accuracy, timeliness and recency, relevance, sufficiency of quantity, dataset representativeness), data integrity (attributability, consistency, completeness, contemporaneousness, traceability, and auditability) and FAIR data (findable, accessible, interoperable, and reusable).
Full documentation of privacy, confidentiality, consent, and data protection due diligence.
This involves demonstrating that data has been handled securely and responsibly from beginning to end of the project's lifecycle so that any potential breaches of confidentiality, privacy, and anonymity have been prevented and any risks of re-identification through triangulation and data linkage mitigated.
Regardless of the jurisdictions of data collection and use, the rights and interests of data subjects should always aimed to be optimally protected by adhering to the highest standards of privacy preservation, data protection, and responsible data handling and storage.
Transparent and accountable reporting of processes and results and appropriate publicity of datasets.
This involves that results of model design, development, and deployment should be carried out in a way that enables the interpretability, reproducibility, and replicability of the results.
Research design, analysis, and reporting should be pursued in an interpretability-aware manner that prioritises process transparency, the understandability of models, and the accessibility and explainability of the rationale behind their results.
An end-to-end process for bias self-assessment.
This should cover all research stages as well as all sources of biases that could arise in the data, in the data collection, in the data pre-processing, in the organising, categorising, describing, annotating, structuring of data (text-as-data, in particular), and in research design and execution choices.
No AI system exists in a vacuum. They are all embedded in a wider socio-technical environment which will affect the way the system's deployment functions. Therefore, considering the wider context the system operates in is imperative for responsible research and innovation in AI.
This translates into think diligently about the conditions and circumstances surrounding the system, its operation and its outputs, including the norms, values, and interests that inform the people undertaking the development of the project and that shape and motivate the reasonable expectations of the project's stakeholders.
Some of the questions to bear in mind when considering context are:
How are these norms, values and interests influencing or steering the project and its outputs?
How could they influence the the users’ meaningful consent and expectations of privacy, confidentiality, and anonymity?
How could they shape a project’s reception and impacts across impacted communities?
Considering these questions will ensure reflection within the project team, and will help to anticipate the potential negative impacts the use of an AI system may have.
Considering context also involves taking into account the specific domain(s), geographical location(s), and jurisdiction(s) in which the project is situated and reflecting on the expectations of affected stakeholders in these specific contexts. Some relevant questions are:
How are do the existing institutional norms and rules in a given domain or jurisdiction shape expectations regarding project goals, practices, and outputs?
How do the unique social, cultural, legal, economic, and political environments in which different projects are embedded influence the conditions of data generation, the intentions and behaviours of the research subjects that are captured by extracted data, and the space of possible inferences that data analytics, modelling, and simulation can yield?
In summary
All in all, contextual considerations should, at minimum, track three vectors:
The first involves considering the contextual determinants of the condition of the production of the project (e.g., thinking about the positionality of the team, the expectations of the relevant community of practice, and the external influences on the aims and means of research by funders, collaborators, and providers of data and research infrastructure).
The second involves considering the context of the users of the system (e.g., thinking about subjects’ reasonable expectations of gainful obscurity and ‘privacy in public’ and considering the changing contexts of their communications such as with whom they are interacting, where, how, and what kinds of data are being shared).
The third involves considering the contexts of the social, cultural, legal, economic, and political environments in which different projects are embedded as well as the historical, geographic, sectoral, and jurisdictional specificities that configure such environments (e.g., thinking about the ways different social groups—both within and between cultures—understand and define key values, research variables, and studied concepts differently as well as the ways that these divergent understandings place limitations on what computational approaches to prediction, classification, modelling, and simulation can achieve).
Transparency as a principle of AI ethics differs a bit in meaning from the everyday use of the term. The common dictionary understanding of transparency defines it as either
(1) the quality an object has when one can see clearly through it or
(2) the quality of a situation or process that can be clearly justified and explained because it is open to inspection and free from secrets.
Transparency as a principle of AI ethics encompasses both of these meanings:
On the one hand, transparent AI involves the interpretability of a given AI system, i.e. the ability to know how and why a model performed the way it did in a specific context and therefore to understand the rationale behind its decision or behaviour. This sort of transparency is often referred to by way of the metaphor of ‘opening the black box’ of AI. It involves content clarification and intelligibility or explicability.
On the other hand, transparent AI involves the justifiability of both of the processes that go into its design and implementation and of its outcome. It therefore involves the soundness of the justification of its use. In this more normative meaning, transparent AI is practically justifiable in an unrestricted way if one can demonstrate that both the design and implementation processes that have gone into the particular decision or behaviour of a system and the decision or behaviour itself are sustainable, safe, fair, and driven by responsibly managed data.
The two-pronged definition of transparency as a principle of AI ethics asks that the project team thinks about transparent AI
in terms of the process behind it (the design and implementation practices that lead to an algorithmically supported outcome); and,
in terms of its product (the content and justification of that outcome).
This also means that explanations are provided to impacted stakeholders that demonstrate how the team and all others involved in the development of the system acted responsibly when choosing the processes behind its design and deployment; and make the reasoning behind the outcome of that decision clear.
Process-based explanation of AI systems are about demonstrating that good governance processes and best practices have been followed throughout the design, development and use of the AI system. This entails demonstrating that considerations of sustainability, safety, fairness, and responsible data management were operative end-to-end in the project lifecycle.
For example, if trying to explain the fairness and safety of a particular AI-assisted decision, one component of this explanation will involve establishing that adequate measures across the system’s production and deployment have been taken to ensure that its outcome is fair and safe.
Outcome-based explanations of AI systems are about clarifying the results of a specific decision. They involve explaining the reasoning behind a particular algorithmically generated outcome in plain, easily understandable, and everyday language that is socially meaningful to impacted stakeholders (understandable in terms of the contextual factors and relationships that it implicates).
In addition, an adequate explanation will also need to confirm that the actual outcome of an AI decision meets criteria previously established in the design process to ensure that the AI system is being used in a fair, safe, and ethical way. An explanation to affected stakeholders should also include a demonstration that a specific decision or behaviour of the system is sustainable, safe, fair, and driven by data that has been responsibly managed.
Abstract
What does this explanation help people understand?
It is about the ‘why?’ of an AI decision. It helps people understand the reasons that led to a decision outcome, in an accessible way.
What does this type of explanation include?
How the system performed and behaved to get to that decision outcome.
How the different components in the AI system led it to transform inputs into outputs in a particular way. This will help communicate which features, interactions, and parameters were most significant.
How these technical components of the logic underlying the result can provide supporting evidence for the decision reached.
How this underlying logic can be conveyed as easily understandable reasons to decision recipients.
How do the system’s results apply to the concrete context and life situation of the affected individual.
What rationale explanations might answer:
Will the selected algorithmic model, or set of models, provide a degree of interpretability that corresponds with its impact on affected individuals?
Are the supplementary explanation tools being used to help make the complex system explainable good enough to provide meaningful and accurate information about its underlying logic?
What information goes into rationale explanations
As with the other types of explanation, rationale explanations can be process-based or outcome-based.
Process-based explanations clarify:
How the procedures set up help provide meaningful explanations of the underlying logic of the AI model’s results.
How these procedures are suitable given the model’s particular domain context and its possible impacts on the affected decision recipients and wider society.
How the system’s design and deployment workflow has been set up so that it is appropriately interpretable and explainable, including its data collection and preprocessing, model selection, explanation extraction, and explanation delivery procedures.
Outcome-based explanations provide:
The formal and logical rationale of the AI system – how the system is verified against its formal specifications. In this way, one can verify that the AI system will operate reliably and behave in accordance with its intended functionality.
The technical rationale of the system’s output – how the model’s components (its variables and rules) transform inputs into outputs, so that the role these components play in producing that output is known. By understanding the roles and functions of the individual components, it is possible to identify the features and parameters that significantly influence a particular output.
Translation of the system’s workings – its input and output variables, parameters and so on – into accessible everyday language. This enables those in charge of the AI system to clarify, in plain and understandable terms, what role these factors play in reasoning about the real-world problem that the model is trying to address or solve.
Clarification of how a statistical result is applied to the individual concerned.
What does this explanation help people understand?
It helps people understand ‘who’ is involved in the development and management of the AI model, and ‘who’ to contact for a human review of a decision.
What does this type of explanation include?
Who is accountable at each stage of the AI system’s design and deployment, from defining outcomes for the system at its initial phase of design, through to providing the explanation to the affected individual at the end.
Definitions of the mechanisms by which each of these people will be held accountable, as well as how the design and implementation processes of the AI system have been made traceable and auditable.
What information goes into responsibility explanations
Process-based explanations clarify:
The roles and functions across the organisation that are involved in the various stages of developing and implementing an AI system, including any human involvement in the decision-making. If the system, or parts of it, are procured, then information about the providers or developers involved should be included.
Broadly, what the roles do, why they are important, and where overall responsibility lies for management of the AI model – who is ultimately accountable.
Who is responsible at each step from the design of an AI system through to its implementation to make sure that there is effective accountability throughout.
Outcome-based explanations:
Because a responsibility explanation largely has to do with the governance of the design and implementation of AI systems, it is, in a strict sense, entirely process-based. Even so, there is important information about post-decision procedures that should be provided:
Cover information on how to request a human review of an AI-enabled decision or object to the use of AI, including details on who to contact, and what the next steps will be (e.g., how long it will take, what the human reviewer will take into account, how they will present their own decision and explanation).
Give individuals a way to directly contact the role or team responsible for the review (this does not necessarily have to be a specific individual within the organisation).
What does this explanation help people understand?
Data explanations are about the ‘what’ of AI-assisted decisions. They let people know what data about them were used in a particular AI decision, as well as any other sources of data. Generally, they also help individuals understand more about the data used to train and test the AI model.
What does this type of explanation include?
How the data used to train, test, and validate an AI model was managed and utilised from collection through processing and monitoring.
Which data was used in a particular decision and how.
What information goes into data explanations
Process-based explanations include:
What training/testing/validating data was collected, the sources of that data, and the methods that were used to collect it.
Who took part in choosing the data to be collected or procured and who was involved in its recording or acquisition.
How procured or third-party provided data was vetted.
How data quality was assessed and the steps that were taken to address any quality issues discovered, such as completing or removing data.
What the training/testing/validating split was and how it was determined.
How data pre-processing, labelling, and augmentation supported the interpretability and explainability of the model.
What measures were taken to ensure the data used to train, test, and validate the system was representative, relevant, accurately measured, and generalisable.
How potential bias and discrimination in the dataset have been mitigated.
Outcome-based explanations:
Clarify the input data used for a specific decision, and the sources of that data. This is outcome-based because it refers to the AI system’s result for a particular decision recipient.
In some cases, the output data may also require an explanation, particularly where the decision recipient has been placed in a category which may not be clear to them. For example, in the case of anomaly detection for financial fraud identification, the output might be a distance measure which places them at a certain distance away from other people based on their transaction history. Such a classification may require an explanation.
What does this explanation help people understand?
The fairness explanation is about helping people understand the steps have been taken (and will continue to be taken) to ensure an AI decisions are generally unbiased and equitable. It also gives people an understanding of whether or not they have been treated equitably themselves.
What does this type of explanation include?
An explanation of fairness can relate to several stages of the design, development and deployment of AI systems:
A) Data fairness: The system is trained and tested on properly representative, relevant, accurately measured, and generalisable datasets (note that this dataset fairness component will overlap with data explanation).
This may include showing that the data used is:
as representative as possible of all those affected;
sufficient in terms of its quantity and quality, so it represents the underlying population and the phenomenon being modelled;
assessed and recorded through suitable, reliable and impartial sources of measurement and has been sourced through sound collection methods;
up-to-date and accurately reflects the characteristics of individuals, populations and the phenomena that is being modeled; and
relevant by calling on domain experts to help the team understand, assess and use the most appropriate sources and types of data to serve the project's objectives.
B) Design fairness:
It needs to be appropriately shown the AI model's architectures do not include target variables, features,processes, or analytical structures (correlations, interactions, and inferences) which are unreasonable or unjustifiable.
This may include showing that the following has been done:
Any underlying structural biases that may play a role in translating the objectives into target variables and measurable proxies have been identified. When defining the problem at the start of the AI project, these biases could influence what system designers expect target variables to measure and what they statistically represent.
Biases in the data pre-processing phase have been mitigated by taking into account the sector or organisational context in which the AI system is being operated. When this process is automated or outsourced, one should be able to show that what was done has been reviewed and that oversight was maintained throughout.
Information on the context of the metadata should also be included, so that those coming to the pre-processed data later on have access to the relevant properties when they undertake bias mitigation.
Mitigated bias when the feature space was determined (i.e., when relevant features were selected as input variables for your model). Choices made about grouping or separating and including or excluding features, as well as more general judgements about the comprehensiveness or coarseness of the total set of features, may have consequences for protected groups of people.
Mitigated bias when tuning parameters and setting metrics at the modelling, testing and evaluation stages (i.e., into the trained model). The AI development team should iterate the model and peer review it to help ensure that how they choose to adjust the dials and metrics of the model are in line with your objectives of mitigating bias.
Mitigated bias by watching for hidden proxies for discriminatory features in the trained model, as these may act as influences on a model’s output. Designers should also look into whether the significant correlations and inferences determined by the model’s learning mechanisms are justifiable.
C) Metric-based fairness: This is about making sure that the model does not have discriminatory or inequitable impact on the lives of the people it affects. This may include showing that:
The formal definition(s) of fairness that has been chosen is made explicit, as well as the reason why this decision. Data scientists can apply different formalised fairness criteria to choose how specific groups in a selected set will receive benefits in comparison to others in the same set, or how the accuracy or precision of the model will be distributed among subgroups; and
the method applied in operationalising the formalised fairness criteria, for example, by re-weighting model parameters; embedding trade-offs in a classification procedure; or re-tooling algorithmic results to adjust for outcome preferences.
D) Implementation fairness: The AI system is deployed by users sufficiently trained to implement it responsibly and without bias. This may include showing that implementers of the AI system have been appropriately prepared and trained to:
avoid automation bias (over-relying on the outputs of AI systems) or automation-distrust bias (under-relying on AI system outputs because of a lack of trust in them);
use its results with an active awareness of the specific context in which they are being applied. They should understand the particular circumstances of the individual to which that output is being applied; and
understand the limitations of the system. This includes understanding the statistical uncertainty associated with the result as well as the relevant error rates and performance metrics.
What information goes into fairness explanations
This explanation is about providing people with appropriately simplified and concise information on the considerations, measures and testing you carry out to make sure that your AI system is equitable and that bias has been optimally mitigated. Fairness considerations come into play through the whole lifecycle of an AI model, from inception to deployment, monitoring and review.
Process-based explanations include:
the chosen measures to mitigate risks of bias and discrimination at the data collection, preparation, model design and testing stages;
how these measures were chosen and how managed informational barriers to bias-aware design such as limited access to data about protected or sensitive traits of concern have been managed; and
the results of the initial (and ongoing) fairness testing, self-assessment, and external validation – showing that the chosen fairness measures are deliberately and effectively being integrated into model design. This can be done by showing that different groups of people receive similar outcomes, or that protected characteristics have not played a factor in the results.
Outcome-based explanations include:
details about how the formal fairness criteria were implemented in the case of a particular decision or output;
presentation of the relevant fairness metrics and performance measurements in the delivery interface of your model. This should be geared to a non-technical audience and done in an easily understandable way; and
explanations of how others similar to the individual were treated (i.e., whether they received the same decision outcome as the individual). For example, information generated from counter-factual scenarios could be used to show whether or not someone with similar characteristics, but of a different ethnicity or gender, would receive the same decision outcome as the individual.
What does this explanation help people understand? The safety and performance explanation helps people understand the measures that have been put in place, and the steps that have been taken (and are continuously being taken) to maximise the accuracy, reliability, security and robustness of the decisions the AI model helps make. It can also be used to justify the type of AI system chosen, such as comparisons to other systems or human decision makers.
Key concepts:
**Accuracy:** the proportion of examples for which model generates a correct output. This component may also include other related performance measures such as precision, sensitivity (true positives), and specificity (true negatives). Individuals may want to understand how accurate, precise, and sensitive the output was in their particular case.
**Reliability:** how dependably the AI system does what it was intended to do. If it did not do what it was programmed to carry out, individuals may want to know why, and whether this happened in the process of producing the decision that affected them.
**Security:** the system is able to protect its architecture from unauthorised modification or damage of any of its component parts; the system remains continuously functional and accessible to its authorised users and keeps confidential and private information secure, even under hostile or adversarial conditions.
**Robustness:** the system functions reliably and accurately in practice. Individuals may want to know how well the system works if things go wrong, how this has been anticipated and tested, and how the system has been immunised from adversarial attacks.
What information goes into safety and performance explanations
Process-based explanations include:
For accuracy:
What was done at the data collection stage to ensure that the training data was up-to-date and reflective of the characteristics of the people to whom the results apply.
What kinds of external validation has undertaken to test and confirm your model’s ‘ground truth’.
What the overall accuracy rate of the system was at testing stage.
What is done to monitor this (e.g., measuring for concept drift over time).
For reliability:
How it is measured and what the assurance process behind it is.
Results of the formal verification of the system’s programming specifications, i.e., how encoded requirements have been mathematically verified.
For security: - How it is measured and what the assurance process behind it is, e.g., how limitation have been set on who is able to access the system, when, and how.
For robustness:
How it is measured.
Why the specific measures were chosen.
What the assurance process behind it is, e.g., how the system has been stress-tested to understand how it responds to adversarial intervention, implementer error, or skewed goal-execution by an automated learner (in reinforcement learning applications).
Outcome-based explanations:
While one might not be able to guarantee accuracy at an individual level, one should be able to provide assurance that, at run-time, an AI system operated reliably, securely, and robustly for a specific decision.
In the case of accuracy and the other performance metrics, however the results of cross-validation (training/ testing splits) and any external validation carried out, should be included in the model's delivery interface.
Other relevant information that could be included is: information related to the system’s confusion matrix (the table that provides the range of performance metrics) and ROC curve (receiver operating characteristics)/ AUC (area under the curve). Include guidance for users and affected individuals that makes the meaning of these measurement methods, and specifically the ones that have been chosen, easily accessible and understandable. This should also include a clear representation of the uncertainty of the results (e.g., confidence intervals and error bars).
What does this explanation help people understand?
An impact explanation helps people understand how the effects that an AI decision-support system may have on an individual, i.e., what the outcome of the decision means for them, have been considered.
It is also about helping individuals to understand the broader societal effects that the use of this AI system may have. This can help reassure people that the use of AI will be of benefit. Impact explanations are therefore often well suited to delivery before an AI-assisted decision has been made.
What does this type of explanation include?
Demonstrate that thought has been put into how an AI system will potentially affect individuals and wider society. It is important to clearly show affected individuals the process done to determine these possible impacts.
What information goes into impact explanations
Process-based explanations include:
Showing the considerations given to the AI system’s potential effects, how these considerations were undertaken, and the measures and steps taken to mitigate possible negative impacts on society, and to amplify the positive effects.
Information on the plan in place to monitor and re-assess impacts while the system is deployed should also be included.
Outcome-based explanations:
Although impact explanations are mainly about demonstrating that appropriate forethought has been given into the potential ‘big picture’ effects, it is also important to consider how to help recipients understand the impact of the AI-assisted decisions that specifically affect them. For instance, one might explain the consequences for the individual of the different possible decision outcomes and how, in some cases, changes in their behaviour would have brought about a different outcome with more positive impacts. This use of counterfactual assessment would help decision recipients make changes that could lead to a different outcome in the future or allow them to challenge the decision.
Success
There are a number of tasks both to help in the design and deployment of appropriately transparent and explainable AI systems and to assist in providing clarification of the results these systems produce to a range of impacted stakeholders (from operators, implementers, and auditors to decision recipients). These tasks make up Transparency and Explainability Assurance Management for AI projects, offering a systematic approach to:
Task 1 (Project Planning): Select priority explanations by considering the domain, use case and potential impacts
Understanding the different types of explanation will serve to identify the dimensions of an explanation that decision recipients will find useful. In most cases, explaining AI-assisted decisions involves identifying what is happening in the AI system and who is responsible. That means prioritising the rationale and responsibility explanation types.
The setting and the sector are important in figuring out what kinds of explanation one should be able to provide. Therefore considering the domain context and use case is crucial to prioritise which explanations the team should be prepared to give.
In addition, consider the potential impacts of the particular use of the AI system to determine which other types of explanation should be provided. This will also help in thinking about how much information is required, and how comprehensive it should be.
Choosing what to prioritise is not an exact science. Hopefully the explanations prioritised will coincide with what the majority of the people impacted want to know, but it is unlikely that every individual will have all their questions answered. Having a clear and documented rationale for the explanations prioritised will probably also be useful for your own accountability or auditing purposes.
Task 2 (Data Extraction or Procurement, Data Analysis): Collect and pre-process data in an explanation-aware manner
The data collected and pre-processed before inputting it into the system has an important role to play in the ability to derive each explanation type. Careful labelling and selection of input data can help provide information for your rationale explanation.
Providing details about who is responsible at each stage of data collection and pre-processing is part of being more transparent. This is part of the responsibility explanation (information can be drawn from Workflow Governance Map, covered in the AI accountability section of this course).
Drawing from the dataset factsheet can aid in providing data explanations, including the following information:
Check the data used within the model to ensure it is sufficiently representative of those it is making decisions about. Another issue to consider is whether pre-processing techniques, such as re-weighting, are required. These decisions should be documented in the Bias Self-Assessments, and will help construct the fairness explanation.
Task 3 (Model Selection): Build a system that is able to extract relevant information for a range of explanation types
Deriving the rationale explanation is key to understanding an AI system (as well as complying with parts of the GDPR). It requires looking ‘under the hood’ and helps in gathering the information needed for some of the other explanations, such as safety and fairness.
However, this is a complex task that requires knowing when to use more and less interpretable models and how to understand their outputs. To choose the right AI model for the particular explanation needs, one should think about the domain the system will be working in, and the potential impact of the system.
When selecting a model for an AI project, it is important to consider whether:
To extract explanations from inherently interpretable models, look at the logic of the model’s mapping function by exploring it and its results directly. On the other hand, there are many techniques used to extract explanations from ‘black box’ systems. Make sure that they provide a reliable and accurate representation of the system’s behaviour.
Task 4 (Model Reporting): Translate the rationale of the system’s results into useable and easily understandable reasons
Once the rationale of the underlying logic of the AI model has been extracted, the statistical output needs to be incorporated into the wider decision-making process.
Implementers of the outputs from the AI system will need to recognise the factors that they see as legitimate determinants of the outcome they are considering.
Decision recipients should be able to easily understand how the statistical result has been applied to their particular case.
Task 5: (User Training) Prepare implementers to deploy the AI system
In cases where decisions are not fully automated, implementers need to be meaningfully involved.
This means that they need to be appropriately trained to use the model’s results responsibly and fairly.
Their training should cover: - the basics of how machine learning works; - the limitations of AI and automated decision-support technologies; - the benefits and risks of deploying these systems to assist decision-making, particularly how they help humans come to judgements rather than replacing that judgement; and - how to manage cognitive biases, including both decision-automation bias and automation-distrust bias.
Task 6 (Model Reporting): Consider how to build and present an explanation Gathering together the information gained when implementing Tasks 1-4 is the first step towards building an explanation. This includes reviewing the information and determine how this provides an evidence base for the process-based or outcome-based explanations.
Additionally, it is important to revisit the contextual factors to establish which explanation types should be prioritised.
The way an explanation is presented depends on the way AI-assisted decisions are made, and on how people might expect those responsible for the AI system to deliver explanations without using AI.
Explanations can la 'layered' by proactively providing individuals with the prioritised explanations first and making additional explanations available in further layers. This helps to avoid information (or explanation) overload.
Delivering explanations should be thought of as a conversation, rather than a one-way process. People should be able to discuss a decision with a competent human being.
Providing an explanation at the right time is also important. Proactively engaging with customers by making information available on how the AI system is used and how it aids in making decisions, can increase the trust and awareness.
****** Question, should we include the template on transparency and explainability assurance?