Fair and Responsible AI: A Focus on the Ability to Contest

There is great potential for Artificial Intelligence (AI) to enhance decision-making, by making it more accurate, efficient, and scalable than human decision-making [5, 13]. To harness these benefits, AI systems should be designed responsibly, to ensure that they are fair, accountable, and transparent [7]. This is particularly important given the increasing use of AI in high-stakes decision-making, including sentencing, hiring, and loan application determination [13].

In response to calls for AI systems to be designed, developed, and deployed responsibly, numerous AI ethics guidelines have been produced. One ‘safeguard’ that is gaining traction within these guidelines is the ability to contest AI decisions (see sidebar for examples) [12]. Article 22(3) of the European Union’s General Data Protection Regulation provides a legal ‘right to contest’ decisions made using solely automated processes. However, none of these documents provide guidance on how AI systems should be designed to enable contestation. In this paper, we outline why design is important, what the design challenges are, and our human-centred approach to designing for contestability.

The importance of designing AI systems to enable ‘contestability’ has been acknowledged in HCI and Algorithmic Accountability literature (e.g. [11, 3]). Using a legal lens, the Algorithmic Accountability work has taken a theoretical approach to proposing requirements of a contestation scheme [4], and design requirements that enable contestation [3]. Within HCI, the focus of contestability research has been on the ability of expert users to work interactively with a system to contest its output [11].

HCI researchers [13, 5, 10] have also drawn on organisational psychology literature to study how the design of AI systems used in decision-making impact human perceptions of procedural fairness, ‘procedural justice’ [19]. The procedural justice literature indicates that having a legitimate way to contest a decision increases a person’s perception of procedural fairness, which can impact their perception of the fairness of the decision (‘distributive justice’), their choice to accept or contest a decision, and their attitude towards the entity making the decision [19, 14]. In line with this literature [14], Lee et al [13] found that having ‘outcome control’, the ability to correct or appeal a decision, in a cooperative group allocation task improved participants’ perceptions of the fairness of the outcome.

The procedural justice literature also indicates that design of a contestation process (not just its availability) impacts perceptions of procedural fairness. For example, having the same decision-maker assess the decision on appeal is seen as less fair than having a new decision-maker [14]. In addition, in a study of content moderation across social media platforms, Myers West [15] found that users were dissatisfied with contestation processes, reporting a lack of clear instruction about how to lodge an appeal, receiving no reply or resolution having lodged a challenge, and no access to human intervention. These findings indicate that the design of a contestation process matters.

To meaningfully challenge a decision, a decision subject requires some form of information in order to understand the decision, decide whether to contest, and to use as grounds for contestation. Many AI systems used in decision-making are effectively “black boxes" [18]; their decision-making processes are hidden due to the use of complex algorithms or techniques (e.g. deep learning) or intentionally by companies to protect trade secrets [6]. This opacity makes it difficult to understand why a decision was made, and consequently, to contest it in any meaningful way. In contrast, with human decision-making a person can generally seek an explanation from the decision maker as to why a decision was made. Often, in high-stakes decisions, reasons must be documented during the decision-making process to mitigate the issue of an inaccurate post-hoc explanation. Promisingly, the field of explainable artificial intelligence (XAI) is progressing work into explainability [20]. To date, XAI has not focused on providing explanations for contestation specifically, which offers a new avenue of research.

A second design challenge is that there are many ways to contest a decision [14]. For example, existing contestation processes for human decisions (e.g. internal review, complaints mechanisms, external review via tribunal or court) could be adapted for decisions made using AI. However, with decisions made at scale, leaving appeal processes to a court to determine would overwhelm an already pressured system. Low perceptions of fairness are associated with procedures that are time consuming, costly and resource intensive [14]. An alternative contestation process might involve a decision subject directly contesting a decision within AI system via an interface. However, the novelty of this approach coupled with a lack of human touch could negatively impact perceptions of fairness. With an abundance of design choices, it is difficult to know where to begin.

We suggest that taking a human-centred approach to explore how people conceptualise contestability in relation to AI systems is a key first step in designing for meaningful contestation. To understand the needs of decision subjects, and the expectations of the community more generally, we are currently conducting a thematic analysis of submissions made to Australia’s ‘Artificial Intelligence: Australia’s Ethics Framework’, a discussion paper that proposed ‘contestability’ as a core ethical principle [16]. The sidebar contains a sample of our preliminary findings.

The increasing use of AI in high-stakes decision-making, which has been deployed without appropriate safeguards like procedural fairness, has had a significant, negative impact on thousands of people, from teachers losing their jobs [2] to the erroneous loss of medical benefits [8]. To reduce negative consequences, AI systems must be responsibly designed, developed, and deployed [7]. Though the ability to contest decisions is not the only mechanism required to ensure that AI systems are ‘fair’, it is a crucial safeguard, and in some circumstances, a legal requirement. How access to contestation, and the contestation process itself, is designed is important given the impact on perceptions of fairness and satisfaction. Yet, there are many design challenges including opacity, and an abundance of design options. A key first step in designing for meaningful contestation is to explore and understand the needs of decision subjects as well as the community more generally.

Henrietta Lyons is supported by the Melbourne School of Engineering Ingenium scholarship program. This research was partly funded by Australian Research Council Discovery Grant DP190103414 Explanation in Artificial Intelligence: A Human-Centred Approach. Eduardo Velloso is the recipient of an Australian Research Council Discovery Early Career Researcher Award (Project Number: DE180100315) funded by the Australian Government.