Towards User-Focused Cross-Domain Testing: Disentangling Accessibility, Usability, and Fairness

Following the guidelines for conducting systematic reviews in software engineering [19] and drawing insights from previous tertiary studies [17, 18], we conducted a search for secondary studies specifically focusing on usability testing, accessibility testing, and fairness testing. In this process, we employed a semi-automated approach to identify papers for our review, deviating from the typical structured search string used in systematic reviews. Instead, we conducted a straightforward search using the term systematic review in combination with either usability testing, accessibility testing, or fairness testing. This strategy aimed to rapidly filter out studies that mention these terms but fall outside the scope of our goal. Additionally, to ensure relevance, the studies retrieved in our search were immediately evaluated to determine whether they fell within the scope of our investigation. Our focus was specifically on systematic reviews addressing usability testing, accessibility testing, or fairness testing. We employed a two-step approach to filter the relevant papers to streamline this process. First, we conducted a rapid screening by scanning each paper’s title, abstract, and conclusion to identify those most likely to align with our research focus. Next, we thoroughly reviewed the full text of the remaining papers to confirm their direct relevance to the topics under investigation. The search was conducted across leading scientific databases, including IEEE Xplore, ACM Digital Library, SpringerLink, and Google Scholar. These databases were selected for their reputation, comprehensive coverage, and relevance in providing high-quality research papers on the topic under investigation.

Subsequently, we conducted a manual search for studies presented at several of the world’s leading conferences on software engineering and software testing. These conferences included the International Conference on Software Engineering, the International Conference on Software Testing, the International Conference on Software Maintenance and Evolution, and the International Conference on the Foundations of Software Engineering. Our selection was guided by their reputation as premier venues in the field, rigorous peer-review processes, and focus on advancing state-of-the-art research. These conferences also serve as essential platforms for discussions on topics directly aligned with our research focus, ensuring relevance and credibility in the studies included. Our search focused on publications from the past decade, specifically between 2015 and 2024, recognizing the increasing prominence of discussions on software fairness within software engineering.

For each topic, the search process was conducted separately. We analyzed the retrieved results based on the paper titles and abstracts. In this tertiary study, we included papers that reported systematic reviews or mapping studies focused on one of the three types of testing or had a broader focus that included discussions on testing. We excluded systematic reviews or mapping studies that did not have usability testing, accessibility testing, or fairness testing as the central focus of their findings or discussions.

Excerpts were extracted directly from the studies and recorded in a spreadsheet to facilitate data analysis. Primarily, we collected qualitative data, including definitions, characteristics, and aspects of each type of testing. Consequently, qualitative analysis was employed to thoroughly examine the data and provide material for our discussion. In this process, we applied thematic analysis [21] to comprehensively explore and interpret the collected data.

We explored the definitions presented in the systematic reviews and the characteristics attributed to each type of testing to examine the interplay among the concepts of usability, accessibility, and fairness. Table II presents the various definitions of each concept identified in the papers. As expected, the literature reviews on software fairness are more recent than those on usability and accessibility. Additionally, we found more reviews on usability than on accessibility. One reason for this discrepancy is that, in many instances, these two aspects of software development were investigated together, with the focus shifting between them. This reinforces our motivation to discuss their interplay. Analyzing this evidence, we can emphasize the following:

The data collected from the reviews demonstrate an intricate interplay among usability, accessibility, and fairness in software development. Traditionally, usability and accessibility have been closely linked, particularly through interface elements that facilitate effective and user-friendly interactions for different individuals. Usability testing focuses on user experience and satisfaction, while accessibility testing ensures essential support through interface design tailored to diverse needs. Now, fairness introduces a new dimension by testing for equitable outcomes derived from data and algorithmic decisions.

In our analysis, we observed a strong correlation between fairness testing and accessibility testing, especially because individuals with disabilities often belong to minority groups. Accessibility testing rigorously evaluates features to accommodate diverse impairments, such as visual or speech impairments. In contrast, fairness testing addresses complexities involving intersecting biases affecting various underrepresented groups, including people with disabilities. Furthermore, there is a significant relationship between usability testing and fairness testing. While usability testing aims to enhance user experience during software interaction, fairness testing ensures equitable interaction across different user groups, thereby preventing exclusions or limited experiences based on user background (e.g., gender, sexual orientation, ethnicity).

Based on the evidence analyzed from the literature, it is evident that usability, accessibility, and fairness play intricate and multifaceted roles in software development. Usability focuses on creating effective and enjoyable interfaces for a wide range of users, while accessibility ensures these interfaces are usable by individuals with disabilities. Fairness, on the other hand, strives to achieve unbiased outcomes, interfaces, and interactions across diverse user groups. This interplay demonstrates a potential challenge in integrating fairness testing as a routine practice in the software development process, highlighting the need for the development of strategies, methods, and tools to systematically incorporate fairness testing into the software industry.

Additionally, the relationship between usability, accessibility, and fairness creates complex challenges in identifying and fixing fairness defects. The combined effect of these usability and accessibility problems can divert attention away from fairness, delaying necessary interventions and perpetuating biases within the software. If testers are solely focused on usability problems, they may not uncover the parts of the software where fairness issues exist, masking the urgency to address these defects. Similarly, accessibility bugs, such as inadequate support for assistive technologies or non-compliance with accessibility standards, can hinder software testing from addressing other types of inclusivity issues, making it difficult to identify and prioritize these concerns.

Usability testing and accessibility testing overlap in their commitment to user-centric design. This intersection ensures that interfaces are not only easy to use but also inclusive, accommodating accessibility needs such as support for disabilities and universality. By integrating these aspects, the software can be tested to guarantee that it serves a diverse user base, ensuring usability features are accessible to everyone, including those with impairments.

Another overlap exists between usability testing and fairness testing, emphasizing the importance of verifying and validating an equitable user experience that incorporates ethical considerations into design decisions. Usability focuses on delivering a pleasant and effective user experience, ensuring interfaces are intuitive and enjoyable, while fairness adds another layer by ensuring these interactions are ethical, providing a positive experience for all users regardless of their background or identity.

The overlap between accessibility and fairness shifts the focus of testing to equitable access and inclusion. Accessibility features are instrumental in promoting fairness and equity by ensuring software is universally accessible. This overlap ensures that bias within the software does not lead to discrimination against users based on disabilities or minority status, promoting equal access and opportunities.

At the heart of these relationships lies a central overlap that integrates usability, accessibility, and fairness into a user-focused, cross-domain testing approach (Figure 1). This approach combines principles from all three domains to ensure the development of software that is user-friendly, inclusive, fair, and ethically sound. By adopting this comprehensive approach, the software industry can produce systems that cater to the varied needs of users effectively while upholding principles of fairness and inclusivity in digital environments.

We acknowledge limitations in this study inherent to tertiary research. The primary challenge in this type of research lies in correctly identifying all relevant systematic reviews for analysis. In this regard, we understand that a significant number of systematic reviews might not have been included in our study because our semi-automatic search method was designed with the sole objective of pinpointing recent reviews to assist in the process of reflecting and discussing the intricate relationships among usability testing, accessibility testing, and fairness testing. Despite this limitation, this study represents an essential initial step in shedding light on the blurred lines within these aspects of software testing.