Fairness in Image Search: A Study of Occupational Stereotyping in Image Retrieval and its Debiasing

Images convey much more information as compared to words. They have a powerful impact on what we recognize and what we remember in the future. At any point in time, they speak louder than words. The web tool Mozcast shows that more than 19% of google searches return images, which means images, rather than texts, are becoming the language of the internet. The growth in visual image search has given rise to a lot of research work in the field of image information retrieval (IIR).

Internet search rankings are known to have a denoting impact on the users’ perceptions and decisions, mainly because most of the search users trust and choose higher-ranked results more than lower-ranked results Epstein & Robertson (2015) and often do not look below the third result Fidel (2012). Surprisingly, even if the high-ranked results are valued the most, there are no standard qualifiers to identify the top results as the most relevant information for a search keyword Mai (2016). Due to the search engine’s proprietary nature, users are unaware of the working of its algorithms. The majority of search engine users consider search engine results to be unbiased and accurate Zickuhr et al. (2012). Highly ranked results not only shape a user’s opinion and impact his beliefs and unconscious bias; they can also affect his search interactions and experiences. In the same context, it is difficult to believe that these results can often be unfair (biased) in terms of gender, language, demography, socio-cultural aspects, and stereotypes. The bias term is very much attached to the search engines for what they index, what they present overall, and what they present to a particular user. This is a deep concern as people are more vulnerable to bias especially when they are unaware of the biases. Hence, the romanticized view of the search engine that the search engine bypasses the structural biases and skewed data does not match the reality at all. The influence carried by the design decisions of the search engines is broad; it does not only impact the perception of individual information seekers, of society at large it influences our cultures and politics by steering peoples’ perspectives towards stereotypical skewed results Robertson et al. (2018).

A google search result in 2016 for the keyword ”three white teenagers” spat out happy and shiny men and women laughing and holding sports equipment. However, the search results for ”three black teenagers” offered an array of mug shots. Google acknowledged this bias and responded that the search algorithms mirror the availability and frequency of the online content.“This means that sometimes unpleasant portrayals of sensitive subject matter online can affect what image search results appear for a given query,” the company said in a statement to the Huffington Post UK. Guarino (2016)

The presentation of black women being sassy and angry presents a disturbing portrait of black womanhood in modern society. In Algorithms of Oppression, Safiya Umoja Noble challenges the claim of Google having equity in all forms of ideas, identities, and activities. She argues that the search algorithms that are privileged towards whiteness and discriminate against people of color, essentially women of color, are due to two main factors i.e., (a) Monopoly of a relatively small number of internet search engines and (b) Private interests in promoting certain aspects of the images, which are typically made available when a cursor hovers on the result.

Image search results are typically displayed in a grid-like structure unlike that of web search results which are arranged as a sequential list. Users can view/scroll results not only in the vertical direction but in the horizontal direction too. These differences in user behavior patterns lead to challenges in evaluating the search results from a user experience standpoint. There are three key differences in Search Engine Result Pages (SERP) of web search and image search. i.e., (1) An image search engine typically places results on a grid-based panel rather than in a one-dimensional ranked list. As a result, users can view results not only vertically but also horizontally. (2) Users can view results by scrolling down without a need to click on the “next page” button because the image search engine does not have an explicit pagination feature. (3) Instead of a snippet, i.e., a query-dependent abstract of the landing page, an image snapshot is shown together with metadata. Xie et al. (2019). These subtle changes in user experience in image search means that users have instant access to more images (i.e., more number of search result outcomes), and because images provide instant access to a large amount of information (as opposed to text/web-pages), tackling biases/unfairness in image search results becomes even more important. The following section enlists some of the biases that are typically observed in an image search. Note that, we use the terms bias and fairness interchangeably, considering that unfairness could result from certain kinds of biases in search algorithms and procedures.

On an unprecedented scale and in many unexpected ways, surprisingly search ranking makes our psychological heuristics and vulnerabilities susceptible Bond et al. (2012). Algorithms trained on biased data reflect the underlying bias. This has led to emerging of datasets designed to evaluate the fairness of algorithms and there have been benchmarks to quantify discrimination imposed by the search algorithms Hardt et al. (2016); Kilbertus et al. (2017). In this regard, the goal of re-ranking search results is to bring more fairness and diversity to the search results without the cost of relevance. However, due to severely imbalanced training datasets, the methods to integrate de-biasing capabilities into these search algorithms still remain largely unsolved. Concerns regarding the power and influence of ranking algorithms are exacerbated by the lack of transparency of the search engine algorithms Pasquale (2015). Due to the proprietary nature of the system and the requirement of high-level technical sophistication to understand the logic makes parameters and processes used by these ranking algorithms opaquePasquale (2015); Gillespie (2014). To overcome these challenges, researchers have developed techniques inspired by social sciences to audit the algorithms to check for potential biases Sandvig et al. (2014). To quantify bias and compute fairness-aware re-ranking results for a search task, the algorithms would seek to achieve the desired distribution of top-ranked results with respect to one or more protected attributes like gender and ageEpstein et al. (2017). This type of framework can be tailored to achieve such as equality of opportunity and demographic parity depending on the choice of the desired distribution Geyik et al. (2019).

According to a study Jain & Varma (2011), there are three limitations for keyword-based image search i.e.,

1. (a)

   There is no straightforward and fully automated way of going from text queries to visual features. In a search process, visual features are mainly used for secondary tasks like finding similar images. Since the search keyword/query is fed as a text, rather than an image to a search engine, the search engines are forced to rely on static and textual features extracted from the image’s parent web page and surrounding texts which might not describe its salient visual information.

2. (b)

   Image rankers are trained on query-image pairs labeled with relevance judgments determined by human experts. Such labels are well known to be noisy due to various factors including ambiguous queries, unknown user intent, and subjectivity in human judgments. This leads to learning a sub-optimal ranker.

3. (c)

   A static ranker is typically built to handle disparate user queries. Therefore, the static ranker is unable to adapt its parameters to suit the query at hand and it might lead to sub-optimal results. In this regard, Jain et al.Jain & Varma (2011) demonstrated that these problems can be mitigated by employing a re-ranking algorithm that leverages aggregate user click-through data.

There are different types of methods for re-ranking keyword-based image search results and those are described in the following subsections.

One way to re-rank the search engine results is through the user click data. For a given query if we can identify images that have been clicked earlier in response to that query, a Gaussian Process (GP) regressor is trained on these images to predict their normalized click counts. Then, this regressor can be used to predict the normalized click counts for the top-ranked 1000 images. The final re-ranking would be done based on a linear combination of the predicted click counts as well as the original ranking scores Epstein & Robertson (2015). This way of modeling tackles re-ranking nicely while still coping with the limitations described earlier.

• •

  The GP regressor is trained on not just textual features but also visual features extracted from the set of previously clicked images. Consequently, images that are visually similar to the clicked images in terms of measured shape, color, and texture properties are automatically ranked high.

• •

  Expert labels might be erroneous, inconsistent with different experts assigning different levels of relevance to the same query-image pair. Such factors bias the training set that results in the learned ranker being sub-optimal. The click-based re-ranker provides an alternative by tackling this problem directly. The hypothesis is that, for a given query, most of the previously clicked images are highly relevant and hence should be leveraged to mitigate the inaccuracies of the baseline ranker.

• •

  As the GP regressor is trained afresh on each incoming query, it is free to tailor its parameters to suit a given query at hand. For example, images named TajMahal.jpg are extremely likely to be of the Taj Mahal. For landmark queries, the query-image file-name match feature is important. But this feature may be uninformative for city queries. For example - Images of Delhi’s tourist attractions are sometimes named delhi.jpg. Also, people’s photographs during their trip to Delhi may be named delhi.jpg. In this case, a single static ranker would be inadequate. However, the GP regressor aims to learn this directly from the click training data and weights this feature differently in these two situations.

Here the key assumption is that ”For a given query the clicked images are highly relevant”. It would work for an image search due to an obvious reason. For a normal textual web search, only a two-line snippet for each document is displayed in the search results. So, the clicked documents might not be relevant to the keyword. The relevance can only be determined if the user goes or does not go through the document. However, in the case of an image search, most search results are thumbnails allowing users to see the entire image before clicking on it. Therefore, the user predominantly tends to click on the relevant images and most likely discards distracting imagesJain & Varma (2011).

The users’ clicks on web search results are one of the key signals for evaluating and improving web search quality, hence is widely used in state-of-the-art Learning-To-Rank(LTR) algorithms. However, this has a drawback from the perspective of fairness of the ranked results. These algorithms can’t justify the scenario when a search result is not necessarily clicked as it is not relevant. Rather it is not chosen because of the lower rank assigned to it on the SERP. If this kind of bias in the users’ click log data is incorporated into any LTR ranking model, the underlying bias would be propagated to the model. In this regard, a reinforcement learning model for re-ranking seems to be very effective and can avoid the proliferation of position bias in the search results.

According to Lee et al. [30], understanding various causes of biases is the first step for adopting effective algorithmic hygiene. But it is a challenging task to assess the search results for bias. Even when flaws in the training data are corrected, the results may still be problematic because context matters during the bias detection phase. When detecting bias, computer programmers generally examine the set of outputs that the algorithm produces to check for anomalous results. However, the downside of these approaches is that not all unequal outcomes are unfair; even error rates are not a simple litmus test for biased algorithms. In this regard, below are some of the evaluation protocols and metric formulations.

• •

  Conducting quantitative experimental studies on bias and unfairness.

• •

  Defining objective metrics that consider fairness and/or bias.

• •

  Formulating bias-aware protocols to evaluate existing algorithms.

• •

  Evaluating existing strategies in unexplored domains.

Automatic gender detection from images deals with classifying images into gender categories such as male, female, both and uncertain. Of late, gender identification has become relevant to an increasing amount of applications related to facial recognition and analysis, especially since the rise of social media. Gender detection typically relies on spatial features related to the human body, specifically the face, which needs identification and alignment of facial elements in the image. The problem becomes harder when the environment becomes unconstrained, such as in the case of web-search retrieved images. This is primarily due to variations in poses, illuminations, occlusions, and interference of other objects with the facial region of interest. Figure 6 show this through top-5 search results obtained from a popular search engine for the keyword “biologist”. As it can be seen, the result images can be diverse and very different from frontal face images typically used by applications requiring gender identification.

In the following section, we summarize some of the existing works on this problem and enlist a few off-the-shelf methods that can be tried for fairness assessment related to gender.

With the best open-source APIs we got the maximum accuracy of 83% and the average accuracy score for all occupations around 60%. We also considered one proprietary system for gender detection i.e., Amazon Rekognition for detecting the gender of the faces in an image. Amazon Rekognition is based on a scalable, deep learning-based architecture for analyzing billions of images and videos with higher accuracy than its open-source counterparts. Amazon Rekognition includes a simple, easy-to-use API that can quickly analyze any image or video file that’s stored in Amazon S3. For our project, we recorded the “FaceDetails” response for an image and retrieved the “Gender” attribute for the faces detected in the given image. With Amazon Rekognition, the average accuracy for gender detection is 79.49% and the maximum accuracy score is for the query term “truck driver” which is 93.94%. So, we considered this proprietary system for our further analysis and implementation of our de-biasing algorithm (Section 7). However, we will also make efforts to increase the accuracy of gender detection by open-source APIs in our future work.

If we consider the top 30 Google search results for all occupations and the gender distribution across these, we get the statistics presented in Figure 11. Please note that in this case, we have ignored images for which Rekognition detects “no face” or “both male and female”.

From the plot, we can see that the distribution of males and females is somewhat balanced for some occupations like biologists and computer programmers. However, for some of the occupations like nurse and primary school teacher, the distribution is skewed towards females to a great extent. This shows a bias in image search results for these occupations.

This sums up our exploration and implementation of frameworks for assessing occupational stereotyping in image search. While we focused on only one vital aspect of occupational stereotyping i.e., gender estimation, we believe that the underlying frameworks and architecture can be extended to other measures of stereotypes such as race and ethnicity. In the following section, we discuss our implementation of image re-ranking techniques that aim to produce a fairer ranked set of images for a given query (than vanilla search engine outputs), while preserving the relevance of the images with respect to the input query.

Ranking reflects search engines’ estimated relevance of Web pages (or in our case, images) to the query. However, each search engine keeps its underlying ranking algorithms secret. Search engines vary by underlying ranking implementation, display of the ranked search results, and showing related pages. Proprietary systems like Google may consider many factors (some of them are personalized to a specific user) along with relevance and also they may be tuned in many ways. A study’s Zhao (2004) findings suggest that a higher rank in a Google retrieval requires a combination of factors like the Google Page Rank, the popularity of websites, the density of keywords on the home page, and the keywords in the URL. However, with the secret ranking algorithm and rapid and frequent changes in these algorithms, it is impossible to have an authoritative description of these ranking algorithms. To make the search results fairer, one may argue that, the search engines and their underlying ranking algorithms can be taught to provide fairer rankings while not compromising on relevance. This is, however, a harder ask for external developers, given the opaque nature of search engines. Another way is to re-rank the retrieved results in a post-hoc manner, to optimize fairness and relevance together. Post-hoc re-ranking not only makes the re-ranking framework agnostic to the underlying search and ranking procedure, but it also is way more scalable, transparent, and controllable.

We propose a fairness ranking algorithm for images incorporating both relevance and fairness with respect to the gender distribution for the given search query. The objective is to assign a higher rank to an image from a lot that maximizes a defined relevance score (or equivalently minimizes the relevance cost) while ensuring fairness. For relevance cost measurement, we propose a scheme depicted in Figure 12. For a given query (say “biologist”) and a retrieved image, we first extract a set of object labels from the image. We use an off-the-shelf system such as Amazon Rekognition (“detect_label” handle) for object detection. Once, object terms are identified, we extract their word embedding representation using GloVe embeddings Pennington et al. (2014). We use the “glove-wiki-gigaword-300-binary” pretrained model. GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training of GloVe is performed on aggregated global word-to-word co-occurrence statistics from a large corpus, and the resulting representations showcase interesting linear substructures of the word vector space. Once embeddings are extracted, they are averaged and we compute the cosine distance between the averaged object-term vector and the vector that represents the query word. Intuitively, the distance indicates how dissimilar the set of objects is with the query. The more the dissimilarity, more becomes the relevance cost.

Our re-ranking method is given as follows. Let us assume that for a given query $q$ a set of images $I^{\prime}$ are already retrieved using an image search engine, from a large indexed image corpus $I$. Our intention is to re-rank $I^{\prime}$ and form a re-ranked list $R$. We initialize $R$ with an empty list and gradually move an image $i$ from $I^{\prime}$. At any given time, the idea is to select an image $i$ in such a way that it minimizes the overall cost of adding it to $R$. The overall cost is given below:

where, $w_{r}$ and $w_{g}$ are user-defined weights, that control how much each term in the above equation, contributes to the overall cost of adding an image ($w_{r}+w_{g}=1$). $w2v(.)$ and $cosine\_dist(.)$ are functions to compute average word embeddings of given terms and cosine distance between two embeddings (vectors). $objects_{i}$ represents objects identified from image $i$. $p(g,.)$ represents the distribution of group property (in our case, gender) in a given set of observations. $KL(.)$ is the Kullback-Leibler divergence between two distributions. The rationale behind this formulation is similar to that of Feng et al. Feng et al. (2020; 2021).

We now describe our experimental setup, datasets and implementation and evaluation details. The implementation can be checked out from https://github.com/swagatikadash010/Image_Search_Bias.

The overall results are plotted in Figure 13. As expected, the random baseline does not re-rank very well and a trade-off between relevance and fairness is seen. There is no clear winner which optimizes both aspects the best and models can be selected based on how sensitive these two aspects can become for specific search applications.

While, for most of the occupations, we get similar plots as the one shown above, queries like “CEO” and “Nurse” yielded different observations. Figure 15 presents performance plots for these two keywords. For “CEO”, the overall relevance score is very low even for the relevance-only baseline. This is because, for CEO the extracted labels are not semantically much relevant to the keyword “Chief Executive Officer”. The minimum relevance score is 0.6974 for all 100 images. For “Nurse” the gender distribution of the retrieved corpus is very highly biased towards Females (around 72%). Biases of such kind, may not have helped the KL divergence term to yield meaningful indications of fairness cost.

Figure 17 presents some anecdotal examples where we qualitatively compare Google-ranked images, images ranked by relevance-only baseline, and by our fairness-aware algorithm. The query keyword here is “Engineer”. As expected we see that while the relevance-only model provides images that are quite relevant for the query term engineer, they are quite male-dominated. This is mitigated by our fairness-aware algorithm (here we have set $w_{r}=w_{g}=0.5$) which distributes the images better across different genders.

• •

  The maximum average accuracy for gender detection for the open-source and the proprietary system is 59.1% and 79.49% respectively. This lower accuracy is primarily due to the open-ended nature of the images and the absence of prominent facial features. In the future, we will explore models that consider the body and environmental features to classify genders better.

• •

  For measuring relevance, we extracted the labels using Amazon Rekognition API, and using word embedding we calculated the semantic similarity between the extracted labels with the occupation keyword. In the future, we will consider joint language and vision models like VisualBERTLi et al. (2019) to compute the relevance scores.

• •

  For de-biasing, we only considered a cost-based re-ranking algorithm that does not improve the search over time. We can use this cost to optimize the ranking procedure itself with the help of reinforcement learning.

• •

  This study only considered the search results from Google. We will take image search results from other popular search engines and open-source search frameworks for our experiments.

• •

  This study only included occupational stereotypes and it is aligned with a few types of biases(as described in the section 3). Mitigating other forms of biases (such as racial bias) is also on our agenda.