Provable Optimization for Adversarial Fair Self-supervised Contrastive Learning

Self-supervised learning (SSL) has become a pivotal paradigm in deep learning (DL), offering a groundbreaking approach to addressing challenges related to labeled data scarcity. The significance of SSL lies in its ability to leverage vast amounts of unlabeled data to learn encoder networks for extracting meaningful representations from input data that are useful for various downstream tasks. One state-of-the-art method for SSL is contrastive learning by minimizing a contrastive loss that contrasts a positive data pair with a number of negative data pairs. It has shown remarkable success in pretraining encoder networks, e.g., Google’s SimCLR model [6] pretrained on image data and OpenAI’s CLIP model [35] pretrained on image-text data, leading to improved performance when fine-tuned on downstream tasks.

However, like traditional supervised learning, SSL is also not immune to the fairness concern and biases inherent in the data. The potential for self-supervised models to produce unfair outcomes stems from the biases embedded in the unlabeled data used for training. The contrastive loss could inadvertently reinforce certain biases present in the data. For example, a biased feature that is highly relevant to the gender (e.g., long-hair) can be easily learned from the data to contrast a female face image with a number of images that are dominated by male face images. As a result, the learned feature representations will likely to induce unfair predictive models for downstream tasks. One approach to address this issue is to utilize traditional supervised fairness-aware approaches to learn a predictive model in the downstream task by removing the disparate impact of biased features highly relevant to sensitive attributes. However, this approach will suffer from several limitations: (i) it requires repeated efforts for different downstream tasks; (ii) it requires labeled data to be annotated with sensitive attribute as well, which may cause privacy issues.

Several studies have put forward techniques to enhance the fairness of contrastive learning of representations [29, 25, 47, 17]. However, the existing approaches focused on modifying the contrastive loss by restricting the space of positive data and/or the space of negative data of an anchor data by using the sensitive attributes of all data or an external image generator [47]. Different from the prior studies on fair contrastive learning, we revisit a classical idea of advesarial fair representation learning (AFRL) by solving the following minimax problem:

where $F_{\text{GCL}}({\bf{w}})$ denotes a self-supervised global contrastive loss (GCL) of learning an encoder network parameterized by ${\bf{w}}$ and $F_{\text{fair}}({\bf{w}},{\bf{w}}^{\prime})$ denotes an adversarial loss of a discrimnator parameterized by ${\bf{w}}^{\prime}$ for predicting the sensitive attribute given the encoded representation. There are several benefits of this approach compared with existing fairness-aware contrastive learning [30, 17, 25, 47]. First, the contrastive loss remains intact. Hence, it offers more flexibility of choosing different contrastive losses including bimodal contrastive losses [35]. Second, only the fairness-promoting regularizer depends on the sensitive attribute. Therefore, it is not necessary to annotate all unlabeled data with sensitive attribute, which makes it more suitable for SSL with a large number of unlabeled data but only limited data with sensitive attribute.

Despite the simplicity of the framework, a challenging question remains: is it possible to design an efficient algorithm that can be proved to converge for solving (3)? There are two hurdles to be overcome: (i) the problem could be non-convex and non-concave if the discriminator is not a linear model, which makes the convergence analysis formidable; (ii) the standard mini-batch gradient estimator yields a biased stochastic gradient of the primal variable due to presence of GCL, which does not ensure the convergence. Although existing studies in ML have shown that solving a general non-convex minimax game (e.g., generative adversarial networks [GAN]) is not stable [20], it is still possible to develop provable yet efficient algorithms for AFRL because: (i) AFRL can use a simple/shallow network for the discriminator that operates on an encoded representation; hence one step of the dual update of ${\bf{w}}^{\prime}$ followed by a step of the primal update of ${\bf{w}}$ is sufficient; (ii) non-convex minimax game has been proved to converge under weaker structured conditions than concavity. Based these two observations, we design an efficient algorithm and and provide its convergence guarante. Our contributions are outlined below:

• 1.

  Theoretically, we propose a stochastic algorithm for optimizing the non-convex compositional minimax game and manage to establish a convergence result under provable conditions without requiring convexity/concavity on either side.

• 2.

  Empirically, we conduct extensive experiments to demonstrate the effectiveness of the proposed approach for downstream classification with eight fairness notions.

This paper is different from tremendous studies on fair representation learning, which either rely on the labeled data with annotated sensitive attribute, or simply use mini-batch based contrastive loss for SSL that has no convergence guarantee, or only examine few standard fairness measures for downstream classification.

While there are tremendous work on fairness-aware learning [31, 26, 1, 32], we restrict our discussion below to most relevant studies about fair representation learning, which is able to learn a mapping function that induces representations of data for fairer predictions.

(Semi-)Supervised Fair Representation Learning. The seminal work [45] initiated the study of fair representation learning. The goal is to learn a fair mapping that maps the original input feature representation into a space that not only preserves the information of data but also obfuscate any information about membership in the protected group. Nevertheless, their approach is deemed as a shallow approach, which is not suitable for deep learning.

For DL, many prior studies have considerd to learn an encoder network that induces fair representation of data with respect to sensitive attributes. A classical idea is adversarial learning by minimizing the loss of predicting class labels and maximizing the loss of predicting the sensitive attribute given the encoded representations. This has been studied in [10, 11, 40, 38, 3, 12, 46, 36] for different applications or different contexts. For example, [12, 7] tackle the domain adaptation setting, where the encoder network is learned such that it cannot discriminate between the training (source) and test (target) domains. Other approaches have explicitly considered fair classification with respect to some sensitive attribute. Among these studies, [3] raised the challenge of collecting sensitive attribute information for all data and considered a setting only part of the labeled data have a sensitive attribute. In addition to adversarial learning, variational auto-encoder (VAE) based methods have been explored for fair representation learning [23, 13, 27, 16]. These methods can work in unsupervised learning setting where no labels are given or semi-supervised learning setting where only part of the data are labeled. However, they do not consider how to leverage data that are not annotated with attribute information. Moreover, VAE-based representation learning methods lag significantly behind self-supervised contrastive representation learning on complicated tasks in terms of performance [4].

Fair Contrastive Learning. Another category of research that is highly related to our study is fair contrastive learning [30, 17, 25, 47]. These methods usually use the sensitive attribute information to restrict the space of negative and positive data in the contrastive loss. [30] utilized a supervised contrastive loss for representation learning, and modifies the contrastive loss by incorporating sensitive attribute information to encourage samples with the same class but different attributes to be closer together while pushing samples with the same sensitive attribute but different classes further apart. Their approach requires all data to be labeled and annotated with sensitive attribute information.

Several papers have modified the contrastive loss for self-supervised learning [25, 17, 47]. For example, in [25] the authors define a contrastive loss for each group of senstive attribute separately. [17] proposed to incorporate counterfactual fairness by using a counterfactual version of an anchor data as positive in contrastive learning. It is generated by flipping the sensitive attribute (e.g., female to male) using a sample generator (cyclic variational autoencoder), which is learned separately. [47] used a similar idea by constructing a positive sample of an achor data with a different sensitive attribute generated by the image attribute editor, and constructing the negative samples as the views generated from the different images with the same sensitive attribute. To this end, they also need to train an image attribute editor that can genereate a sample with a different sensitive attribute.

[5] proposed a bilevel learning framework in the setting that no sensitive attribute information is available. They used a weighted self-supervised contrastive loss as the lower-level objective for learnig a representation and an averaged top-$K$ classification loss on validation data as the upper objective to learn the weights and the classifier. Table 1 summarizes prior studies and our work.

Notations. Let $\mathcal{D}$ represent an unlabeled set of $n$ images, and let $\mathcal{D}_{a}\subset\mathcal{D}$ denote a subset of $k\ll n$ training images with attribute information. Let ${\bf{x}}\sim\mathcal{D}$ denote a random data uniformly sampled from $\mathcal{D}$. For each $({\bf{x}},a)\in\mathcal{D}_{a}$, $a\in\{1,\ldots,K\}$ denotes the sensitive attribute.

We denote an encoder network by $E_{\bf{w}}(\cdot)$ parameterized by ${\bf{w}}$, and let $E_{{\bf{w}}}({\bf{x}})\in\mathbb{R}^{d}$ represent a normalized output representation of input data ${\bf{x}}$. For simplicity, we omit the parameters and use $E(\cdot)$ to refer to the encoder. $\mathcal{P}$ denotes a set of standard data augmentation operators generating various views of a given image [6], and $\mathcal{A}\sim\mathcal{P}$ is a random data augmentation operator. We use $p(X)$ to denote the probability distribution of a random variable $X$, and use $p(X|Y)$ to represent the conditional distribution of a random variable $X$ given $Y$.

A state-of-the-art method of SSL is to optimize a contrastive loss [33, 6]. A standard approach for defining a contrastive loss of image data is the following mini-batch based contrastive loss for each image ${\bf{x}}_{i}$ and two random augmentation operators $\mathcal{A},\mathcal{A}^{\prime}\sim\mathcal{P}$ [6]:

where $\tau>0$ is called the temperature parameter, $\mathcal{B}\subset\mathcal{D}$ is a random mini-batch and $\mathcal{B}_{i}^{-}=\{\mathcal{A}({\bf{x}}),\mathcal{A}^{\prime}({\bf{x}})|{\bf{x}}\in\mathcal{B}\setminus{\bf{x}}_{i}\}$ denotes the set of all other samples in the mini-batch and their two random augmentations. However, this approach requires a very large size to achieve good performance [6] due to a large optimization error with a small batch size [43].

To address this challenge, Yuan et al. [43] proposed to optimize a global contrastive loss (GCL) based on advanced optimization techniques with rigorous convergence guarantee. We adopte the second variant of GCL defined in their work in order to derive a convergence guaratnee. A GCL for a given sample ${\bf{x}}_{i}$ and two augmentation operators $\mathcal{A},\mathcal{A}^{\prime}\sim\mathcal{P}$ can be defined as:

where $\epsilon_{0}$ is a small constant, $\mathcal{S}_{i}^{-}=\{\mathcal{A}({\bf{x}})|\mathcal{A}\in\mathcal{P},{\bf{x}}\in\mathcal{D}\backslash{{\bf{x}}_{i}}\}$ denotes the set of all data to be contrasted with ${\bf{x}}_{i}$, which can be constructed by including all other images except for ${\bf{x}}_{i}$ and their augmentations. Then, the averaged GCL becomes $F_{\text{GCL}}({\bf{w}})=\mathbb{E}_{{\bf{x}}_{i}\sim\mathcal{D},\mathcal{A},\mathcal{A}^{\prime}\sim\mathcal{P}}L_{\text{GCL}}({\bf{x}}_{i},\mathcal{A},\mathcal{A}^{\prime},\mathcal{D})$. To facilitate the design of stochastic optimization, we cast the above loss into the following form:

where

and $\epsilon_{0}^{\prime}=\epsilon_{0}/|\mathcal{S}_{i}^{-}|$ is a small constant and $c$ is a constant.

Our method is built on a classical idea of adversarial training by solving a minimax zero-sum game [12]. The idea is to maximize the loss of predicting the sensitive attribute by a model based on learned representations while minimizing a certain loss of learning the encoder network. To this end, we define a discriminator $D_{{\bf{w}}^{\prime}}(\textbf{v}):\mathbb{R}^{d}\rightarrow\mathbb{R}^{K}$ parameterized by ${\bf{w}}^{\prime}$ that outputs the probabilities of different values of $a$ for an input data associated with an encoded representation vector $\textbf{v}\in\mathbb{R}^{d}$. For example, a simple choice of $D_{{\bf{w}}^{\prime}}$ could be $[D_{{\bf{w}}^{\prime}}(\textbf{v})]_{k}=\frac{\exp({\bf{w}}_{k}^{\prime\top}\textbf{v})}{\sum_{l=1}^{K}\exp({{\bf{w}}^{\prime}_{l}}^{\top}\textbf{v})}$.

Then, we introduce a fairness-promoting regularizer:

where $\phi({\bf{w}},{\bf{w}}^{\prime};\mathcal{A}({\bf{x}}),a)$ denotes the log-likelihood of the discriminator on predicting the sensitive attribute $a$ of the augmented data $\mathcal{A}({\bf{x}})$ based on $E_{\bf{w}}(\mathcal{A}({\bf{x}}))$, i.e., $\phi({\bf{w}},{\bf{w}}^{\prime};\mathcal{A}({\bf{x}}),a):=\log([D_{{\bf{w}}^{\prime}}(E_{{\bf{w}}}(\mathcal{A}({\bf{x}})))]_{a})$. Thus, the minimax zero-sum game for learning the encoder network and the discriminator is imposed by:

where $\alpha$ is a parameter that controls the trade-off between the GCL and the fairness regularizer. There are several benefits of this approach compared with existing fairness-aware contrastive learning [30, 17, 25, 47]. First, the contrastive loss remains intact. Hence, it offers more flexibility of choosing different contrastive losses or even other losses for SSL and also makes it possible to extend our framework to multi-modal SSL [35]. Second, only the fairness-promoting regularizer depends on the sensitive attribute. Therefore, it is not necessary to annotate all unlabeled data with sensitive attribute, which makes it more suitable for SSL with a large number of unlabeled data.

Next, we present a theoretical justification of the minimax framework with a fairness-promoting regularizer. To formally quantify the fairness of learned representations, we define a distributional representation fairness as following.

The above definition of distributional representation fairness resembles the demographic parity (DP) requiring $p(h({\bf{x}})|a=k)=p(h({\bf{x}}))$, where $h({\bf{x}})$ is a predictor that predicts the class label for ${\bf{x}}$. However, distributional representation fairness is a stronger condition, which implies DP for downstream classification tasks if the encoder network is fixed. This is a simple argument. Let $\mathcal{S}$ denote a random labeled training set for learning a linear classification model $h({\bf{x}})={\bf{w}}^{\top}E({\bf{x}})$ based on the encoded representation. Denote by ${\bf{w}}(\mathcal{S})$ a learned predictive model. With the distributional representation fairness, we have $p(E({\bf{x}})^{\top}{\bf{w}}(\mathcal{S})|a=k)=p(E({\bf{x}})^{\top}{\bf{w}}(\mathcal{S}))$ for a random data $({\bf{x}},a)$ that is independent of a random labeled training set $\mathcal{S}$. This is due to that $E({\bf{x}})^{\top}{\bf{w}}(\mathcal{S})|a=k$ and $E({\bf{x}})^{\top}{\bf{w}}(\mathcal{S})$ have the same moment generating functions. Hence, DP holds, i.e., $p(\text{sign}(E({\bf{x}})^{\top}{\bf{w}}(\mathcal{S}))|a=k)=p(\text{sign}(E({\bf{x}})^{\top}{\bf{w}}(\mathcal{S})))$.

Next, we present a proposition for justifying the zero-sum game framework. To this end, we abuse a notation $F_{\text{fair}}(E,D)=\mathbb{E}_{{\bf{x}},a}\phi(E,D;{\bf{x}},a)$ for any encoder $E$ and any discriminator $D$.

Remark: We attribute the credit of the above result to earlier works, e.g., [49]. It indicates the distribution of encoded representations is independent of the sensitive attribute. The proof of the above theorem is similar to the analysis of GAN.

It is notable that the above result is different from that derived in [40, 36], which considers a fixed encoder $E$ and only ensures the learned model recovers the true conditional distributions of the label and sensitive attribute given the representation. It has nothing related to fairness.

The optimization problem (3) deviates from existing studies of fair representation leaning in that (i) the GCL is a compositional function that requires more advanced optimization techniques in order to ensure convergence without requiring a large batch size [43]; (ii) the problem is a non-convex non-concave minimax compositional optimization, which is not a standard minimax optimization.

We have proposed a zero-sum game for fair self-supervised representation learning. We provided theoretical justification about the distributional representation fairness, and developed a stochastic algorithm for solving the minimax zero-sum game problems and established a convergence guarantee under mild conditions. Experiments on face image datasets demonstrate the effectiveness of our algorithm. One limitation of the work is that it focuses on single modality and multi-modality data would be an interesting future work.