CAFE-GAN: Arbitrary Face Attribute Editing
with Complementary Attention Feature

The discriminator $D$ takes both real images and fake images modified by $G$ as input. $D$ consists of three main parts, i.e., $D_{att}$, $D_{adv}$, and $D_{cls}$ as illustrated in right side of Fig. 2. Unlike other arbitrary attribute editing methods [6, 10, 21], a spatial attention mechanism is applied to mid-level features $f$ in our discriminator. $D_{att}$ plays a major role in applying attention mechanism with a novel concept of adopting complementary feature maps. $D_{att}$ consists of an attention branch (AB) and a complementary attention branch (CAB) and they generate $k$ attention maps, which are the number of attributes, respectively. $M=\{M_{1},...,M_{k}\}$, a collection of attention maps from AB, contains important regions of attributes that exist in an input image while $M^{c}=\{M^{c}_{1},...,M^{c}_{k}\}$ from CAB contains casual region of attributes that do not exist. These attention maps are applied to mid-level features by the attention mechanism as

where $M_{i}$ and $M^{c}_{i}$ are the $i$-th attention map from AB and CAB respectively and $(\cdot)$ denotes element-wise multiplication.

The following paragraph describes $D_{att}$ in detail as illustrated in Fig. 3. As explained above, we adopt Attention Branch (AB) to identify attribute-relevant regions following ABN [7]. AB takes mid-level features of an input image and generate $~{}h~{}\times~{}w\times~{}k~{}$ attention features (AF), denoted by $A$, with a $~{}1~{}\times~{}1\times~{}k~{}$ convolution layer. Here, $k$ denotes the number of channels in $A$ which is the same as the number of attributes. $h$ and $w$ denote the height and width of the feature map respectively. AB outputs $k$ attention maps $M_{1},...M_{k}$ with $~{}1~{}\times~{}1\times~{}k~{}$ convolution layers and a sigmoid layer. It also outputs activation of each attribute class by global average pooling (GAP) [20]. The $~{}h~{}\times~{}w\times~{}k~{}$ attention feature map $A$ is converted to $~{}1~{}\times~{}1\times~{}k~{}$ feature map by GAP to produce probability score of each class with a sigmoid layer. The probability score is compared to label $v_{s}$ with a cross-entropy loss to train $D$ to minimize classification errors when a real image (source image) is given as an input. Therefore, attention loss of AB is defined as

where $x$ is real image, $v_{s}^{(i)}$ denotes the $i$-th value of source attribute vector and $D_{AB}^{(i)}(x)$ denotes the $i$-th probability score that is output of AB. Therefore, the values of each channel in $A$ are directly related to activation of the corresponding attribute. AB can extract $A$ which represents spatial information about attributes contained in the input image. However $A$ does not include the information about attributes that are not present in the image because $A_{i}$, the $i$-th channel of feature map $A$, does not have response if $i$-th attribute is not in the input image.

This aspect does not influence the classification models like ABN because it only needs to activate a channel that corresponds to the correct attribute shown in an input image. However, for handling any arbitrary attribute, the generative model needs to be able to expect related spatial region even when an input image does not possess the attribute. Hence, there is a limit to apply existing visual explanation methods into the discriminator of attribute editing model directly.

To address this problem, we developed the concept of complementary attention and implemented the idea in our algorithm by integrating Complementary Attention Branch (CAB).The concept of CAB is intuitive that it extracts complementary attention feature (CAFE), denoted by $A^{c}$, which represents causative region of attribute that are not present in image. For example, CAB detects the lower part of face about attribute $Beard$ if the beard is not in an input image. To achieve this inverse class activation, we exploit complementary attribute vector $\bar{v_{s}}$ to compare with probability score from $A^{c}$ and $\bar{v_{s}}$ is formulated by

hence the attention loss of CAB is formulated as

where, $D_{CAB}^{(i)}(x)$ denotes the $i$-th probability score that is output of CAB. CAB is designed to generate a set of attention maps $M^{c}$ for attention mechanism from $A^{c}$. Therefore $A^{c}$ should contain spatial information to help $D_{cls}$ classify attributes. In other words, $A^{c}$ represents causative regions of non-existing attribute. With AB and CAB, our model extracts attention feature map about all attributes because $A$ and $A^{c}$ are complementary. In other words, for any $i$-th attribute, if $A_{i}$ does not have response values, $A_{i}^{c}$ has them and vice versa.

Two groups of attention maps $M$ and $M^{c}$, outputs of AB and CAB respectively, have different activation corresponding to attributes. In other words, $M$ is about existing attributes of the input image while $M^{c}$ is about absent attributes of that. After attention mechanism, the transformed features are forwarded to two multi-attribute classifiers in $D_{cls}$ and the classifier 1 and classifer 2 classify correct label of image with $f^{\prime}$ and $f^{\prime\prime}$ respectively. Each classifier outputs the probability of each attribute with cross-entropy loss. For discriminator, it learns to classify real image $x$ with two different attention mechanism, i.e.,

where $D_{cls_{1}}$ and $D_{cls_{2}}$ stand for two classifiers using collections of attention maps $M=\{M_{1},...,M_{k}\}$ and $M^{c}=\{M_{1}^{c},...,M_{k}^{c}\}$, respectively. Therefore, the reason CAFE can represent the spatial information of non-existent attributes is that CAB has to generate the attention maps that can help to improve performance of the classifiers while reacting to non-existent attributes by GAP.

In $D$, there is another branch $D_{adv}$ distinguishes real image $x$ and fake image $y$ in order to guarantee visually realistic output with adversarial learning. In particular, we employ adversarial loss in WGAN-GP [9], hence the adversarial loss of $D$ is given as

where $\hat{x}$ is weighted sum of real and fake sample with randomly selected weight $\alpha\in[0,1]$.

The generator $G$ takes both the source image $x$ and the target attribution label $v_{t}$ as input, and then conducts a transformation of $x$ to $y$, denoted by $y=G(x,v_{t})$. The goal of $G$ is to generate an image $y$ with attributes according to $v_{t}$ while maintaining the identity of $x$. $G$ consists of two components: an encoder $G_{enc}$ and a decoder $G_{dec}$. From the given source image $x$, $G_{enc}$ encodes the image into a latent representation $z$. Then, the decoder generates a fake image $y$ with the latent feature $z$ and the target attribute vector $v_{t}$. Following  [21], we compute a difference attribute vector $v_{d}$ between the source and the target attribute vectors and use it as an input to our decoder.

Thus, the process can be expressed as

In addition, we adopt the skip connection methodology used in STGAN [21] between $G_{enc}$ and $G_{dec}$ to minimize loss of fine-scale information due to down sampling. After that, $D$ takes both the source image $x$ and the edited image $y$ as input. $G$ aims to generate an image that can be classified by $D_{cls_{1}}$ and $D_{cls_{2}}$ as target attribute, hence the classification loss of $G$ is defined as

where $v_{t}^{(i)}$ denotes the $i$-th value of target attribute vector.

Although $D_{att}$ in our discriminator can obtain the spatial information about all attributes, it is necessary to ensure that $G$ has ability to change the relevant regions of given target attributes. Attention feature maps of the source image and the edited image should be different on those corresponding to attributes that are changed while the rest of attention feature maps should be the same. Therefore we propose a novel complementary matching method as illustrated in Fig. 4. For the attributes that are not to be changed, the attention feature maps of edited image should be same with that of source image. In other words, $G$ learns to match AF of edited image to AF of source image when the attributes remain the same (black arrows in Fig. 4), and $G$ also learns from matching CAFE of the source image. When the given target attributes are different from the source image, $G$ learns to match AF of edited image to CAFE of source image (red arrows in Fig. 4), same with CAFE of edited image to AF of source image. Let $\{A^{(x)},A^{c(x)}\}$ and $\{A^{(y)},A^{c(y)}\}$ denote set of AF and CAFE from two different samples, real image x and fake image y, respectively. Complementary matching is conducted for changed attributes and thus the complementary matching loss is defined as

where $k$ is the number of attributes and $N(i)$ denotes the number of elements in feature map. $A_{i}^{(x)/(y)}$ and $A_{i}^{c(x)/(y)}$ denote $i$-th channel of $A^{(x)/(y)}$ and $A^{c(x)/(y)}$ respectively and $v_{d}^{(i)}$ denotes $i$-th value of difference attribute vector $v_{d}$.

For adversarial training of GAN, we also adopt the adversarial loss to generator used in WGAN-GP [9], i.e.,

Although the generator can edit face attribute with $\mathcal{L}_{{G}_{cls}}$ and generate realistic image with $\mathcal{L}_{{G}_{adv}}$, it should preserve identity of image. Therefore, $G$ should reconstruct the source image when difference attribute vector is zero. We adopt the pixel-level reconstruction loss, i.e.,

where we use $\ell_{1}$ loss for sharpness of reconstructed image and $\mathbf{0}$ denotes zero vector.

Finally, the full objective to train discriminator $D$ is formulated as

and that for the generator $G$ is formulated as

where $\lambda_{att},\lambda_{D_{cls}},\lambda_{CM},\lambda_{G_{cls}}$ and $\lambda_{rec}$ are hyper-parameters which control the relative importance of the terms.

We use CelebFaces Attributes (CelebA) dataset [23] which consists of 202,599 facial images of celebrities. Each image is annotated with 40 binary attribute labels and cropped to $178\times 218$. We crop each image to $170\times 170$ and resize to $128\times 128$. For comparison, we choose the same attributes used in the state-of-the-art models [6, 10, 21]. In our experiment, coefficients of the objective functions in Eq. (15) and (16) are set to $\lambda_{att}=\lambda_{D_{cls}}=\lambda_{CM}=1,\lambda_{G_{cls}}=10,$ and $\lambda_{rec}=100$. We adopt ADAM [15] solver with $\beta_{1}=0.5$ and $\beta_{2}=0.999$, and the learning rate is set initially to 0.0002 and set to decay to 0.0001 after 100 epochs.

First, we conduct qualitative analysis by comparing our approach with three state-of-the-art methods in arbitrary face editing, i.e., AttGAN [10], StarGAN [6] and STGAN [21]. The results are shown in Fig. 5. Each column represents different attribute manipulation and each row represents qualitative results from the methods compared. The source image is placed in the leftmost of each row and we analyze results about single attribute as well as multiple attributes. First, AttGAN [10] and StarGAN [6] perform reasonably on attributes such as Add bangs. However they tend to edit irrelevant regions for attributes such as Blond hair or Pale skin and they also result in blurry images for attributes such as To bald. STGAN [21] improves manipulating ability by modifying structure of the generator, but this model also presents unnatural images for some attributes like To Bald and Add Bangs. In addition, unwanted changes are inevitable for some attributes such as Blond hair. As shown in Fig. 5, our model can successfully convert local attributes like Eyeglasses as well as global attributes like To female and To old. Last three columns represent results of multi-attribute editing. Hair color in the last column denotes $\textit{Black}~{}\leftrightarrow~{}\textit{Brown hair}$. It can be seen that our model delivers more natural and well-edited images than the other approaches.

We also compare our model with SaGAN [37] which adopt spatial attention concept. As shown in Fig. 6, SaGAN could not edit well in global attributes like To male and To old. Even when the target attribute is on a localized region like Eyeglasses, it performed poorly. However, our model calculates the spatial information in feature-level, not in pixel-level. As a result, our model shows well-edited and natural results in both local and global attributes. In addition, note that SaGAN is a single-attribute editing model, wherein it requires one model to be trained per attribute.

We present quantitative results by comparison with two of the three models compared earlier [10, 21]. In the absence of the ground-truth, the success of arbitrary attribute editing can be determined by a well trained multi-label classifier, i.e., well edited image would be classified to target domain by a well-trained attribute classifier. For fair comparison, we adopted the classifier that was used to evaluate attribute generation score for STGAN [21] and AttGAN [10].To evaluate quantitative result, we use official code which contains network architecture and weights of parameter in well-trained model. We exclude StarGAN [6] in this section because the official code of StarGAN provides only few attributes. There are 2,000 edited images evaluated for each attribute and their source images come from the test set in CelebA dataset. We measure the classification score of each attribute of edited images and they are listed in Table 1. In Table 1, “Blond h.” and “Open m.” denote Blond hair and Open mouth respectively. While our model shows competitive scores on the average for many attributes, it also delivered overwhelming results compared to the other models for specific attributes such as Bald. For attributes such as Mustache and Open mouth, STGAN results in slightly better performance over our model.

This section presents analysis of the proposed method. First we show the result of visualization of our attention features and then we conduct ablation study to demonstrate effectiveness of CAFE.