ARMANI: Part-level Garment-Text Alignment for Unified Cross-Modal Fashion Design

Recently, a great deal of progress (Xia et al., 2020; Karras et al., 2019; Ramesh et al., 2021; Esser et al., 2021; Zhang et al., 2021c; Li et al., 2019c; Cai et al., 2019; Zhu and Ngo, 2020) has been made on the cross-modal image synthesis task, which aims to generate realistic pictures from various control signals, such as pure text, sketch images, or partial images. Existing methods can be divided into two categories, in which the first category generates images via GAN-based approaches. For instance, TediGAN (Xia et al., 2020) learns a StyleGAN (Karras et al., 2019) inversion module and then computes the visual-linguistic similarity to generate an image from cross-modal inputs. The second category of methods produces images via a two-stage image synthesis framework, where the first stage trains a VQVAE (Oord et al., 2017) to convert an image into a sequence of discrete tokens and convert such a sequence of tokens back into an image. During the second stage, various control signals are used as inputs and a Transformer is trained to capture the distribution of the token sequences. Leveraging this two-stage framework, DALL·E (Ramesh et al., 2021) and Cogview (Ding et al., 2021) generate high-resolution images from text descriptions, while M6-UFC (Zhang et al., 2021c) adopts a BERT-based two-stage framework to unify any number of cross-modal control signals for conditional image synthesis. These methods, however, use discrete tokens without textual information, resulting in sub-optimal generation capabilities. ARMANI addresses this problem by integrating textual information in the first stage, leading to more expressive tokens.

One of the key challenges in cross-modal research is the modeling of the interaction between two modalities, which is mainly addressed by two types of approaches: single-stream and dual-stream models. Single-stream models like VisualBERT (Li et al., 2019a), UNITER (Chen et al., 2020), and ViLT (Kim et al., 2021) directly concatenate the patch-wise and textual embeddings and leverage a Transformer to model interactions, while dual-stream models like ViLBERT (Li et al., 2019a), ALIGN (Jia et al., 2021), UNIMO (Li et al., 2020a), CLIP (Radford et al., 2021), FILIP (Yao et al., 2021), and GLIP (Li et al., 2021) learn separate encoders for the different modalities prior to aligning the modalities, which allows flexible use of different models for different modalities and to address various downstream tasks. Inspired by these dual-stream models, ARMARNI introduces the novel MaskCLIP module to learn the correspondence between the fine-grained features of different modalities.

The first stage aims to learn a semantically rich codebook and an expressive generator, where the codebook encompasses a variety of visual features from the dataset, while the generator can synthesize realistic images given a collection of codes sampled from the codebook. This is achieved by introducing a Variational Autoencoder (VAE) with a convolutional encoder (E) and a decoder (G) that is trained to conduct image reconstruction. Concretely, the encoder $E$ first compresses the input image $\mathbf{I}\in\mathbb{R}^{H\times W\times 3}$ into a low-resolution feature map $\mathbf{\hat{Z}}\in\mathbb{R}^{h\times w\times n_{z}}$. Then, for each spatial position (i,j) of $\mathbf{\hat{Z}}$, the vector $\hat{z}_{i,j}\in\mathbb{R}^{n_{z}}$ is quantized into a discrete code $z_{i,j}$ by replacing it with the closest entry $z_{k}$ in the learnable codebook $\mathcal{Z}=\left\{z_{k}\right\}_{k=1}^{K}\subset\mathbb{R}^{n_{z}}$, which can be formulated as:

Finally, the decoder $G$ takes as input the discrete feature map $\mathbf{Z}$ and reconstructs the input image,

Thus, once the VAE is fully trained, the codebook $\mathcal{Z}$ and the generator (i.e., decoder) $G$ can be obtained.

Given the learned VAE and the codebook, each image in the dataset can be converted into a discrete feature map $\mathbf{Z}$ and further be represented as an index sequence $\mathbf{S}\in\{0,...,|\mathcal{Z}|-1\}^{h\times w}$, in which each element represents the entry index in the codebook $\mathcal{Z}$. In the second stage, a sequence generation model (e.g., Transformer) is then used to model the sequence distribution $\mathbf{Pr}(\mathbf{S}|\mathbf{C})$ conditioned on the control signals $\mathbf{C}$. More specifically, taking as inputs the control signals $\mathbf{C}$ and the preceding predicted tokens $\mathbf{S}_{<i}$, the sequence generation model predicts a conditional distribution $\mathbf{Pr}(s_{i}|\mathbf{S}_{<i},\mathbf{C})$ for the next token $s_{i}$. Finally, the predicted index sequence $\mathbf{S}$ is converted to a discrete feature map and fed into the generator $G$ to generate a new image.

The learnable codebook in the existing two-stage-based methods (Ramesh et al., 2021; Zhang et al., 2021c) struggles to represent visual parts that do not (or only rarely) exist in the training set, largely impairing its expressivity during inference. To address this issue, our ARMANI framework combines each visual code in the codebook with its most relevant word embedding, resulting in a text-aware cross-modal codebook with more distinct semantics.

However, directly combining a visual code with its corresponding word embedding is not trivial, since the visual code and the word embedding are in different feature spaces and the correspondence between these two features is not accessible. To align the visual code and the word embedding and at the same time capture the detailed semantic information of the garment, we propose to leverage the garment segmentation mask, which has been proven useful in vision-language understanding to gain more fine-grained image information (Zhang et al., 2021b; Lu et al., 2019). In particular, ARMANI introduces a Mask-level Contrastive Language-Image Pretraining (MaskCLIP) module to explicitly learn the correspondence between the word embedding and the visual part through the segmentation mask (see Fig. 4). Concretely, during training, given a training batch $\{\mathbf{I}^{k},\mathbf{T}^{k}\}_{k=1}^{b}$, a garment segmentation mask $\mathbf{M}^{k}$ is first obtained from the garment image $\mathbf{I}^{k}$ using a mask prediction network to represent the detailed parts of the garment such as button, sleeve, belt, etc. The mask is then encoded as the visual token $\mathbf{\hat{Z}}^{k}\in\mathbb{R}^{n_{1}\times d}$. Similarly, the text sample $\mathbf{T}^{k}$ is encoded producing the text token $\mathbf{\hat{X}}^{k}\in\mathbb{R}^{n_{2}\times d}$. We then compute the similarity between each entry $z_{i}^{k}$ in $\mathbf{\hat{Z}}^{k}$ and all the text tokens $\mathbf{\hat{X}}^{k}$ and let the largest similarity represent the token-wise similarity between the visual token $z_{i}^{k}$ and the whole text:

The similarity between a mask and a text is then computed as the average over the token-wise similarity for the given mask. Thus, the similarity of $\mathbf{M}^{p}$ to $\mathbf{T}^{q}$ can be formulated as:

where $j^{*}=\arg\max_{0\leq j<n_{2}}({z_{i}^{p}}^{\top}\cdot x_{j}^{q})$. Similarly, the similarity of $\mathbf{T}^{q}$ to $\mathbf{M}^{p}$ can be formulated as:

where $i^{*}=\arg\max_{0\leq i<n_{1}}({z_{i}^{p}}^{\top}\cdot x_{j}^{q})$. Based on the definition of the token-wise similarity between the image and the text, the image-to-text token-wise contrastive loss for a training batch and its text-to-image counterpart can be calculated as:

After training MaskCLIP, we learn the cross-modal codebook by injecting the relevant word embedding into each visual code. More specifically, when learning the codebook, we first get the mask label of the input image $\mathbf{M}$ and its related text $\mathbf{T}$. For each visual part token $\hat{z}_{i}$ in $\mathbf{\hat{Z}}$, we then obtain its related text token $\hat{x}_{j}$ according to the similarity of the mask token and the text tokens, namely,

A cross-modal local feature $\tilde{z}_{i}$, which incorporates the visual information as well as the fine-grain text information, can then be obtained by averaging the visual token $\hat{z}_{i}$ and its corresponding text token $\hat{x}_{j}$. Finally, all cross-modal features $\tilde{z}_{i}$ are quantized using Eq. 1, and form the discrete feature map $\mathbf{Z}$, which is further sent to the generator to reconstruct the input image $\mathbf{I}$. In addition, we use the proposed MaskCLIP loss to ensure that the cross-modal codebook captures semantically accurate information by optimizing a symmetric cross entropy loss over the similarities of an image and text embedding batch. This results in the semantic loss

where $D$ is the similarity between the MaskCLIP embeddings of its two arguments, $T$ is the text description of the picture, and $M$ is the mask prediction model.

The total loss in the first stage can thus be formulated as:

where $\mathcal{L}_{VQ-GAN}$ denotes the training losses used in VQ-GAN (Esser et al., 2021).

In contrast to existing cross-modal image synthesis approaches (Esser et al., 2021; Patashnik et al., 2021; Tan et al., 2021; Kenan et al., 2020; Zhang et al., 2021a) which can only accomplish text-to-image synthesis or image-to-image synthesis, our ARMANI framework is a versatile model that can handle diverse control signals within a unified network. The main challenge of such a unified model is how to represent diverse control signals in a uniform way. To address this challenge, ARMANI introduces a cross-modal Transformer to jointly take diverse control signals as input and predict a sequence of discrete tokens that are used to generated the image (see Fig. 3). As opposed to CNN-based approaches, Transformers are capable of handling this task, since data from various modalities (e.g., images, text, audio, etc.) can be transferred into the same representation, the token sequence. Besides, since Transformers predict the next token in an autoregressive manner, according to the previous token sequence or the condition signals, we can easily add a new control signal into the model by prepending the token sequence of the new signal to the input tokens.

Specifically, the control signals in our ARMANI framework are comprised of pure text, sketches, and partial images, in which the pure text is directly obtained from a cross-modal dataset, while sketches and partial images can be obtained by applying the Canny edge detection algorithm (Canny, 1986) and extracting a random crop of the RGB image, respectively.

The control signals are first converted into text and image token sequences, where the text token sequence is obtained directly by a Transformer-based encoder. For the image token sequence, we instead learn a codebook for each type of signal and quantize each signal into a discrete image token sequence. The different token sequences are combined and a special token [SEP] is used to indicate the separation between the modalities. The cross-modal Transformer then predicts the token sequence and feeds it to the image decoder to synthesize the image.

During training, the cross-modal Transformer is trained by maximizing the log-likelihood of the generated token sequence conditioned on various control signals by minimizing the following training loss:

where $s$ represents the predicted token sequence while $c$ denotes the conditional token sequence. To further ensure that the generated image represents the text, we apply our semantic loss $\mathcal{L}_{semanitic}$ for the token prediction to improve the ability to capture the relation between text and image. Finally, the total loss in the second stage can thus be formulated as:

In this section, we visualize MaskCLIP’s capability of capturing fine-grained semantically-accurate cross-modal correspondence using the proposed word-part alignment.

As the corresponding ground truth is not available when generating images based on text conditions, we perform a human evaluation study in order to assess the image quality and its relevance to the text jointly. We observe from Fig. 8 that ARMANI’s results are preferred according to the human evaluation on all the three tasks and we also observe that, compared to the machine evaluation, the human evaluation indicates a larger difference among the models.

To validate the effectiveness of the semantically accurate alignment of our proposed MaskCLIP, we perform an ablation study, where we compare ARMANI with a version of ARMANI without the MaskCLIP module. We compare these models on all three tasks and observe that the full ARMANI model obtains higher CLIPscore (Hessel et al., 2021). This illustrates that the generated results are more relevant to the text description, thus validating the effect of MaskCLIP in enabling a semantically accurate cross-modal codebook. To validate the effectiveness of the semantic loss, we perform another ablation study, where we compare ARMANI to two versions of ARMANI, one without the semantic loss in the first stage and one without the semantic loss in the second stage. Again, we compare the CLIPScore (Hessel et al., 2021) for all three tasks and observe that the CLIPScore worsen as we remove the semantic losses, indicating that the semantic loss plays a significant role for the generation. QWe further visualize qualitative results of these ablation studies on the Text-to-Image task in Fig. 9. These results further demonstrate the validity of our method. In particular, we observe that if MaskCLIP alignment is removed, the codebook’s ability to express garment parts will be weakened. While the model can still leverage the garment part information via the semantic loss in the second stage, the generation results are worse. If we instead remove the semantic loss in the first stage, we observe that the information obtained from the MaskCLIP alignment will not be retained in the codebook, again worsening the generation ability for the garment details. Finally, if the semantic loss is removed in the second stage, it will lead to the garment details being ignored in the code sequence prediction of the generated picture and some parts will be ignored and not generated. Additional qualitative results for the other two tasks are provided in the appendix. We further show quantitative results on all three tasks in Table 2 to evaluate the effectiveness of our model components and observe that the absence of each component leads to a decrease in CLIPScore.