Leveraging Off-the-shelf Diffusion Model for Multi-attribute
Fashion Image Manipulation

Given a clean data distribution $x_{0}\sim q(x_{0})$, a Markovian noising process can be defined as:

with $\beta_{t}$ being the noise scale controlling the diffusion process and $\mathbf{I}$ being the identity matrix. It is shown that with a large enough total time steps $T$, $x_{T}$ can be regarded as an isotropic Gaussian variable. As we are gradually adding Gaussian noises, an arbitrary noising step can be directly computed as

with a scalar $\bar{\alpha}_{t}=\prod_{i=1}^{t}(1-\beta_{i})$.

To sample under the diffusion framework, we reverse the forward denoising process. In other words, we begin with a Gaussian noise $x_{T}\sim\mathcal{N}(0,\mathbf{I})$, and go through a series of posterior sampling steps, $x_{t-1}\sim q(x_{t-1}|x_{t})$, which is typically modeled with a neural network with learnable parameters $\theta$:

[11] shows that with mild assumptions, this posterior can be represented as a Gaussian distribution with a diagonal covariance and the mean $\mu_{\theta}(x_{t},t)$ can be computed from $\epsilon_{\theta}(x_{t},t)$. Thus, the neural network is trained to predict the noise $\epsilon_{\theta}(x_{t},t)$ instead of the mean under the following reconstruction objective:

Several works [22, 7] recently introduce empirical strategies to produce samples with better quality, such as model architecture and objective weighting.

In this work, we propose to tailor this generic editing framework for fashion domain with concise modifications. We first show that naively applying this general framework yields suboptimal results, then present an efficient finetuning technique to prepare a multi-attribute fashion classifier that is capable of reasoning different spatial regions for classification of different attributes. Lastly, we leverage its natural attention map for mask-free fashion attribute editing, completing an extremely simple and generic fashion image manipulation framework. The overall procedure of ours is described in Procedure 1.

As shown in Fig. 1, our early experiments show that naively applying the guidance signal from a pretrained model (e.g., CLIP) as in blended-diffusion yields unsatisfactory results in fashion domain due to the severe domain gap. To provide a more fine-grained guidance to our diffusion model, we explore ways to efficiently train a domain-specific classifier.

Vision Transformers (ViTs) [8] have recently shown impressive performances across diverse computer vision tasks, frequently replacing the state-of-the-arts previously held by convolutional neural networks [4, 19]. In conventional ViTs, the global semantic of an input image is aggregated to a single [CLS] token, which is commonly projected with additional learnable layers for downstream tasks. However, in our multi-attribute classification setting, this can lead to over-simplification of visual semantics, as different attributes reside in different spatial regions. In other words, we have to look at different parts of a clothing to determine different attributes such as neckline or sleeve length. Hence, we adopt recently proposed attention-pooling mechanism [36] to aggregate the spatial features in a learnable way with an additional cross-attention layer. This enables our model to attend to different spatial regions for classification of different attributes. After the attention pooling, each attribute token is projected for classification, and they are optimized with the classic cross-entropy loss formulated as follows:

Here, $|A|$ is the number of attributes, $x^{(i)}$ refers to the model prediction for the $i$-th attribute, $C^{(i)}$ is the number of classes for the attribute, and $y^{(i)}$ indicates the ground truth label. We mask those with missing attributes, e.g., neckline for a skirt. Note that we drop the mini-batch notation for brevity.

As ViTs are famous for being notoriously data-hungry [8], we choose to finetune a ViT pretrained on a larger dataset. We explore different initialization and finetuning methods and present the best practice for our setting in Sec.4.1.

One of the challenges of image editing is to concisely alter the regions to suit the given condition while keeping the irrelevant parts unchanged. Instead of relying on a user-provided mask as done in [3], we employ the attention map of our classifier to determine the relevant regions. This is distinguished from other works that use separate procedures such as Class Activation Map [38] or Grad-CAM [31], as we simply use the attention map of each attribute token (in the attention pooling layer), which is computed during the classifier forward pass.

Formally, for a multi-attribute image editing, we average-pool the spatial attention maps for the target attribute tokens, and impose background preservation loss on the low-attention regions as follows:

where $\bar{M}$ refers to the average-pooled spatial attention maps, $\odot$ indicates the Hadamard product, $\hat{X}$ is the predicted output from $X$, and $pd$ is Learned Perceptual Image Patch Similarity (LPIPS) [37], also known as the perceptual distance.

We build our framework upon blended-diffusion [3], and make modifications to suit the off-the-shelf diffusion model for fashion attribute editing. First, we replace CLIP guidance with our finetuned classifier guidance that supports fine-grained multi-attribute editing. Then, we get rid of the user mask and enforce background preservation with the classifier attention map. We go through the diffusion steps with the initial input image, where the classifier guidance signal pulls the diffusion process towards the target attributes. The overall pipeline is illustrated in Fig. 2.

We use the commonly used Shopping100k [1] dataset for training and evaluations. As our diffusion model does not need additional training, we only train the classifier on this dataset. For baselines, we compare our method with StarGAN [5] and Fashion-AttGAN [24], two representative attribute manipulation methods of which the latter is further suited for the fashion domain. We train these models on Shopping100k with 8 most common attributes, summing to 100 labels total. The details are specified in Tab. 1.

We leverage the unconditional diffusion model pretrained on 256x256 scale Imagenet [7]. Surprisingly enough, despite the fact that the diffusion model has never seen any fashion-specific images, the classifier alone can guide fashion attribute editing in the rich latent space of the diffusion model. For the classifier, we test three initialization methods: random, Imagenet ViT and CLIP ViT. ViT-Large model was used for all experiments, and we added an attention pooling layer on top of the final transformer block. This attention pooler is essentially a cross-attention layer with 8 queries, each corresponding to an attribute. These tokens are linearly projected for final classification.

Previous methods using attribute-editing GANs typically focus on one or two attributes (e.g., color or sleeve length), as these methods enforce the generator itself to learn to reason about different fashion attributes. Hence, we first observe how these conventional GANs perform in the more generic manipulation setting, where we tackle most of the widely used fashion attributes, ranging from item category to fit and neckline, with a single framework.

Fig. 3 clearly illustrates that naively applying attribute-editing GANs [5, 24] leads to severe training collapse. As these methods aim to embed attribute manipulation capacity into the generator, they have inherent disadvantage for scalability. For multi-attribute editing, e.g., editing the neckline and the sleeve length, multiple generative models have to be trained to perform a sequence of manipulation operations, which greatly limits their practical applications.

As demonstrated in the previous section, training an attribute-aware generative model is not a scalable option. Therefore, we investigate ways to leverage a well trained generative model with a domain-specific attribute-aware classifier, deferring the attribute reasoning capacity to the classifier for the sake of a generic and scalable attribute editing system.

We first present empirical findings from the classifier training. As annotated data is relatively scarce in the fashion domain, we regard the initialization scheme to be a key factor in the final classifier performance. Hence, we explore three settings: (1) random initialization, (2) Imagenet-pretrained initialization and (3) CLIP-pretrained initialization. For the last two, we further compare different finetuning approaches, i.e., how many layers or blocks to set as learnable (unfreeze) during the adaptation period.

Tab. 2 shows the result. We observe that with relatively limited data, freezing most of the parameters while training a learnable attention pooling layer yields the best overall performance. This indicates that the pretrained ViT backbone retains a reasonable level of general visual reasoning, and the additional attention pooling layer successfully extracts diverse visual information to perform the fine-grained multi-attribute classification. Moreover, initializing with pretrained weights helps, and CLIP pretraining covers a wider range of visual semantic compared to Imagenet, possibly leading to better outcomes. Lastly, as different attributes demand different aspects of visual reasoning, the classification accuracy trend is not always consistent. We basically choose the model with the highest average score for the best overall guidance.

Data augmentations have been shown to affect the performance of visual classifier models [34]. Thus, we explore the influence of data augmentations in our framework. From Tab. 3, we gain conflicting insights; random augmentation boosts the performance for CLIP-pretrained ViT but damages it for the Imagenet-pretrained variant. We note that this result is obtained from finetuning the attention pooling layer on Shopping100k dataset, so it is possible to observe different trends in different settings. In our situation, we hypothesize that as we freeze most of the model parameters, imposing strong augmentations could result in over-pressuring the limited learnable parameters. As CLIP-pretraining prepares the model for a much greater diversity (compared to Imagenet, in terms of data scale), this potential adversarial effect only realizes for the less robust model variant.

Lastly, we explore the performance trend depending on the model size. Tab. 4 shows that bigger models (or longer input sequence) consistently yield better outcomes, even under a strict finetuning scheme. Hence, we adopt the best performing model variant, CLIP-pretrained ViT-L/14, as our diffusion guidance in the following sections.

Thanks to the domain-specific multi-attribute classifier, we can leverage a pretrained off-the-shelf diffusion model (not necessarily adapted for fashion domain) for generic fashion attribute editing. Fig. 4 shows qualitative manipulation results on diverse attributes. We note that these editing operations are all done with a single finetuned classifier and an off-the-shelf diffusion model. Unlike attribute-editing GANs that collapse as the number of attributes increases (see Fig. 3), our framework shows satisfactory editing capabilities across various attributes as the classifier can handle a much wider set of attributes with little instability when compared to the generator. Especially, to our knowledge, attributes like item category or fabric have not been directly explored in the previous works on attribute editing mainly due to their difficulties. Since we employ a powerful diffusion model trained with a large dataset, providing the right guidance yields impressive outcomes.

Moreover, it is noteworthy how the diffusion model controls both the texture and the shape in a stable manner. The left side of Fig. 4 displays shape deformation, and the right side shows texture manipulations, where samples with convincing qualities are consistently delivered. The whole editing pipeline does not require any texture- or shape-specific modules, which is distinguished from the prior art [18], making our framework extremely simple and generic.

In Fig. 5, we present multi-attribute editing results, i.e., manipulating multiple attributes at once. This task is apparently more challenging as it requires the editing of different attributes to be in harmony with one another. We observe that our framework produces reasonable outcomes across different attribute combinations, sometimes deforming the original image to a significant extent to meet the complex target condition.

As our method uses the attribute-wise classifier attention-map for background preservation, the ability to generate adequate spatial attention-maps is critical. Fig. 6 displays the classifier attention maps for different attribute editing operations, where we see that the classifier is capable of attending to the relevant regions with the learned attention-pooling layer. This component plays a vital role in removing the manual region masking that is necessary in blended diffusion [3], rendering our manipulation pipeline simple and compact.

Classifier-guidance is the key component in our framework that injects conditioning information (target attribute) to the diffusion process. In Fig. 7, we illustrate how the capacity of the classifier affects the editing process. We set blended diffusion as the baseline, as CLIP guidance can be regarded as the coarsest level of guidance. Then we present three classifier variants with different classification accuracies. We observe that the classification performance clearly affects the spatial attention quality and the final editing performance, highlighting the importance of effective classifier finetuning.

Fig. 8 shows qualitative ablations on the core loss components and their hyperparameters. From the first row, we observe that background preservation loss is crucial for our framework, as there is no separate spatial mask that explicitly marks the region to be edited. Hence, without this term, the diffusion model generates random images that satisfy the given condition, e.g., random pictures of v-neck t-shirts.

The second row illustrates how the guidance weight affects the sample quality. With small guidance scale (low $\lambda$ value), we obtain realistic samples that are not aligned with the attribute condition. When the guidance scale is too big, the diffusion process yields unsatisfactory outputs, possibly due to the train-test mismatch pointed out by [30]. We find the sweet spot at around $\lambda=100$, where a good balance between the fidelity and the alignment is achieved. Hence, all of our experiments are conducted in this setting.