DRDM: A Disentangled Representations Diffusion Model for Synthesizing Realistic Person Images
^††thanks: This research is supported by the National Natural Science Foundation of China (62262045). Corresponding Email: [email protected]

The goal of the controllable person image synthesis task is to explicitly manipulate the body posture and appearance of the source individual picture, generating a photo-realistic person image that aligns with user expectations [1, 2, 3]. At present, majority of the approaches utilize conditional Generative Adversarial Networks (GANs) to transform the source individual photo into the desired target person image. For example, MUST [4] and ADGAN [5] encode style information from various semantic parts using encoders and incorporate style features through the Adaptive Instance Normalization (AdaIN) operation. GFLA [6] estimates the relationship between the source and target poses to guide the generation of the target photo, allowing it to preserve texture details. By contrast, CASD [7] and NTED [8] use attention mechanisms to better preserve the textural properties of the source picture. Nevertheless, these techniques frequently yield images with unclear and indistinct details, primarily because GANs struggle to accurately establish a direct transformation pathway from the source to the target image [9, 10, 11, 12, 13, 14, 15]. Recently, the diffusion model [16] has served as a source of inspiration for researchers to produce high-quality synthetic images by using a denoising score matching mechanism. For instance, models such as PIDM [1], OOTD [17], CFLD [2], and HumanSD [3], etc. can effectively preserve the texture details of a source photo by using a diffusion-based approach. However, these models face difficulties in fully maintaining the textural properties of every body parts because it directly encodes the texture of the entire source figure, leading to inaccuracies in clothing styles and limb textures, particularly when occlusions are present and postures are changed. As shown in Figure 1, PIDM tends to generate stretched limbs and false style garments. In fact, these problems are also commonly encountered in other spatial-temporal analysis and embodied AI tasks [18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28].

To address the aforementioned issues, we propose a Disentangled Representations Diffusion Model (DRDM) that consists of the body-part subspace decoupling block (BSDB), pose encoder, and noise prediction block to achieve the disentanglement of the different body parts and the decoupling of structure and texture in person images. Subsequently, we reassemble these disentangled appearance features according to the given target pose to generate a target person image. In particular, BSDB utilizes the human paring map as a regularization factor to extract disentangle features from different body parts of the source person images, and then disentangled textural features were obtained through a series of dedicated texture encoders. We introduce self-attention mechanisms within each encoder to address texture inaccuracies arising from imprecise semantic segmentation maps, allowing the texture encoders to focus solely on the accurate features of the corresponding body parts. Finally, we use a cross-attention mechanism to pass these disentangled features to the different layers of the noise prediction block to supply the network with rich disentangled features for generating a realistic target person image.

DRDM effectively addresses common occlusion issues, distortion at limb extremities and deviation of the garment style through the disentangled feature representation. This approach also enables explicit control over the source person image’s appearance and body posture by substituting texture features of different body parts of the source individual photo and pose features (Figure 1). During the inference, we devised a disentangled classifier-free guided sampling based on parsing map, which amplifies the conditional signals of texture and pose. This approach ensures a close alignment between the appearance of the generated person image and the target pose to preserve the realism of the generated result under varying poses. The experimental and evaluation results demonstrate that our method achieves state-of-the-art results on the Deepfashion dataset. Moreover, our approach generates images with more realistic details, and its superiority is confirmed through quantitative and qualitative comparisons and user studies. Our major contributions are as follows:

• •

  A diffusion model with a disentangled regularization module for synthesizing highly realistic person images. We can reassemble the disentangled textural features according to the parsing map, and demonstrate that this regularization factor matters the performance in addressing common issues such as occlusion and distortion of limbs.

• •

  A parsing map-based classifier-free disentangled guidance sampling method to ensure a close alignment between the generated appearance of person image and the target pose, thereby guaranteeing the authentic restoration of details during pose transformations.

Overall Framework: Our model focuses on achieving feature-decoupled person image synthesis, enabling realistic persona generation by flexibly manipulating the appearance and body pose of the source person image. This method excels in addressing common occlusion challenges, severe distortion at limb-end and deviation at cloth style within this task. The overall model architecture is illustrated in Figure 2. The DRDM framework consists of the BSDB, pose encoder $E_{p}$, and noise prediction block $\mathcal{H}_{N}$. During the generation of the target person image, we employ $E_{p}$ to encode the input pose $P_{t}$ into a high-dimensional space, which guides the network in recombining the decoupled texture features of the source person image. We employ a BSDB to decouple features of different body parts in the source person image $I_{s}$ to address occlusion and limb-end distortion issues. The disentangled textures of the various body parts are separately encoded and stacked to form a multi-scale disentangled texture feature $F_{s}$. Subsequently, the BSDB uses a cross-attention mechanism guided by the target pose to extract the required texture style from $F_{s}$ and inject it into different layers of $\mathcal{H}_{N}$. This process enables the network to generate high-fidelity texture details. Decoupling the textural features of the source person image helps the network in learning the relationship between body part textures and human structure, facilitating effective resolution of common issues, such as occlusion and distortion at limb-end extremities. Additionally, we can explicitly manipulate the appearance and body pose of the source portrait by replacing body part textures and body poses through the disentangled feature representation. During the inference, we propose a classifier-free disentangled guided sampling method to amplify the conditional signals of the disentangled textures and the target pose. This approach enables us to closely align the generated person image’s appearance with the target pose to enhance the quality of the generated images.

Experimental Details: We evaluate the performance of DRDM by conducting experiments on the widely used DeepFashion dataset,utilizing the Rebalanced Parsing model [31] to generate parsing maps. During the training phase, the Adam optimizer is configured with a learning rate of $2e-5$, and the batch size is $7$. We randomly set 10$\%$ of variables ${I}_{s}$, ${M}_{s}$, and ${P}_{t}$ to zero tensors to encourage the model to effectively learn the unconditional distribution. For sampling, we set $w_{s}$ to 3.0 and $w_{p}$ to 2.5.

Our experiments were conducted on a system equipped with four NVIDIA V100 GPUs.

In this work, we proposed a DRDM that allows for a precise control of the pose and appearance of the source person image to synthesize realistic person images. In practice, there are three types of degeneration key issues existing in portrait composition, i.e., details missing, limbs distortion and garment style deviation. We exploit the parsing map as a regularization factor for preserving the figure shape and body part subspace texture style, and then extract structure and texture style features in each subspace. The introduction of the disentangled texture fusion block enables the decoupling of textures in different semantic parts of the source person image, which can then be guided by the target pose to compose the appearance of the target person. Additionally, we utilize a parsing map-based disentangled classifier-free guided sampling approach to achieve accurate detail reconstruction under pose variations. We demonstrate the effectiveness of DRDM for controllable person image synthesis task through extensive quantitative, ablation and user studies.