FlexPose: Pose Distribution Adaptation with Limited Guidance

As illustrated in Figure 2, our framework consists of three phases: Generic Pose Distribution Estimation. We learn a generator $g_{s}(\cdot)$ on the pose set $S$ to estimate the pose distribution $\mathcal{D}_{s}$. The generator takes a latent code $\bm{z}$ as input and outputs a skeleton $\hat{\bm{x}}_{s}$, i.e. $\hat{\bm{x}}_{s}=g_{s}(\bm{z})$. We take the distribution of generated $\hat{\bm{x}}_{s}$ to mimic that of the generic pose $\bm{x}_{s}$. Here, $\bm{x}$ is the corresponding skeleton image of an annotation $\bm{y}$ as shown in the left part of Figure 2. Pose Distribution Adaptation. Given the limited target annotation set $T$, we transfer $g_{s}(\cdot)$ to fit the pose distribution $\mathcal{D}_{t}$ and learn a new generator $g_{t}(\cdot)$ of the target pose domain. Considering the limited knowledge acquired from target pose annotation $T$, we introduce three regularizations, Linear, Sparse and Pose-mixup, to avoid reaching a collapse solution. Target Pose Sampling. The transferred generator $g_{t}(\cdot)$ can flexibly synthesize any number of fake pose annotations by randomly sampling in the latent space. This generated annotation set $\hat{T}$ will be treated as an extension of given annotations set $T$ in the downstream tasks, e.g., Keypoints Annotation and Pose-conditional Human Image Generation, since poses within both of them follow the distribution $\mathcal{D}_{t}$.

Deep generative models have been widely verified that they have a rich capacity to well approximate image distributions when given sufficient training data. Motivated by the success of these generative models (Karras, Laine, and Aila 2019) on natural/artificial image generation, we treat 2D pose annotations $\bm{y}_{s},\bm{y}_{t}\in\mathbb{R}^{M\times 2}$ as skeleton images $\bm{x}_{s},\bm{x}_{t}\in\mathbb{R}^{C\times W\times H}$ and extend an image generator to generate 2D pose annotations by synthesizing corresponding skeleton images. As shown in the left part of Figure 2, the transformation from the 2D keypoints to the skeleton images can be implemented by functions $\alpha(\cdot)$, namely $\bm{x}=\alpha(\bm{y})$, where $\alpha(\cdot)$ simply draws keypoints from $\bm{y}$ and connects them with straight lines on a blank figure. The visual effect is similar to the stick man. To achieve precise semantic alignment with the appearance correspondence, each bone in the skeleton image is assigned a unique color. Therefore, $C$ of each skeleton image is set as three (RGB channels). Compared with Black&White, the colorful embedding brings marginal improvement in the quality of generated skeletons.

A generator can be formulated as a mapping function $g(\cdot)$, which gets a latent code $\bm{z}$ and outputs a skeleton image $\bm{x}$. The probability distribution of skeleton images hence is estimated by $p(\bm{x})=p(\bm{z})p_{g}(\bm{x}|\bm{z})$. We assume that the pose distributions of different datasets share similar pose prior, and their distributions can transfer to one another by geometric transformations. Based on this assumption, we further factorize the generator $g(\cdot)$ as $g=\phi\circ\delta$. Therefore, the source skeleton image generator can be formulated as

$$p(\hat{\bm{x}}_{s})=p(\bm{z})~{}p^{s}_{g}(\hat{\bm{x}}_{s}|\bm{z})=p(\bm{z})~{}p_{\delta}^{s}(\bm{h}_{s}|\bm{z})~{}p_{\phi}(\hat{\bm{x}}_{s}|\bm{h}_{s}),$$

(1)

in which $\phi(\cdot)$ preserves the learned pose prior and $\delta(\cdot)$ records the mapping from the learned prior $\bm{h}$ to the skeleton image $\bm{x}_{s}$ of a certain pose domain. Similarly, the distribution of target domain can be formulated as $p(\hat{\bm{x}}_{t})=p(\bm{z})~{}p_{\delta}^{t}(\bm{h}_{t}|\bm{z})~{}p_{\phi}(\hat{\bm{x}}_{t}|\bm{h}_{t})$. With the prior sharing assumption, the pose distribution adaptation aims at transferring pre-trained conditional probability $p_{\delta}^{s}(\bm{h}_{s}|\bm{z})$ to $p_{\delta}^{t}(\bm{h}_{t}|\bm{z})$ with guidance from the pose annotation set $T$:

$$p_{\delta}^{s}(\bm{h}_{s}|\bm{z})\stackrel{{\scriptstyle T}}{{\longrightarrow}}p_{\delta}^{t}(\bm{h}_{t}|\bm{z}).$$

(2)

Considering the ability of StyleGAN in separating high-level attributes and in the interpolation between these attributes, we utilize StyleGAN network architecture to disentangle the pose prior and the transformation of the source skeleton image generator,

$$g_{s}=\phi\circ{\delta}_{s}=\phi\circ(A\circ f)_{s},$$

(3)

where $f(\cdot)$ is a non-linear mapping that takes random noise as input and outputs a random vector. $A(\cdot)$ is a learned affine transformation and can be treated as a block diagonal matrix with $L$ blocks, where $L$ is the number of layers. The output of $A(\cdot)$ is the style code to modulate the synthesis network $\phi(\cdot)$ by adaptive instance normalization. Due to the ability of StyleGAN in style control, we can directly adapt the distribution of source skeleton image to the target domain by adjusting the style code.

As illustrated in Figure 3, to transfer $p_{\delta}^{s}(\bm{h}_{s}|\bm{z})$ to $p_{\delta}^{t}(\bm{h}_{t}|\bm{z})$, we adjust the style code by introducing a transfer function $\tau(\cdot)$ at the output of $\delta(\cdot)$, and therefore the target domain generator is defined as

$$g_{t}=\phi\circ{\delta}_{t}=\phi\circ(\tau\circ{\delta}_{s}).$$

(4)

To learn the transfer function $\tau$, we first randomly sample $|T|$ latent codes $\bm{z}$ (one for each pose in $T$) from the latent space, and require the generator maps each code to corresponding skeletons in $T$. This transferring procedure can be achieved by minimizing the following perceptual loss,

$$\min_{\theta_{\tau}}\mathcal{L}_{s\rightarrow t}=\min_{\theta_{\tau}}\sum\Big{\|}\Gamma\big{(}g_{t}(\bm{z});\theta_{\tau}\big{)}-\Gamma\big{(}\bm{x}\big{)}\Big{\|}^{2}_{2},$$

(5)

where $\theta_{\tau}$ is the parameter of $\tau(\cdot)$, $\Gamma$ is a pre-trained feature extractor, $\bm{z}$ is from the set of sampled latent codes, and $\bm{x}$ is the skeleton image drawn from the pose annotation set $T$.

However, the problem is we only have few-shot guidance $T$ from the target domain distribution. Given a data-starving deep learning model, the guidance is insufficient to reach a satisfactory solution. For that reason, we introduce three regularizations to alleviate the data-insufficient issue.

Once the transferred generator $g_{t}(\cdot)$ is obtained, we can generate theoretically as many target skeleton images $\tilde{\bm{x}}_{t}$ as possible by randomly sampling latent codes in the estimated target distribution ${\mathcal{D}_{t}}$. Unfortunately, the generated target skeleton images are not perfect and may bring in artifacts which mislead the training of a neural network. To address this issue, we utilize $\beta(\cdot)$ to filter out the random noise. $\beta(\cdot)$ is a neural network regressor pre-trained on $S$ and acts as a tight information bottleneck that preserves skeleton information and ignores random noise. Following the generation of the fake skeleton images $\tilde{\bm{x}}_{t}$ from $g_{t}(\cdot)$, we extract the coordinates of interpretable 2D keypoints $\hat{T}=\{\hat{{\bm{y}_{t}}}\}$ from it, e.g., hands, by applying $\hat{\bm{y}_{t}}=\beta(\tilde{\bm{x}}_{t})$. Thereafter, we can get a clean generated skeleton $\hat{\bm{x}}_{t}$ by a re-render process,

$$\hat{\bm{x}}_{t}=\alpha(\hat{\bm{y}_{t}})=\alpha\big{(}~{}\beta(\tilde{\bm{x}}_{t})~{}\big{)}.$$

(8)

These generated skeleton images and the corresponding generated 2D pose annotations can be further utilized to assist any pose-related down-stream tasks.

In this subsection, we conduct a transformation experiment between COCO (Lin et al. 2014) and Human3.6M (Ionescu et al. 2014) to show how FlexPose works.

In Figure 6, we plot the t-SNE embedding of the poses generated by FlexPose (Adapted COCO), comparing it with the embedding of poses from the source (COCO) and target (Human3.6M) dataset. As can be seen, the embedding of poses from the source and target dataset are separated, and the distribution of generated poses significantly overlaps with the target ones. We also noted that the pose distributions are ‘mismatched’ in the upper right region. Considering that only two shots are utilized as guidance for each class during the transformation, such a mismatch is reasonable.

Quantitatively, we measure the similarity between the transferred distribution and target distribution using the Fréchet distance (FD), which follows from the Wasserstein-based definition of FID (Heusel et al. 2017) without the application of the pre-trained Inception network. We also measure the square of Maximum Mean Discrepancy (MMD) to provide more insights. The measurements are conducted on the keypoint coordinates space.

We compare our method with three strong competitors. AdaGAN (Noguchi and Harada 2019), FreezeD (Mo, Cho, and Shin 2020) and LoRA (Hu et al. 2021) (rank $r$=8). Both AdaGAN and FreezeD suggest finetuning-based strategies with regularization. LoRA is the most related work to our FlexPose, introducing low-rank regularization to the generative model. For all methods, we generate 50k samples from the transferred generative model and compare them with all samples in the target dataset Human3.6M. In Table 1, experiment results suggest that FlexPose gives superior performance on both MMD and FD evaluation, indicating the transferred distribution shares more similar characteristics to the target distribution. The finding remains the same as that of the observation on qualitative evaluation.

To further evaluate the quality of transformed pose quantitatively and show potential downstream application of FlexPose, we show how can we annotated unlabelled human-related dataset with the help of FlexPose.

Quantitatively, we compare their performance with FlexPose on human pose estimation in Table 2. As shown in Table 2, the Baseline has much lower performance on the target dataset as the pose annotations are from different datasets, especially when some of them have a distinct pose distribution from that of the target dataset, e.g., when 3DHP or SURREAL is the source dataset and Simplified Human3.6M is the target one. FlexPose gets better results on all settings under both metrics. FlexPose largely reduces the performance gap when the pose distribution of the source dataset is very different from that of the target distribution, e.g., MSE 12.7$\rightarrow$6.0 and PCK10 0.00$\rightarrow$0.47 when adaptation occurs from 3DHP to Simplified Human3.6M. The results show that FlexPose is effective at generating similar poses with the target dataset, even with less to only two poses per class in the target dataset.

Introducing FlexPose to human face landmarks transfer is straightforward since both the human pose and the human face consist of a set of pre-determined keypoints.

We show the detected human face landmarks in Figure 7. The human face detector trained with generated face landmarks can handle human faces in different directions well.

In Table 4, ablation studies are conducted: