Prior-Aware Synthetic Data to the Rescue:
Animal Pose Estimation with Very Limited Real Data

The most common statement in animal pose estimation articles is the lack of labeled dataset for training, which has been mentioned in many works [Wu et al.(2020)Wu, Buchanan, Whiteway, Schartner, Meijer, Noel, Rodriguez, Everett, Norovich, Schaffer, et al., Cao et al.(2019)Cao, Tang, Fang, Shen, Lu, and Tai, Li and Lee(2021), Mu et al.(2020)Mu, Qiu, Hager, and Yuille, Mathis et al.(2018)Mathis, Mamidanna, Cury, Abe, Murthy, Mathis, and Bethge, Graving et al.(2019)Graving, Chae, Naik, Li, Koger, Costelloe, and Couzin]. Numerable variety of species and subspecies, and considerable differences in physical characteristics and behavior patterns between them cause it difficult to form a labeled dataset with adequate samples. The cross-domain adaptation challenge exacerbates the situation. For each new animal, it is necessary to collect data from scratch and label them, since it is insufficient to learn the prior knowledge exclusively from the data of other animals. [Cao et al.(2019)Cao, Tang, Fang, Shen, Lu, and Tai] proposed a cross-domain adaptive method and a large multi-species dataset, Animal-Pose [Jinkun et al.(2019)Jinkun, Hongyang, Hao-Shu, Xiaoyong, Cewu, and Yu-Wing] to learn the shared feature space between human and animals, in order to transfer the prior knowledge between them. [Yu et al.(2021)Yu, Xu, Zhang, Zhao, Guan, and Tao] built their own large multi-species dataset, called AP10k [Yu et al.(2021)Yu, Xu, Zhang, Zhao, Guan, and Tao] in order to train a robust and cross-domain model. However, both models are still unable to achieve the same level of pose prediction as animals in-domain when facing unseen species.

Synthetic data is a promising substitute for real data in previous works. Coarse prior can be learnt and then it would be used to build pseudo-labels for enormous amounts of unlabeled real animal data [Mu et al.(2020)Mu, Qiu, Hager, and Yuille, Li and Lee(2021)]. The fly in the ointment is that works such as [Mu et al.(2020)Mu, Qiu, Hager, and Yuille, Li and Lee(2021)] still use significant amounts of real data (such as TigDog dataset [Pero et al.(2016)Pero, Susanna, Rahul, and Vittorio]) in training, which may not be possible to access for the unseen species. The work in [Zuffi et al.(2017)Zuffi, Kanazawa, Jacobs, and Black], which focuses on animal model recovery, also gives an extraordinary hint on this problem. It purposed a general 3D animal model (called SMAL) by learning the animal toys’ scan, and use the SMAL model and a few real pictures to generate a large amount of realistic synthetic data by adjusting the model’s texture, shape, pose, background, camera, and other parameters. They also trained an end-to-end network [Zuffi et al.(2019)Zuffi, Kanazawa, Berger-Wolf, and Black] using only synthetic Grevy’s zebra data, which came from direct reconstruction from animal 2D images. However, their results are much worse than the current state-of-the-art in terms of 2D animal pose estimation.

Although synthetic data has advantages in terms of cost, it still faces the problem of fatal domain gap with the real data, which is even challenging to alleviate by current domain adaptation methods [Ganin et al.(2016)Ganin, Ustinova, Ajakan, Germain, Larochelle, Laviolette, Marchand, and Lempitsky]. Pasting synthetic animal directly to the real background will result in the strong incongruity, mainly from the difference in projection, brightness, contrast, saturation, etc., between the two images. This incongruity can lead to excessive domain differences between synthetic and real data. Besides, to increase the texture diversity of synthetic animals to enhance the appearance robustness of the model, a common approach is to assign the synthetic data random textures from ImageNet. However, this further increases the discrepancy between synthetic and real domains and thus weakens the improvement of synthetic data in joint training of pose estimation tasks. The approach to increase model robustness and reduce domain variance by style transfer has been applied in medicine [Wagner et al.(2021)Wagner, Khalili, Sharma, Boxberg, Marr, Back, and Peng], monocular depth estimation [Atapour-Abarghouei and Breckon(2018)], person re-identification [Chong et al.(2021)Chong, Peng, Zhang, and Pan]. We, therefore, adopt style transfer to alleviate the domain gap between the synthetic animal and the real background while increasing the texture diversity of the synthetic animal.

As evident based on the sample results shown in Fig. 1, when the real target data provided to a pose estimation model is scarce, to improve the model performance, the prior knowledge of the target animal pose can be learned from a carefully curated synthetic dataset. To learn the animal pose priors, we use a variational autoencoder (VAE) framework. VAE has already been proven to be useful for human pose priors learning [Pavlakos et al.(2019)Pavlakos, Choutas, Ghorbani, Bolkart, Osman, Tzionas, and Black], when it is trained on large-scale 3D human pose datasets. However, appropriate 3D datasets cannot be easily obtained for animal pose studies. Therefore, we feed the animation of multiple simulated animals made by human animators (inexpensively purchased from the CGTrader website), to the VAE, for it to learn the probabilistic distribution of the feasible poses of the targeted animals. Then, the trained VAE is used as a generative model to create our 3D synthetic animal pose (SynAP) dataset. As seen in Fig. 3(a), the legs of quadrupeds like horses and dogs are similar in structure. This structure is obtained by simplifying the leg skeleton. Due to its applicability across quadrupeds, this structure is useful for training animal pose priors. Angles between neighboring bones (12 of them) in the structure shown in Fig. 3(c) are chosen as data for training animal leg pose priors to minimise the influence of model sizes or bone lengths on training.

Our network follows the basic VAE process, including input, encoder, random sampling, decoder, and output as demonstrated in Fig. 4. $A\in\mathbb{R}^{36}$ is a $1\times 36$ vector containing the angles between 12 pairs of adjacent bones, and each space angle is decomposed into three directions, and $\hat{A}$ is a vector of the same shape representing the generated angles as the model output. The role of the encoder which is composed of dense layers with rectified linear unit (ReLU) activations is to calculate the mean $\mu$ and variance $\sigma$ of the network input. We use the reparameterization trick to randomly sample $z$ in latent space $Z\in\mathbb{R}^{16}$, where $z=\mu+\epsilon\times\sigma$ and $e$ satisfies the distribution in the shape of $\mathcal{N}(0,I)$. Finally, the decoder which has a mirror structure of encoder is used to reconstruct the set of space angle, and make the reconstruction results $\hat{A}$ as close to $A$ as possible. Considering that one infeasible angle can ruin an entire pose, during the pre-test, we first assign random sampling from a normal distribution to the variable $z$, and then we create the histograms of the 12 joint angles using enough $\hat{A}$ ($\approx$10,000 poses). Based on the statistical results, we can establish the sampling range for each angle. For each range, 5% are excluded of total angles, which are far from the mean value, and all of the ranges are recorded. After the pre-test, when we generate the new poses, a ‘pose filter’ is designed to remove the pose $\hat{A}$ as long as there is an angle outside these specified ranges and finally produce the refined poses $\hat{A^{\prime}}$. We can also obtain poses with higher diversity by increasing the variance of the sampling distribution after determining the pose filter. The training loss of the VAE is:

		$\displaystyle\mathcal{L}_{\text{total}}=w_{1}\mathcal{L}_{KL}+w_{2}\mathcal{L}_{\text{rec}},$
	$\displaystyle\mathcal{L}_{KL}=KL(q(z|A)||$	$\displaystyle\mathcal{N}(0,I)),\,\,\,\text{and}\,\,\,\mathcal{L}_{\text{rec}}=\|A-\hat{A}\|_{2}^{2},$		(1)

where $w_{i}$ is the weight of each loss term. The Kullback-Leibler term, $\mathcal{L}_{KL}$, represent the divergence between the encoder’s distribution $q(z|A)$ and $\mathcal{N}(0,I))$. $\mathcal{L}_{\text{rec}}$ is the reconstruction term. $\mathcal{L}_{KL}$ encourages a normal distribution noise while $\mathcal{L}_{\text{rec}}$, in contrast, encourages to reconstruct the $A$ without any divergence.

As shown in Fig. 2, we use the large number of new synthetic poses obtained through VAE to rig the 3D zebra models and render synthetic animal images by changing parameters such as lighting and camera angles. Then, we assign a real (and context-related) background from forest scenes to each image. To alleviate the domain gap between the background image $I_{sty}$ (style) and synthetic animal image $I_{cont}$ (content), while increasing the texture diversity of the synthetic animal we employ a style transfer technique. Unlike common convolution-only style transfer methods, we adopt an innovative image style transfer method, called StyTr² [Deng et al.(2022)Deng, Tang, Dong, Ma, Pan, Wang, and Xu] based on multi-layer transformers. The $I_{sty}$ and $I_{cont}$ images are segmented into patches to generate feature embedding and eventually formed encoded content sequence by adding positional embedding for each patch. This process is calculated through the content-aware positional encoding based on the semantics of the image content, thereby eliminating the negative effect of scaling on the spatial relationship between patches. Two transformers are applied to encode the content sequence and style sequence, respectively. Then, a transformer decoder is utilized to decode the encoded content sequence according to the encoded style sequence in a regressive fashion. Finally a convolution-based decoder is used to decode the output sequence, consisting of a linear combination of encoded content and style sequence as well as positional embedding, to obtain the stylized content image $I^{sty}_{cont}$. Due to strong representation ability of transformers, this method can better capture accurate content representations and avoid loss of details than classical methods.

However, StyTr² cannot freely adjust the intensity of style transfer, unlike the adaptive instance normalization (AdaIN) [Huang and Belongie(2017)]. Besides StyTr², we also perform a simple pixel summation with normalization of the stylized synthetic animal data with the original data to control the intensity of style transfer with the fusion rate $\alpha$. The final stylized synthetic animal image is formulated as $I^{final}_{cont}=(1-\alpha)I_{cont}+\alpha I^{sty}_{cont}$.

Synthetic Animal Pose (SynAP) dataset contains 3,000 synthetic animal images generated by our prior-aware synthetic animal (PASyn) pipeline. Since zebra was selected as the main species for quantitative evaluation in this work, we collected 5 zebra toys and textured zebra 3D models and synthesized the images from these models with Blender. 3D synthetic model generation pipeline (3D-SMG) introduced in [Vyas et al.(2021)Vyas, Jiang, Liu, and Ostadabbas] was used to reconstruct two models from zebra toys, while the remaining 3 are adapted from 3D models. In addition to SynAP, which only contains zebra data, SynAP+ extends the SynAP with 3,600 images of 6 other animals, including horses, cows, sheep, dogs, giraffes and deer. We purchased one textured 3D model for each animal from a 3D model website, CGTrader and rendered 600 images for each of the aforementioned animals. The VAE model generates the animal pose in each image and 300 grass, savanna, and forest real scenes are collected from Internet to stylize the synthetic animal.The annotations will be automatically created with each image through calculating the coordinates of each zebra joint in the pose using bone vector transformations.

PASyn Pipeline: We keep the setting in Vposer [Pavlakos et al.(2019)Pavlakos, Choutas, Ghorbani, Bolkart, Osman, Tzionas, and Black] for training the VAE model. The learning rate of Adam optimizer is 0.001 and we set $w_{1}$ and $w_{2}$ which are the weights of $\mathcal{L}_{KL}$ and $\mathcal{L}_{\text{rec}}$ in Equation (3.1), as 0.005 and 0.01, respectively. The model is trained on 600 poses of animated realistic 3D models for 200 epochs with 128 poses in a batch. The models include horse, dog, sheep, cow, and deer. Each of these animals has over 20 common actions designed by 3D animators, such as running, walking, and jumping. To increase the diversity of the poses, we choose to generate random samples from a Gaussian distribution $\mathcal{N}(0,2I)$. The sampling value range of the pose filter will be described in the supplementary. For our domain transfer part, we use the pre-trained transformer decoder provided by [Deng et al.(2022)Deng, Tang, Dong, Ma, Pan, Wang, and Xu] and set the fusion rate as 0.5.
Backbone Pose Models: We employ several state-of-the-art pose estimation structures with varying complexity as our backbone networks, including the ResNet-50 and HRNet-W32 of AP10K [Yu et al.(2021)Yu, Xu, Zhang, Zhao, Guan, and Tao] and EfficientNet-B6 of DeepLabCut [Mathis et al.(2018)Mathis, Mamidanna, Cury, Abe, Murthy, Mathis, and Bethge] to reflect the general effect of our PASyn pipeline. All of the backbones are pre-trained on ImageNet [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei].

Zebra-300 Dataset is the primary test set we use. It contains 40 images from the AP10K test set, 160 unlabeled images randomly selected from AP10K, and 100 images from the Grevy’s zebra dataset [Zuffi et al.(2019)Zuffi, Kanazawa, Berger-Wolf, and Black]. We labeled the images of the latter two according to the AP10K dataset. The images in Zebra-300 dataset are mostly taken in the wild, so the environment occlusion makes the pose estimation task challenging.
Zoo Zebra Dataset is made up of pictures and videos we took at a zoo. The dataset contains 100 zebra images, including 2 different mountain zebra individuals. We performed manual animal pose labeling on the cropped and resized images following the label setting of the AP10K dataset. Each label has 17 key points: nose, eyes, neck, shoulders, elbows, front paws, hips, knees, back paws and root of tail. The occlusion of barbed wire makes this dataset more challenging. We will release this dataset along with the paper.

The number of real (R) and synthetic (S) data shown in training set in Table 1 means the total number of images we used for model training. SynAP and SynAP+ are divided into training and validation sets with a ratio of 7 to 1 as the AP10K setting. For real data, considering the data scarcity, we divide it with a ratio of 4 to 1. From Table 1, we can clearly know that only using 99 images for training is insufficient for any backbone network. The highest prediction accuracy is 83.2%, reached by EfficientNet-B6. After adding SynAP to the training set, the prediction results dramatically increase to around 90%. And after adding SynAP+ to the training set, the accuracy is further improved. Among them, HRNet-w32 achieves the highest accuracy of 92.4% when trained with SynAP.

Different animals have domain similarity and the possibility of mutual transfer learning, which is proven in many works [Cao et al.(2019)Cao, Tang, Fang, Shen, Lu, and Tai, Yu et al.(2021)Yu, Xu, Zhang, Zhao, Guan, and Tao]. Thus, we cannot ignore the existing labeled data, which can improve the prediction of unseen animal. In Table 1 and Table 2, we can see that training the model with 99 zebra images and 8,000 images of the other animals can increase the average PCK from 78.7% to 91.4%. However, Table 1 shows that even with only 99 real we are still able to surpass that result and reach SOTA by adding SynAP and SynAP+. Moreover, Table 2 points that the models trained with 8,000 real and SynAP or SynAP+ is further improved to 93.8% and 94.2%. When there is no target animal (i.e., zebra) in the training set, while the state-of-the-art model suffers from this situation (accuracy drops to 78.3%), the model can recover its performance if given the SynAP dataset, as Table 2 shows. This confirms that our method can achieve high-precision prediction even for the unseen animals.

As seen from Table 1, even if only trained on a small amount of real data, the model can accurately predict the keypoints with apparent features and less flexibility, such as nose and eyes. Various alternative candidate positions and large degrees of freedom of keypoints on the limbs would considerably affect the prediction results. The main improvement brought by SynAP and SynAP+ is the significant increase of the prediction of keypoints in animal limbs. To verify that the VAE, StyTr² models [Deng et al.(2022)Deng, Tang, Dong, Ma, Pan, Wang, and Xu], and the higher variance of VAE sampling distribution in our PASyn pipeline can mitigate the mismatching prediction of the limbs and reduce the domain discrepancy between the real (R) and synthetic (S) domains, we test them on the Zoo Zebra and Zebra-300 datasets. Table 3 shows the outcomes of seven experiments (a-g) with various training conditions. When the model trained without VAE, we will manually set a reasonable value range for the angles between adjacent bones on the limbs, and conduct randomly sampling, similar to the method used in [Zuffi et al.(2019)Zuffi, Kanazawa, Berger-Wolf, and Black, Mu et al.(2020)Mu, Qiu, Hager, and Yuille]. We will keep the original texture of the model when StyTr² is not used. The various training and test sets clearly reflect that the model trained with both of VAE and StyTr² in our PASyn pipeline can achieve obviously higher accuracy than the model trained without them. Also, the effect of variance of the random sampling distribution, $\sigma^{2}$ on the pose estimation performance is shown in experiments (g) and (h). The higher diversity of the poses can improve the pose estimation, hence our choice of $\sigma^{2}=2I$.

Based on the Fig. S1, which is made by decoding of 10,000 random samples from a normal Gaussian distribution, we can set a special sampling range for each joint. The angle value within this range can be regarded as a valid angle, and the pose that satisfies the twelve ranges can be seen as an appropriate pose. These ranges are shoulder_right [40, 100], elbow_right [-125, 0], front-paw_right [-25, 100], shoulder_left [40, 100], elbow_left [-125, 0], front-paw_left [-25, 100], hip_right [-120, -60], knee_right [0, 80], back-paw_right [-125, 0], hip_left [-120, -60], knee_left [0, 80], back-paw_left [-125, 0]. In order to increase the variety of poses and generate more poses with angles near the boundary, we choose to generate random samples from a Gaussian distribution $\mathcal{N}(0,2I)$ after setting the pose filter. The dropout rate of the pose is 68.0%.

We explored the effect of the size of other synthetic animal data on target animal’s pose predictions during training. The result is shown in Fig. S2.