Knowledge as Priors: Cross-Modal Knowledge Generalization
for Datasets without Superior Knowledge

From a Bayesian perspective, a neural network can be viewed as a probabilistic model $P(\mathbf{y}_{i}|\mathbf{x}_{i},\boldsymbol{\theta})$: given an input $\mathbf{x}_{i}$, the network assigns a probability to each possible $\mathbf{y}_{i}\in\mathcal{Y}$ with the parameters $\boldsymbol{\theta}$. Here, we consider a regression problem where $P(\mathbf{y}_{i}|\mathbf{x}_{i},\boldsymbol{\theta})$ is a Gaussian distribution which corresponds to a mean squared loss, and $\mathbf{x}_{i}$ is mapped onto the parameters of a distribution on $\mathcal{Y}$ using network layers parameterized by $\boldsymbol{\theta}$. Given a dataset $\mathcal{D}={\{\mathbf{x}_{i},\mathbf{y}_{i}\}}_{i}$, $\boldsymbol{\theta}$ can be learned by maximum likelihood estimation (MLE):

We assume that $\log P(\mathcal{D}|\boldsymbol{\theta})$ is differentiable w.r.t. $\boldsymbol{\theta}$, and then Eq. (5) is typically solved by gradient descent.

The objective of cross-modal knowledge generalization is to find the parameters $\boldsymbol{\theta}_{\text{DIST}}$ by using the training examples in $\mathcal{D}_{T}$ with intractable knowledge $\mathcal{K}_{T}$. This leads to maximize the posterior density of the parameters $\boldsymbol{\theta}$ directly depends on $\mathcal{D}_{T}$ and implicitly depends on $\mathcal{K}_{T}$. In order to explicitly capture this dependence, we introduce a latent variable $\boldsymbol{\phi}$ summarizing the knowledge carried by $\mathcal{K}_{T}$:

Note that the last equation is the result of assuming that $\mathcal{K}_{T}$ and $\boldsymbol{\theta}$ are conditionally independent given the latent variable $\boldsymbol{\phi}$. Since both $\mathcal{K}_{T}$ and integrating Eq. (6) over $\boldsymbol{\phi}$ are intractable, we make an approximation that uses a point estimation $\boldsymbol{\phi}_{\text{META}}$. This point estimation is obtained via the meta-learning approach described in Sect. 4.2, hence avoiding the need to perform integration over $\boldsymbol{\phi}$ or interact $\mathcal{K}_{T}$. Consequently, maximizing the logarithm of the posterior density of Eq. (6) can be written as:

where the last equality results from a direct application of Bayes rule. So, finding the parameters $\boldsymbol{\theta}_{\text{DIST}}$ involves a two step training procedure: (1) optimizing the prior term which obtains the point estimation $\boldsymbol{\phi}_{\text{META}}$ using meta-learning and (2) optimizing the likelihood term which maximizes Eq. (7) using the learned parameters $\boldsymbol{\phi}_{\text{META}}$.

In a Bayesian setting, priors on the parameters can be interpreted as regularization. Thus the prior term in Eq. (7) is implemented as a regularizer during network training. Several other regularization schemes have been proposed in the literature such as weight decay [20], dropout [42, 48] and batch normalization [17]. While they aim to reduce error on examples drawn from the test distribution, the objective of our work is to learn a regularizer that captures cross-modal knowledge learned from the source dataset.

As mentioned above, we model the prior term as a regularizer $\mathcal{R}(\boldsymbol{\theta};\boldsymbol{\phi})$. Given the input $\boldsymbol{\theta}$, $\mathcal{R}$ is implemented with a neural network parameterized by $\boldsymbol{\phi}$.

As described in Sect. 3, cross-modal knowledge distillation leads to optimize the following objective:

where $\mathcal{L}_{\text{REG}}$ is the regression loss minimizing the mean squared errors of the prediction and ground truth, and the distillation loss $\mathcal{L}_{\text{DIST}}$ distills knowledge from the teacher to student. Using the regularizer $\mathcal{R}$, we introduce a regularized regression loss which is defined as:

During the training procedure on the source dataset, we aim to learn the regularizer $\mathcal{R}$ in Eq. (9) which mimics the behavior of $\mathcal{L}_{\text{DIST}}$ in Eq. (8), so that $\mathcal{F}$ can be applied to the target dataset where the superior knowledge is missing. We now describe the procedure for learning $\mathcal{R}$.

When training the student network on the source dataset, at iteration $k$, we begin by sampling a mini-batch from the dataset. Using this batch, $l$ steps of gradient descent are first performed with the regularized regression loss $\mathcal{F}$. Let $\ddot{\boldsymbol{\theta}}_{k}$ denote the network parameters after these $l$ steps. Then the full loss $\mathcal{G}$ on the same batch computed using $\ddot{\boldsymbol{\theta}}_{k}$ is minimized w.r.t. the regularizer parameters $\boldsymbol{\phi}$. The regularizer $\mathcal{R}$ is finally updated with the gradients which unroll through the $l$ gradient steps. This ensures that $\mathcal{G}$ can be approximated by $\mathcal{F}$ using $\mathcal{R}$. After finishing the training, since the same regularizer is trained on every pair of $(\mathbf{x}_{i},\tilde{\mathbf{x}}_{i})$, the resulting $\mathcal{R}$ captures the notion of cross-modal knowledge contained in the source dataset. Please refer to Fig. 3 for an illustration of the meta-training step. The entire algorithm is given in Algorithm 1. Note that $l$ is set to 1 empirically in this paper, as we observe that a $1$-step update is sufficient to achieve good performance.

Once the regularizer is learned, its parameters $\boldsymbol{\phi}_{\text{META}}$ are frozen and the final student network initialized from scratch is trained on the target dataset using the regularized loss function $\mathcal{F}$. This meta-testing procedure generalizes the learned knowledge to the target dataset with $\mathcal{R}$ parameterized by $\boldsymbol{\phi}_{\text{META}}$ as summarized in Algorithm 2.

Our meta-learning approach is general and can be implemented by any type of regularizer. In this paper, we use weighted $\ell^{2}$ loss as our regularization function:

where $\phi_{i}$ and $\theta_{i}$ are the $i$-th weight and parameter of the network. The use of weighted $\ell^{2}$ loss can be interpreted as a learnable weight decay mechanism: weights $\theta_{i}$ for which $\phi_{i}$ is large will be decayed to zero and those for which $\phi_{i}$ is small will be boosted. By using our meta-learning approach, we select a set of weights that carry cross-modal knowledge across every pair of inputs $(\mathbf{x}_{i},\tilde{\mathbf{x}}_{i})$.

This section gives a theoretical understanding of Algorithm 1 in Sect. 4.2. We draw its connection to the expectation maximization (EM) algorithm and thus its convergence is theoretically guaranteed. To achieve this, we first derive the lower bound of the target objective and then show how it is solved by our meta-learning algorithm using EM.

In a Bayesian framework, given the evidence $\mathcal{D}_{S}$, learning the parameters $\boldsymbol{\phi}$ of priors leads to maximize the likelihood $P(\mathcal{D}_{S}|\boldsymbol{\phi})$. Proposition 1 indicates its lower bound.

Note that $\text{KL}[\cdot\|\cdot]$ in Eq. (12) represents the KL divergence between two distributions $q$ and $P$. The proof to this proposition can be found in our supplementary material. According to Proposition 1, the following proposition shows that Algorithm 1 is an instance of EM maximizing $\mathcal{E}$.

For simplicity, we use the same architecture for teacher and student networks. We choose ResNet [15] as the backbone, and adjust the final fully connected layer to output a vector representing the 3D positions of 21 hand joints. All corresponding depth maps of RGB images in the dataset are employed as the modality containing superior knowledge.

Data Augmentation. Recent methods [4, 12, 47] show that learning from synthetic data improves the performance of 3D pose estimation, as it offers more effective hand variations than traditional data augmentation technologies, e.g., random cropping and rotation. Hence, we create a synthetic dataset of paired hand images and depth maps with their 3D annotations using the MANO [37] hand model for synthetic data augmentation. Following the setting of [4], hand geometries are obtained by sampling pose and shape parameters from $[-2,2]^{10}$ and $[-0.03,0.03]^{10}$, respectively. Meanwhile, hand appearances are modeled by the original scans with 3D coordinates and RGB values from [37]. We create example hand appearances using these registered scan topologies. After rotations, translations and scalings are applied to hand models, the textured hands are finally rendered on background images which are randomly sampled and cropped from [18, 24]. In total, we synthesize 50,000 hand images with large variations for training.

Network Training. The input image is resized to $256\times 256$. For CMKD, all networks are trained using Adam [19] with mini-batches of size 32. The learning rate is set as $2.5\times 10^{-4}$. The teacher is pre-trained for 200 epochs, while the student is trained with only the regression loss for 100 epochs and then fine-tuned with the full loss for another 100 epochs. For CMKG, the regularizer is optimized using Stochastic Gradient Descent (SGD) with the learning rate of $1.0\times 10^{-3}$ during the fine-tuning of the student network.

Our proposed approach is comprehensively evaluated on two publicly available datasets for 3D hand pose estimation: RHD [61] and STB [56] with the standard metrics.

Datasets. Rendered Hand Pose Dataset (RHD) [61] is a synthetic dataset built upon 20 different characters performing 39 actions. It provides 41,258 images for training and 2,728 images for evaluation with a resolution of $320\times 320$. All of them are fully annotated with a 21 joint skeleton hand model and additionally the depth map for each hand. This dataset is challenging due to the large variations in viewpoints and textures. We employ RHD for training and evaluating our knowledge distillation method.

Stereo Hand Pose Tracking Benchmark (STB) [56] is a real-world dataset which contains 18,000 stereo image pairs as well as the ground truth 3D positions of 21 hand joints from different scenarios. This benchmark has 12 different sequences and every sequence contains 1,500 stereo pairs. Following the evaluation protocol of [5, 12, 40, 61], we use the sequence of B1 for evaluation and the others for training. STB is utilized for evaluating the proposed cross-modal knowledge generation algorithm.

To make the joint definition consistent across different datasets, we reorganize the joints of each hand according to the layout of MANO [37]. Especially, we move the root joint location from palm center to wrist of each hand in STB. Following the same protocol used in [5, 12, 40, 61], the absolute depth of root joint (wrist) and global hand scale, which is set as the bone length between MCP and PIP joints of the middle finger, are provided at test time.

Metrics. We evaluate the performance of 3D hand pose estimation with three common metrics in the literature: (1) EPE: the mean hand joint error which measures the average Euclidean distance in millimeters (mm) between the predicted 3D joints and the ground truth; (2) 3D PCK: the percentage of correct key points which are within the Euclidean distance of a certain threshold to its respective ground truth position; (3) AUC: the area under the curve on 3D PCK for different error thresholds.

To evaluate the performance of the proposed knowledge distillation approach for 3D hand pose estimation, we train three networks for each setting: a baseline network trained with RGB images (RGB), a teacher network trained with depth maps (Depth) and a student network trained using the knowledge distillation algorithm presented in Sect. 3 (KD). All the experiments are conducted on RHD dataset.

Ablation Study. We first evaluate the impacts of different losses used in knowledge distillation, data augmentation, and network architecture on the performance of 3D hand pose estimation. The results of EPE are presented in Table 1. We can see that the model trained with the full distillation loss ($\mathcal{L}_{\text{ACT}}$ and $\mathcal{L}_{\text{ATT}}$) achieves higher performance improvement, from 1.27 (mm) to 2.49 (mm), which indicates that all the losses have contributions to distilling cross-modal knowledge from depth maps for 3D hand pose estimation. Moreover, synthetic data augmentation and employing deeper network during the training procedure can further boost the performance.

Comparison to State of the Art. We compare the 3D PCK curves with state-of-the-art methods [5, 40, 53, 54, 61] on RHD dataset in Fig. 4. We use ResNet-50 as the backbone. Note that some other works [4, 12, 58] aim to predict the 3D hand shape other than hand joint locations, which are with different research targets compared with ours. Therefore, they are not included here. In Fig. 4 (left), our method surpasses most existing methods except [5], which has a higher AUC of 0.015. However, it is not directly comparable, as [5] incorporates 2D annotations as an additional supervision during network training.

In Fig. 4 (middle), we further compare our approach to Yuan et al. [54] which is the most related work also distilling cross-modal knowledge from depth maps for 3D hand pose estimation. We can find that our method substantially outperforms [54]. More importantly, the performance gain achieved by our approach ($\Delta_{\text{AUC}}=0.045$) is larger than [54] ($\Delta_{\text{AUC}}=0.041$), which shows that the proposed knowledge distillation algorithm is more efficient.

In order to evaluate the effectiveness of the proposed knowledge generalization algorithm, we transfer the learned cross-modal knowledge in RHD to STB and compare our approach to other regularization functions.

Effect of Regularizers. In this experiment, we study the effect of different regularizers including the proposed $\mathcal{R}$ in Eq. (10) on the performance of network trained on STB. We compare our formulation with the default regularizers commonly used in the literature: $\sigma\sum_{i}\|\phi_{i}\|^{p}$, where $\|\cdot\|^{p}$ is the $p$-norm of the parameter and $\sigma$ is a constant weight manually selected for each network. We experiment on the $\ell^{1}$ and $\ell^{2}$ regularizers (where $p$ equals 1 or 2, respectively) and different choices of $\sigma$. We also implement a variant of the proposed $\mathcal{R}$ which is $\ell^{1}$-regularized. The performance of these regularizers are reported in Table 2. We observe that our proposed regularizers outperform the default regularization functions by a large margin. Especially, our $\ell^{2}$-regularized $\mathcal{R}$ achieves the best performance. These results demonstrate that $\mathcal{R}$ carries effective knowledge learned from the source dataset which helps the training of the target network.

Visualization of Parameters. To give an intuitive understanding of how our regularizer $\mathcal{R}$ affects the network learning, we plot the histograms of the parameters learned by the network with and without the use of $\mathcal{R}$ in Fig. 5. We can make the following observations. First, for the network trained with regularization, there is a sharper peak at zero. This is due to the positive $\phi_{i}$ in Eq. (10) which decays the corresponding $\theta_{i}$ to zero. Second, on the other hand, the parameters of the network with regularization have wider spread, since they are boosted by the negative $\phi_{i}$.

Comparison to State of the Art. We further compare the proposed CMKG to other approaches [27, 31, 40, 61] on STB in Fig. 4 (right). We can see that our regularized network matches the state-of-the-art performance without using complex network architecture, loss functions or additional constraints like previous methods. Our visual results are shown in Fig. 6. As seen, our method is able to accurately predict 3D hand poses across different datasets and generalize the learned knowledge to some novel cases.

One potential concern about the learned knowledge (regularizer) from the source dataset is how it performs when applied to different target datasets. First of all, we point out that it is impossible to learn a domain-independent regularizer from a single source which performs consistently well on all other datasets, since their data usually follow different statistics. Here, we hypothesize that the effect of the learned regularizer depends on two factors: (1) the domain shift between the source and target dataset, and (2) the effect of regularization on the target dataset.

The first factor is straightforward as large domain shifts always lead to difficulties in network generalization. This is a well-defined problem in transfer learning which is tackled by domain adaptation [30, 33, 34]. To illustrate the second factor, we conduct an additional experiment which applies the same regularizers in Sect. 5.4 to a number of different target datasets. Due to the space limitation, we ask the readers to refer to the supplementary material for detailed setups of this experiment. Looking at Table 3, we see a strong correlation between the default and the proposed regularizer: if there is a large increase obtained by the default regularizer, $\mathcal{R}$ can boost the performance even further; otherwise, our improvement is limited. This is intuitive since our formulate is consist with the default regularization technique.

Our findings suggest multiple directions of future work. For one, the proposed scheme currently has access to only one single source dataset; we believe that learning from multiple sources will result in better generalizability of the model. On the other hand, we treat target priors as a regularization term in this work, which is perhaps the simplest formulation. We believe that a further exploration on choices of this term will result in improved performance.