Iterative Optimisation with an Innovation CNN for Pose Refinement

End-to-End Regression: End-to-end pose regression from a single image is a highly challenging task due to the nonlinearity of the space of rigid 3D rotations $SO(3)$. This is addressed in [KGC15] by separating and carefully balancing translation and rotation terms in the loss function, although this is for camera pose estimation, a slightly different problem. More recently, [XSNF17] decouple or ‘disentangle’ translation and rotation terms by moving the centre of rotation to the centre of the object and representing translation in 2D image space. In practice, regression to 3D orientation has had limited success [MAV17, SMD⁺19]. This is partly due to such methods only covering the small subsection of pose space that is seen during training [KMT⁺17], which has in turn led to discretisation of the pose space as discussed below. Recently, [WMS⁺20] obtained state-of-the-art results on real data by separately regressing rotation and translation within a self-supervised framework that also utlises RGBD images.

Pose Classification: Pose classification has typically involved discretising $SO(3)$, while the translation component is obtained via regression. However, as even a coarse discretisation of $SO(3)$ ($\sim 5^{\circ}$ precision) leads to over 50,000 possible cases [SMD⁺19]. Kehl et al. [KMT⁺17] approach this problem by decomposing 6D pose into viewpoint and discretised in-plane rotation. However, as noted in [SMD⁺19], this approach leads to ambiguous classifications as a change of viewpoint can be nearly equivalent to a change of in-plane rotation. Furthermore, such classification leads to coarse pose estimates that require further refinement [KMT⁺17, LWJ⁺19a], as discussed below.

Pose via an Intermediate Representation: A recent, popular approach to pose estimation is to first regress to an intermediate representation such as keypoints, from which pose can be obtained via 2D-3D correspondences and a PnP algorithm [TSF17, RL17, PLH⁺19, SSH20, ZSI19, lWJ19b, HBM20]. Of these approaches, some, including [TSF17, RL17] regress to a set of bounding box corners. Such methods encounter difficulty when objects are occluded or truncated. Alternatively, [PLH⁺19] predicts pixel-wise unit-vectors that in turn indicate direction to keypoints. Song et al. [SSH20] implements these vector directions from [PLH⁺19] while also estimating object edge vectors and symmetry correspondences. Zakharov et al. [ZSI19] apply texture to objects with a 2D image generated from spherical or cylindrical projections, and uses these to estimate dense correspondences. Li et al. [lWJ19b] apply this intermediate representation approach to estimate rotation, but estimates translation separately via regression.

Of the networks discussed above, several have additional refinement steps that can be added to improve results. For example, DeepIM [LWJ⁺19a] is presented as a refinement step for POSECNN [XSNF17], the work [MKNT18] is presented as a refinement step for SSD6D [KMT⁺17], and DPOD [ZSI19] is presented along with a refinement step. In each case a synthetic image of the target object is rendered using the initial pose estimate and a 3D model of the target object.

The key difference in these approaches is in the representation of the relative pose. DeepIM [LWJ⁺19a] utilise an ‘untangled representation’ in which the centre of the object is the centre of rotation, and translation is estimated in pixel space. CosyPose [LCAS20] improve upon DeepIM by utilising a more recent feature detection network, and implement their framework to estimate object poses in a multi-view scene. Manhardt et al. [MKNT18] represent rotation as a unit quaternion and translation as a vector in $\mathbb{R}^{3}$. DPOD [ZSI19] combines these approaches by estimating rotation relative to the object centre, and estimating translation as a vector in $\mathbb{R}^{3}$. These approaches therefore differ mainly in their loss functions, and all require textured 3D object models.

Motivated by filtering and optimisation principles [AM79], we treat object pose estimation as an offline state estimation task. In this context a state is an internal representation of a system that is sufficient to fully define the future evolution of all system variables. A state estimation algorithm is a dynamical system that takes measurements from the true system as inputs and whose state is an estimate of the true system state.

Consider a dynamical system in a state $\bm{X}$ defined by

where $f(.)$ is the state model $\textbf{V}(t)$ is an measured input signal, $h:\bm{X}(t)\to\mathbf{y}(t)$ is an output map, and $t$ represents the current time. We think of the output $\mathbf{y}(t)$ as encoding the specific target information that the engineer is interested in, while the state $\mathbf{X}$ encodes the full information necessary for the propagation of the dynamical system.

The class of estimation algorithm considered in this paper are of the form

where $\Delta$ is the innovation term that is chosen so as to drive the estimated outputs towards the true outputs, and $\widehat{\cdot}$ represents an estimation.

The proposed approach involves taking the output state from an existing object pose estimation network and refining this output using a state estimation framework to obtain more accurate pose estimates. We choose PVNet [PLH⁺19] to be the baseline object pose estimation network. In established algorithms such as PVNet, the state is estimated directly via a deep neural network. A typical formulation for such a problem can be written

where the estimator $\mathcal{E}$ is implemented as a CNN or more classical computer vision algorithm.

The output of PVNet is a unit vector field, with vectors pointing from each pixel to each of a set of keypoints. The vectors are defined to be

where $\bm{\xi}_{ij}$ denotes a 2D pixel with coordinates $(i,j)$ within an image $\mathcal{I}$ with dimensions $M\times N$, and $\bm{\xi}^{k}$ represents the pixel location of keypoint $k\in\mathcal{K}$. The unit vector field is

This unit vector field forms the state for the estimation problem.

We assume a static state model

Approximating the time derivative by $\mathbf{\dot{X}}=\delta t(\mathbf{X}(t)-\mathbf{X}(t-1))$, the corresponding state estimation algorithm has the form

where $\delta t$ is absorbed into the innovation $\Delta$.

The output of the state estimation algorithm $\mathbf{\widehat{y}}$ is the object pose. This is a discrete system with time steps $t\in\{0,1,...,T\}$.

The function $h$ maps a vector field representation of keypoints to object pose via a RANSAC and uncertainty-driven PnP framework (EPnP), as discussed in [PLH⁺19]. The task then becomes to learn an appropriate $\Delta$, such that the error in the state estimation algorithm is iteratively driven towards zero.

Consider the standard L2-norm loss function

The gradient of this function is

where $\mathbf{X}_{ij}^{k}$ is the ground truth vector field, and $\nabla_{x}$ denotes the gradient operator with respect to $x$. Set

where $\alpha(t)$ is a sequence of step-sizes that must be specified.

In practice, the gradient term $\nabla\Phi(t)$ is not directly measured. Instead, we use an estimate of this value,

where $\mathcal{F}$ represents the output of the CNN described in Section 3.3.

The iterative update is applied by employing gradient descent

Within this iterative framework the initial state $\mathbf{\widehat{X}}_{ij}^{k}(0)$ is the vector field estimate obtained from PVNet, and $\widehat{\nabla\Phi}(t)$ is estimated by our network. Algorithm 1 summarises the proposed information pipeline.

The network consists of two encoder-decoder blocks operating in parallel (see Figure 2). The top block in Figure 2 is labeled the Innovation Estimator, and the bottom block is labeled the Estimate Autoencoder. The Innovation Encoder consists of a pre-trained ResNet-18 [HZRS16] architecture. The Innovation Decoder consists of successive convolution and upsampling operations. The Estimate Autoencoder consists of a pre-trained ResNet-18 backbone followed by three successive upsampling layers. The input dimensionality of the Estimate Autoencoder modified to accept a vector field state estimate.

The Innovation Estimator receives an image and returns an estimate of the gradient of the vector field state $\widehat{\mathbf{\nabla}\Phi}$. The Estimate Autoencoder receives a vector field state $\bm{\widehat{X}}_{ij}^{k}$ and returns an estimate of the vector field state $\bar{\bm{X}}_{ij}^{k}$. The Estimate Autoencoder is therefore an auxiliary task aimed at allowing information from the initial state estimate to be injected into the Innovation Estimator via skip connections. This auxiliary task leads to improvement on the main task by leveraging domain specific information via multi-task learning.

The state loss is applied to the Esimate Autoencoder, and is defined by

where $(i,j)\in\mathcal{S}$ indicates that the pixel is within the segmentation mask, and the norm that is used is the ‘smooth l1 norm’ for unit vectors proposed by [Gir15]. The state $\bar{\bm{X}}^{k}_{ij}$ represents the same information as $\widehat{\bm{X}}^{k}_{ij}$ and is used only for the purpose of training the autoencoder.

The state gradient loss is applied to the Innovation Estimator, and is defined by

The loss is backpropogated after each iteration of the gradient descent defined by Eq. (15).

The loss function used to train the network is

where $\gamma$ is a scaling factor.

The network is trained for $T=2$ iterations of Eq. (15) at each epoch, with a constant step size of $\alpha=0.6$. We use a pretrained PVNet [PLH⁺19] to provide the initial estimate $\bm{X}^{k}_{ij}(0)$ for each object. We use a batch size of 64, and down-sample training images by four to fit the learning algorithm on our machine. We compute 8 keypoints via farthest point sampling, from which object pose is obtained via uncertainty-driven PnP [PLH⁺19]. We also render 10,000 images and synthesis 10,000 images for each object. We employ data augmentation in the form of random cropping, resizing, rotation, colour jittering. An Adam optimizer [KB15] is employed with an initial learning rate of 1$e^{-3}$, which is decayed to 1$e^{-5}$ by a factor of 0.85 every 10 epochs. The learning hyperparameter in Eq. (19) is set to $\gamma=10$. We train our network for 50 epochs.

During testing, we employ Hough voting to localise keypoints, before using uncertainty-driven PnP to obtain object pose from keypoints. We set the step size to $\alpha=0.01$ and perform iterations of Eq. (13) until the estimate converges. Once the gradient output of our Innovation CNN approaches zero we consider the estimate to be converged. Figure 4 provides qualitative results that illustrate this iterative convergence.

We conduct experiments on two popular datasets for object pose estimation.

LINEMOD [HKN] consists of 15783 images of 13 objects, with approximately 1200 instances for each object. Challenging aspects of this dataset include lighting variations, scene clutter, object occlusions, and texture-less objects.

Occlusion LINEMOD [BKM⁺14] is a subset of LINEMOD images with each image containing multiple annotated objects under severe occlusion, thus presenting a more challenging scenario for accurate pose estimation.

We evaluate our approach using two standard metrics for object pose estimation.

The ADD metric [HKN] computes the average 3D distance between the points of the 3D model under transformation from the ground truth and estimated pose respectively. Specifically, given the ground truth rotation $\mathbf{R}$ and translation $\mathbf{T}$ and the estimated rotation $\mathbf{\hat{R}}$ and translation $\mathbf{\hat{T}}$, the ADD metric computes

where $\mathcal{M}$ denotes the set of 3D model points and $m$ is the number of points. For symmetric objects we use the similar ADD(-S) metric [XSNF17], where the mean distance is computed based on the closest point distance,

We denote both metric as ADD(-S) and use the one appropriate to the object. In each case a pose is considered correct if the average distance is less than 10% of the 3D model diameter. The reported metric is the percentage of poses that are considered correct.

The 2D projection metric [HKN] computes the mean distance between pixel locations of the 2D projections of the 3D model model, when projection is performed with the estimated and ground truth poses. The metric is defined via

where $V$ is the set of all object model vertices, $\mathbf{Y}$ is the pose, and $\mathbf{P}$ is the camera matrix. The estimated pose is considered correct if the average error is less than 5 pixels. The metric reported is the percentage of correct poses.

Finally, we propose an additional metric, dubbed the State Distance (SD), designed to quantify the difference between the estimated and ground truth vector field states. This is used to provide an indication of whether the state estimate $\bm{\widehat{X}}^{k}_{ij}$ is converging to or diverging from the true state $\bm{X}^{k}_{ij}$. This metric is defined by

where $(i,j)\in\mathcal{S}$ indicates that the pixel is within the segmentation mask, and $s$ denotes the number of pixels within the segmentation mask.

Results in terms of the ADD(-S) and 2D projection metrics on the LINEMOD and Occlusion LINEMOD datasets are presented in Tables 1, 2, and 3. Qualitative results illustrating our final pose estimates are provided in Figures 3 and 5.

In this paper we focus on refining an initial pose estimate without leveraging any extra supervision. In Tables 1,2,3 we compare our results to the top three other pose estimation networks in this category. Our approach outperforms all previous methods and achieves state-of-the-art performance on both the LINEMOD and Occlusion LINEMOD datasets.

Most significantly, our network generates pose estimates that lead to a higher average on both the ADD(-S) and 2D projection metrics, compared to the baseline network. This includes a performance increase of 3.69% on the LINEMOD dataset and 3.31% on the Occlusion LINEMOD dataset in terms of the ADD(-S) metric. This is equivalent to improving the performance by 4.2% and 8.12% respectively compared to the performance of the baseline network.

It can be seen that the largest improvement is obtained for objects for which the baseline network provides a relatively poor initial estimate. Such objects include the relatively texture-less ‘ape’ and ‘duck’ and all objects in the Occlusion LINEMOD dataset. Specifically, on the ape and duck objects we improve baseline estimates by 21.14% and 10.73% respectively in terms of the ADD(-S) metric. Qualitative results for these two objects are presented in Figure 4, for which the ADD(-S) metric is evaluated at each iteration of Eq. (15). Figure 4 also illustrates the iterative convergence of the state distance SD. In this figure, the ADD(S) and SD metrics are plotted as a function of the interpolation distance,

The interpolation distance quantifies how far we have iterated from the initial estimate.

Qualitative results illustrating the iterative application of our method are presented for the Occlusion LINEMOD dataset in Figure 6. To generate Figure 6 the initial and final pose are displayed, along with two intermediate poses sampled from the iterative framework. For example, for the Ape object $T=120$ iterations were used, and Figure 6 displays poses for T={0,40,80,120}.

We conducted experiments with a range of configurations before choosing the configuration used to produce the presented results. Results of these experiments are summarised in Table 4. In Table 4, column one shows a comparison of training with (✓) and without (✗) the Estimate Autoencoder. Column two shows a comparison of backpropograting each iteration of Eq. (15) (✓) and backpropograting after $T$ iterations (✗). Column three shows a comparison of choosing the output of PVNet as $\bm{X}^{k}_{ij}(0)$ (✓) and choosing the ground truth vector field computed from keypoints perturbed with 1% noise (✗). All ablation experiments were conducted on the Ape object of the LINEMOD dataset.

Results shown are the mean ADD(-S) results obtained from testing the final 20 epochs of the model. In Table 4 we compare the results of experiments with and without the Estimate Autoencoder (the bottom network block in Figure 2). We also compare our backpropgation scheme (backprop each iteration of Eq. (15)) with the alternative scheme of accumulating the loss and backpropogating after $T$ iterations, as discussed in [HR19]. Finally we compare the impact of using a pretrained PVNet as the initial estimate $\bm{X}^{k}_{ij}(0)$ against using the ground truth vector field, or the ground truth vector field perturbed with different levels of noise. We trialed a range of different noise levels, and the result of applying 1% noise is presented in Table 4.