Spacecraft Pose Estimation Based on Unsupervised Domain Adaptation and on a 3D-Guided Loss Combination

Pose estimation is the process of estimating the rigid transform that aligns a model frame with a sensor, body, or world reference frame [13]. Similarly to [27] we categorise these methods into: end-to-end, two-stage, and dense.

End-to-end methods map the two-dimensional images or their feature representations into the six-dimensional output space without employing geometrical solvers. Handcrafted approaches compare image regions with a database of templates to return the 6 Degree-of-freedom (DoF) pose associated with the template [10, 32]. Learning-based approaches employ CNNs to either learn the direct regression of the 6DoF output space [15, 20, 14, 28] or classify the input into a discrete pose [34]. End-to-end methods have been explored in [30], where an AlexNet was used to classify input images with labels associated to a discrete pose. The authors in [28] employed a network to regress the position and do continuous orientation estimation of the spacecraft with soft assignment coding.

Two-stage pose estimation methods first estimate the 2D image projection of the 3D object key-points and then apply a Perspective-n-Point (PnP) algorithm over the set of 2D-3D correspondences to retrieve the pose. Traditionally, this has been solved via key-point detection, description and matching between the images and a 3D model representation based on handcrafted [19, 2, 1] or learnt [36, 21] key-point detectors, descriptors and matchers. Other approaches directly employ CNNs designed for 2D key-point localisation that detect the vertices of the 3D object bounding-box or regress the key-point position by employing heatmaps, hence unifying the detection, description and matching into a single network [22, 39]. These approaches are the current state-of-the-art in satellite pose estimation. The authors from [4] employed an HRNet and non-linear pose refinement which obtained the first position in the Spacecraft Pose Estimation Challenge of 2019. In [11] a multi-scale fusion scheme with a 3D loss independent of the depth was employed to improve the pose estimation accuracy under different scale distributions of the spacecraft.

Dense pose estimation methods establish dense correspondences, at pixel or patch level, between a 2D image and a 3D surface object representation. These approaches can be based on depth data, RGB images or image pairs [27, 12, 6].

In visual-based navigation for space applications the distribution of the data to be acquired during the mission is often unknown, as no or little data is available prior the mission. This entails the introduction of computer graphic simulators [26] to generate the data required to design and test the algorithms. Although computer-graphic simulated images have greatly improved their quality and fidelity, there still exists a noticeable domain gap between the simulated (source) data and the real (target) data. Methods to overcome the reduction in algorithms performance for the same task under data with different distributions fall under the category of Domain Adaptation [23]. More specifically, when labels are only provided in the source domain is referred as Unsupervised Domain Adaptation (UDA). UDA methods can be categorised into: 1) domain-invariant feature learning; 2) aligning input data; and 3) self-training via pseudo-labelling.

Domain-invariant feature learning aims to align the source and target domain at feature level. It is based on the assumption that exists a feature representation that does not vary across domains, i.e. the information captured by the feature extractor is the one that does not depend on the domain. This family of methods try to learn the representation either by minimising the distribution shift under some divergence metric [35], enforcing that both feature representations can be used to reconstruct the target or source domain data [9], or employing adversarial training [8].

Input data alignment. Instead of aligning the domains at the feature level, these approaches aim to align the source domain to the target domain at input level, a mapping usually confronted via generative adversarial networks [42, 33].

Self-training approaches employ generated pseudo-labels to iteratively improve the model. However, these labels generation process is inevitably noisy. Some works have tried to mitigate the effect of the label noise by designing robust losses [41] but they often fail to handle real-world noisy data [40]. Other approaches have used self-label correction that relies on the agreement between different learners [17] or stages of the pipeline [40] to correct the noisy labels. Online domain adaptation is employed in [24] for satellite pose estimation across different domains. A segmentation-based loss is employed to update the feature extractor of a network, adapting the features to the new input-domain.

The ground-truth heatmaps $h_{gt}$ are obtained by representing a circular Gaussian centred at the ground-truth image key-point coordinates $p_{gt}=[u_{gt},v_{gt}]^{T}$. Given an image of the target spacecraft, represented by the 3D key-points $P_{gt}=[x,y,z]^{T}$ and acquired with camera intrinsics $K$ in a ground-truth pose with rotation matrix $R_{gt}$ and translation vector $t_{gt}$, the ground-truth key-point coordinates can obtained via the projection expression:

The heatmaps $h^{i}_{est}$ predicted by each head $i$ are learnt by comparing the output of the key-point prediction head vs the ground-truth heat-map $h_{gt}$ via mean squared error. To keep a fixed range for the key-point heatmap loss, we normalise each heatmap by its maximum value and weight them by a parameter $\beta=1e3$, ensuring a large difference between the minimum and maximum value of the heatmap. The key-point heatmap loss is then computed via:

Pose information is here considered by comparing the ground-truth 2D key-point image coordinates $p_{gt}$ with the projection of the 3D key-points $P_{gt}$ onto the image, obtained using the rigid transform estimated via PnP (see Fig. 2 - Model b)). The predicted key-point locations $p^{i}_{est}$ used to estimate such rigid transform are originally expressed as heatmaps. The application of the arg-max function over $h^{i}_{est}$ to retrieve the predicted key-point pixel positions $p^{i}_{est}$ is a non-differentiable operation which cannot be used for training. We instead use the integral pose regression method [37] and apply an expectation operation over the soft-max normalized heatmaps $h^{i}_{est}$ to retrieve the pixel coordinates $p^{i}_{est}$. Then, we employ the backpropagatable PnP algorithm from [5] to retrieve the estimated rotation matrix $R^{i}_{pnp}$ and translation vector $t^{i}_{pnp}$.

The PnP loss evaluates the mean squared error between the ground-truth key-point pixel locations $p_{gt}$ and the key-point pixel locations $p^{i}_{pnp}$ obtained by projecting $P_{gt}$ onto the image with $R^{i}_{pnp}$ and $t^{i}_{pnp}$ as in Eq. 1. The loss has an additional regularisation term, comparing $p_{est}$ with $p_{pnp}$, to ensure convergence of the estimated key-point coordinates the desired positions [5]. The final PnP loss term $\ell_{pnp}$ is computed via:

We incorporate 3D structure by comparing the rotation $R^{i}_{3d}$ and translation $t^{i}_{3d}$ that align the predicted key-points lifted to the 3D space $P^{i}_{3d}$ with the key-points representing the spacecraft $P_{gt}$ with the ground-truth pose (see Fig. 2 - Model c)). Following the approach described in Section 3.1, we retrieve the estimated key-point coordinates $p^{i}_{est}$. Then we lift them to the 3D space by applying the camera intrinsics $K$ and the estimated depth $d^{i}_{est}$. Instead of providing direct supervision to the depth, we choose to sample the depth prediction head at the position of the ground-truth key-point positions $p^{i}_{gt}$, and supervise the output with the 3D alignment loss. Sampling at the ground-truth key-point positions instead of the estimated positions helps convergence, as the number of combinations of key-point locations and depth values that satisfy a given rotation and translation is not unique. $R^{i}_{3d}$ and $t^{i}_{3d}$ are found by minimising the sum of square distances between the target points $P_{gt}$ and the lifted 3d points $P^{i}_{3d}$ :

The algorithm that solves this expression is referred as the Umeyama [38] (or Kabsch) algorithm and has a closed form solution. We employ the Procrustes solver from [3] to obtain the arguments that minimise the above expression. The loss is then computed by:

The final learning objective is then defined by a weighted sum of the three losses. The values of $\gamma_{1}=\gamma_{2}=0.1$ are selected so the contributions of the 2D-3D and 3D-3D losses represent a 10% of the total learning objective.

The evaluation metric used in the Challenge was the pose score $S_{total}$. The pose score is based on the combination of the position score $S_{pos}$ and the orientation score $S_{ori}$. The position score for an image $j$ is defined as the absolute difference between the estimated and ground-truth translation vectors, normalised by the module of the ground-truth position vector. The position score accounts for the precision in position of the robotic platform, which is of $2.173$ millimetre per metre of ground truth distance. Values of $S_{pos}$ lower than $0.002173$ are zeroed.

The orientation score $S_{ori}$ is defined as the angle that aligns the estimated and ground-truth quaternion orientations. This metric also accounts for the precision of the robotic arm which is 0.169º, zeroing $S_{ori}$ than the precision.

The total score is the average of the sum of the orientation and position scores corresponding the $N$ images of the test set:

The ground-truth labels of the two test sets (Lightbox and Sunlamp) from the SPEED+ dataset have not been made publicly available. Instead, the Challenge evaluation was performed by submitting the estimated poses to a evaluation server; the results were then published on public leaderboards only if the obtained score improved a previous result from the same participant. We provide a summary of the results by showing the error achieved by the 10 first teams on the Lightbox and Sunlamp datasets on Table 1 and Table 2 respectively. Teams are represented in decreasing order based on the achieved rank in the leaderbord, corresponding the first row to the winner of the category. The winning teams were TangoUnchained for the Lightbox dataset and lava1302 for the Sunlamp dataset. Our solution VPU ranked 2nd on both datasets, and achieved the best total average error as shown in Table 3, where we represent the averaged scores of the two datasets. This shows that our solution is robust to different domain changes and can be applied without fine-tuning on different settings with different illumination and noise conditions.

For the Challenge, we trained the model described in Section 3 during 35 epochs on the Synthetic dataset at a resolution of 640x640 pixels. Then, we trained individual models following Algorithm 1 for the Sunlamp and Lightbox target datasets during 50 iterations. Fig. 3 and Fig. 4 provide visual results of the estimated poses for the Sunlamp and Lightbox datasets. The estimated poses are represented with the 3-axis plot and the wireframe model projected over the image. They show that the proposed method can handle strong reflections, presence of other elements on the image, complex background and close views of the object.

We show in Fig. 5 the effect of iterating over Algorithm 1 on the number of generated pseudo-labels. The left plot corresponds to the strategy followed over the Sunlamp dataset, where the reprojection error that controls the convergence criteria of RANSAC-EPnP remains fixed during all iterations with a value of 2.0. It can be observed that the percentage of pseudo-labels reaches 90% after 10 iterations and slowly increases from there, suggesting a possible overfit to wrongly estimated labels. The right plot corresponds to the Ligthbox dataset, where a different approach was followed. Motivated by the fact that the Lightbox test set is more visually similar to the training set than the Sunlamp test set, we hypothesize that restricting the label generation by reducing the reprojection error would lead to better pseudo-labels and hence better perfromance. To test this, the reprojection error was decreased after 10 iterations (red dashed line) to 1.0. It can be observed that the percentage of pseudo-labels quickly drops and then grows at a slower rate. However, based on the final results we cannot conclude that this strategy improved the results compared to using a fixed reprojection error, as the achieved errors for both test sets are in a similar range.

To show the effectiveness of the proposed method on the target test sets, we evaluated the proposed pose estimation algorithm against a baseline considering only the 2D heatmap loss. The models were trained on the source synthetic dataset during 10 epochs and tested over a pseudo-test dataset. The pseudo-test set is obtained by running Algorithm 1 until approximately 30 % of the target dataset is labelled. This number is chosen as a compromise between the necessary number of pseudo-labels to obtain a representative analysis and the quality of the labels obtained. The pseudo-test set approach is followed as the real labels of the test set are not made publicy available in the challenge. The results are shown in Table 4 with the validation set from the source dataset as reference. It can be observed that the proposed method, despite obtaining similar values on the validation (Source set) is able to improve the results on the target (test) sets.