Deep auxiliary learning for visual localization using colorization task

Following the typical CNN-based colorization algorithm [15], we perform our task in CIE Lab color space as distances in this space model perceptual distance. Our C-net predicts the corresponding a and b color channels $\tilde{Y}^{a,b}$ (we denote ground truth as $Y^{a,b}$) with lightness L channel $X^{L}$ as input. Different from [15] using a VGG-style architecture, we adopt the lightweight U-Net [38] that has four downsampling and followed four upsampling operations as well as skip connections employed between layers with same resolution. In order to search for a prediction of vibrant and realistic colorization to increase the diversity of colors, some algorithms think colorization as a classification problem and use cross-entropy loss. In our case, colorization is designed as an complementary task for auxiliary learning of localization, we do not overly entangle with colorization performance but focus on localization improvement by aggregating semantic representation. Thus, we define a $l_{1}$ Euclidean distance loss function between predicted and ground truth color version to constrain the weights updating of C-net.

$$L_{c}(X^{L})=\sum_{h,w}{\lVert{\tilde{Y}^{a,b}-Y^{a,b}}\rVert}_{1}$$

Some previous works evaluated how representation learning via colorization can serve for classification task. It was demonstrated that best classification performances are achieved when feature representations from the maximum downsampled layer are contributed into linear classifiers. From our C-net, we follow this conclusion and choose 16-times downsampled feature representations with respect to the input dimension in order to introduce high-level semantic information into localization. We define these representation as $M_{c}$ for the ease of following description.

Given a RGB image $I\in\mathbb{R}^{H\times W\times 3}$, L-net can predict the absolute pose $P=[x,q]$, where $x\in\mathbb{R}^{3}$ denotes the translation and $q\in\mathbb{R}^{4}$ denotes the rotation in a unit quaternion representation. L-net consists of four parts, a backbone for feature extraction, a fusion operation, an attention module and a regressor to predict associated pose.

Backbone Following typical localization architecture, our backbone is designed with five standard residual blocks which have the same bottleneck structure and unit setting with ResNet-50 [39] architecture. The output feature representations of the Res5 block is 16-times downsampled with respect to input image, and we define it as $M_{l}$.

Fusion operation The aim of this step is to fuse representations $M_{l}$ from localization backbone and semantic representations $M_{c}$ from colorization. Since L-net and C-net are two independent tasks with different input modalities and prediction purposes, it is crucial to intelligently select helpful information from $M_{c}$ and avoid introducing irrelevant feature representations which may have a negative effect on pose regression. To this end, we apply a typical linear weighting and activating operations after the cross-channel concatenation between $M_{l}$ and $M_{c}$ to active elements that are useful for pose regression. This fusion operation can be formulated as:

$$M_{fuse}=max(W\ast(M_{l}\oplus M_{c})+b,0)$$

Where $\oplus$ represents concatenation across channels; $W$ and $b$ are weight parameters updated by learning. $M_{fuse}$ is the feature representation after fusion. This fusion operation is completed by a convolution and activate layer in our experiments.

Attention module. As previously analyzed, different feature representations do not equally contribute to localization task. Thus, an attention module is designed upon prior similar works [36] and [37]. Reference to human visual localization mechanism, regions of interest consisting of salient objects or special texture ought to be enhanced in localization-specific attention module.

This attention operation can be described with two steps. The first squeeze and excitation process. Activating $M_{fuse}$ with an channel-wise vector $V=GMP(M_{fuse})$, where $V\in\mathbb{R}^{C}$ obtained by global max pooling operation of squeezing spatial information into a global representation. And excitation is completed by per-channel multiplication between $M_{fuse}$ and $V$.

$$M_{atten}=M_{fuse}\otimes(GAP(M_{SaE}))$$

$$M_{SaE}=M_{fuse}\otimes GMP(M_{fuse})$$

The next step aims to achieve an attention mask to weight different regions of interest in feature representations $M_{fuse}$. Such mask is computed by applying a cross channel global average pooling operation on $M_{SaE}$ and is a single channel map with high response to regions of interest. As shown in Figure 2, some regions with salient objects, e.g., gates and windows of building, bicycles and stairs are activated to impel network to learn localization from more useful feature respresentations. Finaly, to not loss information, we fuse the original holistic representations $M_{fuse}$ with the region enhanced representations $M_{atten}$ via aforementioned fusion operation and use them for the final regression.

Regressor. After the attention module, pose regressor is designed with three inner fully connected layers with 2048 neurons, 3 neurons and 4 neurons respectively. The first one embeds feature representations into a high-dimensional vector and the last two separately output absolute position and orientation. In order to tackle the over parameterization problem of 4-dimension unit quaternion, we follow the method proposed by [10] and adopt a de-parameterization strategy during training process. The idea is to convert 4-dimentional unit quaternion $q=[u,\textbf{v}]$ ($u$ is a scalar and v is a 3 dimensional vector) into 3 dimensions by the following formula:

$$\log{q}=\frac{\textbf{v}}{\|\textbf{v}\|}\cos{u^{-1}}$$

Finally, we define the loss function with typical $l_{2}$ Euclidean distance to constrain the weights updating of L-net. Meanwhile a hyper parameter is introduced to balance translation and rotation error.

$$L_{l}(I)={\lVert{\tilde{x}-x}\rVert}_{2}+\beta_{intra}\ast{\lVert{\tilde{q}-q}\rVert}_{2}$$

Our proposed architecture can be viewed from the perspective of multi-task learning as a soft-parameter sharing approach. The objective of auxiliary colorization task is to enable the model to learning representations that are helpful for the main localization task. Thus we adopt a joint training strategy that allows the model to learn beneficial representations and also to be more robust against random data noise. The final loss function is then defined as:

$$L=\beta_{inter}\ast L_{c}(X^{L})+L_{l}(I)$$

The former term corresponds to the colorization regularizer and the latter describes localization related loss function. Similar to aforementioned intra-task parameter, $\beta_{inter}$ is also a hyper parameter that keeps balancing inter-task loss terms. It can be trained online or preset.

We benchmark performance of our method on three outdoor datasets: DeepLoc [11] , Cambridge landmark [8] and Oxford robotcar [43, 44]. DeepLoc is a large-scale urban outdoor localization dataset in an area spanning $110m\times 130m$. It is collected by a robot moving along a loop road in an university campus. The captured images are with resolution of $1280\times 720$ and the ground truth is computed by a LiDAR-based SLAM algorithm. The other dataset King’s College, with an area of $140m\times 40m$, is widely used for benchmarking localization tasks. It is collected around a landmark building and the ground truth pose is obtained by a Structure From Motion algorithm. Both datasets are very challenging since training data and test data are captured from different points in time and distinct walking paths. Moreover, significant urban clutter such as pedestrians and vehicles are present in King’s College dataset, and scene views vary largely within different paths of DeepLoc.

A well-known indoor dataset - Microsoft 7-Scene [45] is also used for evaluating our method. Seven difference scenes are recorded from a handheld Kinect RGB-D camera at $640\times 480$ resolution and ground truth camera pose is provided by KinectFusion algorithm. The existence of motion blur and weak texture under office environment makes this 7-scene dataset very challenging.

Although preprocessing is widely used for many visual tasks, prior works demonstrate that typical preprocessing techniques such as cropping and mirroring can not yield performance improvements for localization. In some cases, they even negatively affect the pose accuracy. In our experiments, we only take already proved well-performanced preprocessing steps like resize input images into $320\times 240$ and normalized by pixel mean subtraction and standard deviation division operation.

We use the Adam solver for optimization with $\beta_{1}=0.9$, $\beta_{2}=0.99$ and $\xi=10^{-10}$. We initialize five residual blocks of L-net with weights of ResNet-50 pre-trained on ImageNet and remaining layers with Gaussian distribution. The balance parameters, $\beta_{intra}$ and $\beta_{inter}$ are set as 3 and 0.2 for all datasets. Then we train all layers with mini-batch size of 10. The learning rate of the backbone layers of L-net and all layers of C-net are initialized as 0.0003. All other layers of L-net are initialized as a learning rate of 0.001, and both learning rates decay by a power=0.9 every 10 epochs. The total number of iteration is about 150 epochs for two outdoor datasets and 80 epochs for indoor Microsoft 7Scene dataset. The work is implemented based on Tensorflow deep learning library. All the experiments are performed on a NVIDIA Titan V GPU with 16GB on-board memory.

The evaluation results of our method against prior CNN-based methods are shown in Table 1. PoseNet [8] and its variant BaysianPoseNet [17] as well as ours have input of a single image. SVS-Pose [40] leverages additional depth information to do data augmentation in 3D space. VLocNet [21] takes advantages of sequential information. And VLocNet++ [11], as a multi-task framework for odometry, localization and semantic segmentation learning, requires much more information such as successive images, semantic segmentations annotation and depth information. Results of PoseNet2 [20] on DeepLoc and VLocNet++ [11] on King’s College are absent since no publish data is available.

According to the median translation and rotation error indicated in Table 1, our method outperforms all other algorithms on King’s College. “ - ” denotes no data provided. On Deeploc dataset, our method also shows outstanding localization performance. Although VLocNet++ has slightly smaller localization error than ours, it should be noted that our method requires a single image for both training and inference, while VLocNet++ should be trained with tremendous semantic segmentation annotation cost and ground truth depth. Also, from camera pose trajactories for test sequences from DeepLoc and King’s College datasets shown in Figure 3, predicted results with our method have relatively few outliers and higher localization accuracy.

Colorization results from our method are presented in Figure 4. Four groups of image colorization results are shown. Three each group, the first image is the original color image, the second image is grayscale and the last one is the colorization image. Apparently our colorization results are quite natural and indistinguishable from original image. In order to quantitatively evaluate colorization performance, we display statistical analysis in Table 2 by counting the percentage of pixel whose colorized value is close to the corresponding pixel in the original image. We define this closeness by judging if the color difference is smaller than a threshold (eg. 5 or 10). The results show that our colorization version is much similar to the original color. The main differences exist in some areas with bright and unfixing colors like sky, as the regression strategy used in our colorization has an averaging effect on color prediction.

Since auxiliary learning and attention strategy are both introduced in our method, in this section we will separately analyze their influences on localization performance. Therefore we design three different network architectures for different ablative levels. At first, we evaluate the localization performance of the baseline L-net by cutting the representation input from C-Net and removing attention module. In the second network, we re-add C-Net for auxiliary learning to L-net but keep removing attention module. The third network is our whole architecture with both auxiliary learning and attention mechanism.

These three networks are trained with the same initialization and hyper-parameter setting, and tested on two outdoor datesets. The results are shown in Table 3. Apparently our introduced auxiliary learning strategy via colorization leads to significant performance improvements. This improvement is more evident from the test on DeepLoc. We believe that this is because environment situations in DeepLoc are more complicate than those in King’s College. Images from King’s College is mainly based on a landmark building. But DeepLoc is a normal urban scenario with various streets and buildings. Semantic information from this situation seems more difficult to be learned by localization network directly. In this case, semantic feature representations learned via colorization are especially beneficial for localization. Moreover, attention mechanism also contributes to localization task by proving accuracy improvement as well as training convergence acceleration. For instance, during the test in Deeploc, required training epoch for the whole network is 155, but it increases to 189 when we remove the attention module.

In this section we replace our colorization task by semantic segmentation and compare their auxiliary learning function on localization. This means that we keep C-net unchanged but its training purpose turns into semantic segmentation. Thus the input of C-net should be changed to a RGB image. And the output becomes into a pixel-wise segmentation prediction whose channel number is equal to the number of semantic annotation categories. This new network is then trained with DeepLoc dataset. Since semantic segmentation labels are available for only half training data in this dataset, we train the whole network (semantic segmenation task and localization task) with semantic segmentation labeled images. Then we fine tune L-net with rest semantic unlabeled images by fixing weights of C-net. The localization results after both stage are presented in Table 4.

According to the results shown in Table 4, it seems that colorization and semantic segmentation equally contribute when they are considered as auxiliary task for localization. But semantic segmentation suffers from additional annotation cost.

Above evaluations are based on localization performance and segmentation is considered as auxiliary task. Here we present semantic segmentation performance and compare it with other segmentation algorithms. The experiment is performed on DeepLoc dataset and the Intersection over Union (IoU) score for major individual categories as well as the mean IoU are shown in Table 5. The segmentation images can be seen in Figure 5. From the results, our method outperforms AdaptNet [46] and DeepLabv3+ [47] as well as is very close to VLocNet++ that is attributed to the warping computation by successive images and depth information.

In addition to the above outdoor datasets, experiments are also performed on indoor dataset Microsoft 7Scene in order to fully demonstrate our localization ability. We compare our method with other CNN-based methods like PoseNet [8], LSTM-Pose [19], PoseNet2 [20], and MapNet/MapNet+ [10]. The median translation and rotation errors for each method are shown in Table 6 meanwhile camera pose trajactories for test sequences of stairs scene are presented in Figure 3. Our method has similar accuracy with MapNet+, but outperforms other methods. To be mentioned that MapNet+ is an advanced version of MapNet by fusion additional information (eg. visual odometry) to update the weights of MapNet with self-supervised learning. Therefore a correct visual odometry algorithm is required for MapNet+. While our method is more independent as only baseline information (image and corresponding ground truth pose) is required. Moreover, our model is easier to converge than others. For example, our model takes about 80 epochs iterations to converging while MapNet takes 300 epochs iterations.

To sum up, our proposed auxiliary learning for visual localization via colorization exhibits significant accuracy-efficiency balance performance without requiring extra information. All these properties makes it suitable for many applications including indoor robots and outdoor autonomous vehicles.