PYRAMID FEATURE ATTENTION NETWORK FOR MONOCULAR DEPTH PREDICTION

In this paper, we propose Pyramid Feature Attention Network based on encoder-decoder architecture. DenseNet-161 [5] pre-trained on ILSVRC as our encoder. Decoder is composed of Dense ASPP [10], Dual-scale Channel Attention Module, and Spatial Pyramid Attention Module.

The proposed network architecture is shown in Fig.2. Input a single RGB image with resolution H. In encoder, the five convolutional blocks {$E_{1},E_{2},E_{3},E_{4},E_{5}$} output feature maps with different resolutions that are H/2, H/4, H/8, H/16 and H/32 respectively. The high-level features are from $E_{3}$, $E_{4}$ and $E_{5}$. The low-level features are from $E_{1}$ and $E_{2}$, which upsample to resolution of $E_{2}$. After the backbone network, for high-level features, we apply Dense ASPP module to fuse high-level features and expand the receptive field. This module produces an H/8 feature map via various dilated convolutional operations. The dilation rates r are 3, 6, 12, 18 and 24 respectively. Following Dense ASPP, we place DCAM to extract global context and local information from high-level features. Then, we apply SPAM to capture spatial information at multi-scale from the low-level feature maps. To get the high-level and low-level features with same resolution H, we utilize up-convolutional layer, which consists of upsampling operations and a 3×3 convolutional layer. Finally, them are concatenated and fed into the final convolutional layer to get the depth estimation $\widetilde{d}$.

The previous channel attention methods based on the squeeze and excitation [16] capture global context from the feature maps. However, this way ignores local information in features. To aggregate global context and local information simultaneously, we propose Dual-scale Channel Attention Module, as shown in Fig.3. DCAM consists of global channel attention block, local channel attention block and recalibration block. Its core idea is to implement channel attention on different scales.

In global channel attention block, average pooling layer and max pooling layer apply to reduce computational cost. To aggregate global context across channels, the dimension of the input feature maps are fused and reduced to 2C/r by 1×1 convolution layer, where C is the dimension of input $x\in\mathbb{R}^{C\times H\times W}$, r is reduction rate. And then this block produces global channel-attention vector $g(x)\in\mathbb{R}^{C\times 1\times 1}$ via another 1×1 convolution layer. Similarly, in local channel attention block, we place two 1×1 convolution layers to extract local information across channels, which generate local channel-attention map $l(x)\in\mathbb{R}^{C\times H\times W}$. Local channel-attention map and global channel-attention vector are fused to channel-attention map $A_{c}(x)\in\mathbb{R}^{C\times H\times W}$, before calibration (see Eq.(1)). Thus, we can effectively employ channel attention information and avoid introducing interference. Finally, we calibrate the original channel-attention map to improve feature representation, and new channel-attention map $\tilde{A}_{c}(x)$ as shown in Eq.(2).

$$A_{c}(x)=l(x)\otimes g(x)$$

(1)

$$\tilde{A}_{c}(x)=h[f(x)\otimes A_{c}(x)]\oplus x$$

(2)

where f and h are 1×1 convolutional layers in recalibration block. $\oplus$ denotes element-wise addition. And $\otimes$ denotes element-wise multiplication. Note that each convolutional layer is followed by an activation function ReLU.

The high-level feature maps always processed by channel attention module, since it can capture channel’s dependency. However, this ignores structural information of the feature maps. To extract more local detailed information from the low-level feature maps, we proposed the spatial pyramid attention module, which utilizes the spatial pyramid structure. Fig.4 depicts the paradigm of SPAM. This module contains downsampling operation, spatial attention block and upsampling operation. Suppose the input low-level features $y\in\mathbb{R}^{C\times H\times W}$ via down-sampling operation, get down-sampling feature maps $y_{i}$ (i=1, 2, 3), the resolution is 1, 1/2, 1/4 of the input, respectively. Then spatial attention block learns the spatial attention map $s(y_{i})\in\mathbb{R}^{C\times H\times W}$, as shown in Eq. (3), these three blocks form a pyramid structure. Finally, through upsampling operation, we fuse the multi-scale spatial attention map. The output of SPAM $A_{s}(y)$ can be presented as Eq. (4).

$$s(y_{i})=\sigma(Conv_{2}(\delta(Conv_{1}(y_{i}))))$$

(3)

$$A_{s}(y)=\sigma[s(y_{1})\oplus s(y_{2})\oplus s(y_{3})]\otimes y$$

(4)

where $Conv_{1}$ and $Conv_{2}$ refer to convolutional layers in spatial attention block. $\delta$ refers to ReLU function, $\sigma$ refers to Sigmoid function.

The loss function to constraint our network contains two terms, i.e., scale-invariant loss (in log space) $L_{d}$ and scale-invariant gradient loss $L_{s}$. We describe in detail each loss item as follows.

Scale-invariant loss. Scale invariant loss is proposed in [1] by Eigen et al., as shown in Eq. (5).

$$L_{d}(e)=\frac{1}{T}\sum_{i}e_{i}^{2}-\frac{\lambda}{T^{2}}(\sum_{i}e_{i})^{2}$$

(5)

where $e_{i}=log(\tilde{d}_{i})-log(d_{i})$. $\tilde{d}_{i}$ denotes ground truth of depth. ${d}_{i}$ denotes predicted depth. $\lambda=0.5$. T refers to the number of pixels with valid ground truth depth value. By rewritig Eq. (5):

$$L_{d}(e)=\frac{1}{T}\sum_{i}e_{i}^{2}-\frac{1}{T^{2}}(\sum_{i}e_{i})^{2}+\frac{(1-\lambda)}{T^{2}}(\sum_{i}e_{i})^{2}$$

(6)

we can think of Eq. (6) as the sum of variances and weighted mean square error in log space. Setting $\lambda=0$ is L2-norm, and setting $\lambda=1$ is scale invariant error. Inspired by [3], we set $\lambda=0.85$ in this work to accelerate minimizing the variance of error.

Scale-invariant gradient loss. Scale-invariant gradient loss (see Eq. (7)) is defined by [11] by Ummenhofer et al., which based on gradient loss. This loss function emphasizes sharpness at object boundaries and increases smoothness with in similar fields. To cover gradients at multi-scale, we utilize 5 different spacings $s\in\left\{1,2,4,8,16\right\}$.

$$L_{g}(g)=\frac{1}{T}\sum_{s\in\left\{1,2,4,8,16\right\}}\sum_{i,j}\left\|\tilde{g_{s}}(i,j)-g_{s}(i,j)\right\|_{2}$$

(7)

$$g_{s}(i,j)=(\frac{d_{i+s,j}-d_{i,j}}{|d_{i+s,j}+d_{i,j}|},\frac{d_{i,j+s}-d_{i,j}}{|d_{i,j+s}+d_{i,j}|})^{T}$$

(8)

where $\tilde{g_{s}}$ and $g_{s}$ refer to gradient pixel (i,j) in predicted depth map and ground truth respectively. $d_{i,j}$ is the depth value of pixel (i,j).

Total loss. We find that appropriately scaling the range of loss function can accelerate convergence and improve the final predicted result. Our total loss function is defined as follows:

$$L_{total}=\alpha\sqrt{L_{d}}+\beta\sqrt{L_{g}}$$

(9)

where $\alpha$ and $\beta$ are constants we set to 10 and 2 for all experiments.

The KITTI dataset [20] consists of 61 outdoor scene images, each with a resolution of 375×1241. Since previous work is based on the training set and test set divided by Eigen et al. [1], we also follow it to compare with those works. The training set contains 23488 images from 32 different scenes, and the test set contains 697 images from 29 scenes. The maximum depth of the image in the KITTI dataset is 80.

We implement the proposed network based on public deep learning framework PyTorch. In training phase, we use Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$ and $\varepsilon=10^{-6}$. The learning strategy applies polynomial decay with initial learning rate $lr=10^{-4}$ and power $p=0.9$. We train our network on two NVIDIA TITAN RTX GPU with 24GB memory. Epoch is set to 50 with batch size 32, which applies to all experiments of this work. As the backbone network for encoder, we use ResNet [4], ResNext [6] and DenseNet [5] pretrained in ILSVRC dataset. Upconvolution in decoder uses the bilinear neighbor upsampling followed by convolutional layer. Downsampling operation and upsampling operation in spatial pyramid attention module utilizes the nearest neighbor method. We set reduction ration $r$ is 16. To improve training results, we augment images before input to the network using random rotating, random horizontal flipping, random brightness, contrast and color changing. We randomly crop the image size 352×704 to train the network.

We quantitatively compare our network with state-of-the-art methods both using the following commonly used metrics:
$-$ Accuracy with Threshold $t$: $\delta=max(\frac{d_{i}}{\tilde{d_{i}}},\frac{\tilde{d_{i}}}{d_{i}})<t$, for $t\in\left\{1.25,1.25^{2},1.25^{3}\right\}$
$-$ Absolute Relative Error (Abs Rel): $\frac{1}{N}\sum_{i=1}^{N}\frac{\left|\tilde{d_{i}}-d_{i}\right|}{d_{i}}$
$-$ Squared Relative Error (Sq Rel): $\frac{1}{N}\sum_{i=1}^{N}\frac{\left\|\tilde{d_{i}}-d_{i}\right\|^{2}}{d_{i}}$
$-$ Root Mean Squared Error (RMSE): $\sqrt{\frac{1}{N}\sum_{i=1}^{N}\left\|\tilde{d_{i}}-d_{i}\right\|^{2}}$
$-$ Root Mean Squared Error in log space (RMSElog): $\sqrt{\frac{1}{N}\sum_{i=1}^{N}\left\|log(\tilde{d_{i}})-log(d_{i})\right\|^{2}}$
where N is total number of pixels that the ground truth values are available. $\tilde{d_{i}}$ and $d_{i}$ are predicted depth values and ground truth for pixel i.

Table 1 shows quantitative results compared with the state-of-the-art methods. Our network far outperforms all existing methods. As shown in Fig. 5, our method shows much more precise object boundaries and much more continuous object surfaces. To prove the effectiveness of our proposed method, we utilize various backbone network as encoder, and keep other settings. Table 2 provides experimental results. And the results show that ResNext-101 achieve state-of-the-art result.

To investigate the importance of different modules in our method, we conduct the ablation study. It can be seen from Table 3 that the proposed model contains all modules (i.e. DCAM, SPAM, scale-invariant gradient loss) to achieve the best performance, which demonstrates that all modules are necessary to get the best monocular depth estimation result.