Exploring Rich and Efficient Spatial Temporal Interactions for Real Time Video Salient Object Detection

Conventional methods estimate the temporal information mainly using the contrast computation over the hand-crafted features. Liu et al. [16] computed superpixel-wise spatiotemporal contrast histogram to sense the temporal information. Similarly, Liu et al. [4] proposed the intra-frame similarity matrices to perform the bi-directional temporal propagation as the temporal information. Wang et al. [17] acquired the spatiotemporal feature by computing robust geodesic measurement for locating spatial edges and motion boundaries.  [3] adopted the gradient flow field to obtain intra-frame boundary information as well as inter-frame motion information. Further, Chen et al. [18, 19] adopted the long-term information to enhance the spatiotemporal saliency consistency.

Benefited by the development of deep learning techniques, the current SOTA methods have widely adopted the bi-stream network structure, in which one of its streams aims the color saliency over the spatial information, and another stream extracts the motion saliency over the temporal scale. Le et al. [14] has adopted such bi-stream structure, in which one of its streams computes the superpixel-wise spatial saliency cues, and another one aims the temporal saliency computation by applying the 3D convolution directly over multiple video frames. However, its direct usage of 3D convolution has totally overlooked the spatial information, obscuring its detection boundary severely. Wang et al. [20] proposed to use 2D convolution network to sense the differences between adjacent two frames as the temporal information, in which its behind rationale is quiet similar to the deep learning based optical flow computation. Song et al. [12] adopted several dilated ConvLSTMs to extract multi-scale spatiotemporal information and feed it into bi-directional ConvLSTM to obtain the spatiotemporal information. Wang et al. [21], Fan et al  [2] further adopted the human visual fixation to enable its video saliency shifting between different objects. Li et al. [22] has adopted the FlowNet based optical flow to sense the temporal information, and then these temporal information will be used to enhance the saliency consistency over temporal scale by using the ConvLSTM network. Most recently, Li et al. [1] have developed a multi-task network. As usually, it has adopted the optical flow to sense the temporal information, in which its major highlight is that it utilizes its temporal branch to accomplish its spatial branch, achieving a significant performance improvement. In  [1], though its temporal branch is able to affect its spatial branch, it has overlooked the usage of its spatial branch to interact with its temporal branch, in which the color saliency obtained by the spatial branch can indeed affect the temporal branch positively.

Compared to the conventional 2D convolution (e.g., a $3\times 3$ flat kernel), which can only sense the spatial information in single video frame, the 3D convolution (e.g., a $3\times 3\times 3$ cubic kernel) is able to sense the temporal information. As a basic computational unit, our 3D convolution itself is exactly the same as the plain 3D convolution, yet the major difference is how we use it to capture temporal information. In general, the single plain 3D convolution frequently with limited temporal sensing ability, however, our 3D convolution with fast cyclic padding scheme kills two birds with one stone:
1) It avoids the conventional padding induced performance degradation when using sequential 3D convolutions to enhance the temporal sensing ability;
2) It cyclicly uses other frames as the padding data, enhancing the temporal ability naturally without additional computational costs.

Therefore, for each encoder spatial layer, we use the 3D convolution to sense its spatial common consistency over the temporal scale as the spatiotemporal deep features, e.g., the ${ST}_{\emph{i},3}$ in Eq. 2, which is also marked by red dash line in Fig. 2. As we have mentioned in Eq. 2, the input data of our temporal model consists of 3 aspects, including the multi-scale spatial deep features ($F$), the attention maps ($A$), and the recurrent data from the precedent decoder layer ($R$). For simplicity, we use $T_{i,j},i\in\{1,2,3\}$ to denote the input deep features (i.e., 3 frames) of the temporal model in the $j$-th decoder layer. To reveal the temporal information, we use the sliding window scheme to apply the 3D convolution ($Conv3D$) over $T_{i}$, and thus the temporal model output ($ST_{i,j}$) can be fast computed by Eq. 4.

where $pad$ denote the padding data. Thus far, we have applied the 3D convolution over spatial deep features to sense temporal information, however, we found that only using single 3D convolution is incapable to obtain temporal information accurately. Thus, in our implementation, we use 3 sequential 3D convolutions in our temporal model, and thus the spatiotemporal deep features of the 2rd 3D convolution can be updated by Eq. 5.

As shown in Eq. 4 and Eq. 5, there exists 2 major problems toward the sequential usage of multiple 3D convolutions:
1) Due to the miss-aligned spatial information between different video frames, the direct usage of multiple 3D convolution easily lead to loss the tiny spatial details, blurring the object boundaries in its detection results;
2) The intuitive padding scheme (e.g., zero padding) may lead the computed spatiotemporal deep features problematical after using multiple sequential 3D convolutions.

To solve the problem 1), we simply add each $ST$ with the deep features computed by 2D convolution, e.g., $ST^{1}_{1,j}\leftarrow\{ST^{1}_{1,j}+Conv(T_{1,j})\}$, $ST^{2}_{1,j}\leftarrow\{ST^{2}_{1,j}+Conv(ST^{1}_{1,j})\}$. Meanwhile, we use the cyclic padding scheme to handle the problem 2), see Eq. 6.

However, such cyclic padding scheme is time consuming if we reorganize the input data. Therefore, we directly use the repeat operation on GPU 3 times to expand the original $\{T_{1},T_{2},T_{3}\}$ into $\{T_{1},T_{2},T_{3},T_{1},T_{2},T_{3},T_{1},T_{2},T_{3}\}$, and thus the cyclic padding can be fulfilled by using a sliding window over these expanded features, and such implementation is 5 times faster than the conventional feature reorganization, see the example in Eq. 7.

Also, we have shown the complete data flow of our temporal model in Fig. 3

The temporal information sensed by the 3D convolutions is mainly from the 3rd dimension of the adopted 3D kernels (e.g., “Spatial”:$\{3\times 3\}\times$“Temporal”:$\{3\}$), which may lead the final spatiotemporal deep features bias to the spatial domain, degenerating the performance of the temporal model. To solve it, we propose a fast temporal shuffle scheme to enhance the temporal information sensing ability of the temporal model.

Inspired by the ShuffleNet [23] which enhances its spatial feature diversity by randomly scrambling its feature orders, here, we enhance the temporal part in our temporal model by swapping the deep features between consecutive video frames. For an example, as shown in Eq. 8, we swap the deep feature $f_{a}$ in frame 1 (i.e., $ST_{1}$) with the deep feature $f_{b}$ in frame 2 (i.e., $ST_{2}$), and we swap $f_{c}$ with $f_{d}$ as well.

In fact, the above temporal shuffle can be fully implemented in GPU, which can be extremely simple and fast, see the pictorial demonstration in Fig. 4. We first sequentially divide the original $192\times 1$ deep features into 64 groups, and each group includes 3 deep features. Next, we reshape the original $192\times 1$ deep features into $64\times 3$ according to their group orders, and then we transpose it into $3\times 64$, and finally we flatten it back to $192\times 1$ deep features. In this way, we have automatically inserted the temporally neighbored spatial features into the current frame.

In our implementation, we repeat the above temporal shuffle 2 times, i.e., one for the output ($ST^{1}$) of the first 3D convolution , and another for the output ($ST^{2}$) of the second 3D convolution. Thus the complete dataflow in temporal model can be represented by Eq. 9.

where $T$ and $ST$ respectively denote the input and output of the temporal model, $S$ represents the temporal shuffle operation.

We have compared our method with 23 SOTA saliency detection methods, especially, 20 of which are video saliency detection methods, MGA [1], AGS [21], SSAV [2], PDBM [12], MBNM [31], FGRN [22], DLVS [20], SCNN [32], SCOM [33], SFLR [19], SGSP [4], STBP [34], MSTM [35], GFVM [3], SAGM [17], MB+M [36], RWRV [37], SPVM [16], TIMP [38] and SIVM [39]. We also compared our method with the most recent 3 SOTA image saliency detection methods, CPD [28], PoolNet [29], EGNet [30].

In Tab. V, we have conducted multiple component evaluations to verify the effectiveness of each component, in which the baseline represents the original spatial branch using 2D convolution only.

Regarding our temporal shuffle, it will introduce noisy information indeed, and its effect reported in Tab. V is not significant even in the case where the temporal sensing ability has been enhanced via temporal shuffle. We choose to use the attention model to compensate the lost location information during temporal shuffle, and the performance improvement after using the attention model (i.e., 3D+S+MA) is significant.

The qualitative comparisons between different components can be found in Fig. 6.

Because our method only takes 3 consecutive video frames as input each time, its sensing scope for temporal information is quite limited, leading those regions which undergo long-period static status being undetected, see pictorial demonstrations in Fig. 7. Also, this limitation is also quite common in the SOTA methods, and we believe it will may be alleviated by introducing the long-term spatiotemporal information, which deserves our future investigation.