Object-to-Scene: Learning to Transfer Object Knowledge to Indoor Scene Recognition

The proposed OFAM can extract object features based on the segmentation network. In this paper, PSPNet that is pretrained on ADE20k dataset is used as the segmentation network [24, 25]. Fig. 4 illustrates the proposed OFAM. OFAM uses Conv4_6 feature map $F\in{R^{C\times N}}$ and segmentation score map $S\in{R^{C^{{}^{\prime}}\times N}}$ that obtained from PSPNet to calculate object features $X_{obj}$, where $C$ is the number of channels, $C^{{}^{\prime}}$ is the number of objects and $N$ is the number of feature spatial positions. In this paper, $C^{{}^{\prime}}$ is set to 150 because of the object number of ADE20K, and $C$ is set to 1024 because Conv4_6 is used. For convenience, we call each spatial position in $F$ as a unit, the channel values of each unit form the feature vector $U\in{R^{C\times 1}}$ of the unit. The binary mask $M\in{R^{C^{{}^{\prime}}\times N}}$ of objects is calculated as:

$$M_{i,j}=\begin{cases}1,\ if\ max(S_{i,1},\cdots,S_{i,C^{{}^{\prime}}})\ ==\ S_{i,j}\\ 0,\ otherwise.\end{cases}$$

$(1)$

where $i$ is the unit index and $j$ is the object index. Then, the object feature vector $O_{j}\in{R^{C\times 1}}$ is calculated as:

$$O_{j}=\frac{\sum_{i=1}^{N}{(M_{i,j}*S_{i,j}*U_{i})}}{\sum_{i=1}^{N}{(M_{i,j}*S_{i,j})}}$$

$(2)$

where $i$ is the unit index and $j$ is the object index, and $N$ represents the unit number. Finally, the feature vectors stacked together to form the object features $X_{obj}\in{R^{1024\times 150}}$.

In addition to object features, relations between objects are also required to improve scene representation ability and recognition accuracy. For example, a model will be more confident about the kitchen label if stove and refrigerator have been detected together. In that case, an attention mechanism is a good choice since it can capture long-range dependencies between all objects [26]. However, only global attention mechanism is suitable for our case because object features do not include spatial relations.

Self-attention [4] and non-local [5] are one of the most effective global attention mechanisms where three nodes are calculated in parallel with 1$\times$1 convolutions and the input feature map. We call the three nodes Query ($Q$), Key ($K$), and Value ($V$) for convenience. $Q$ and $K$ are then multiplied together to calculate the correlation matrix of each unit in the feature map, and softmax is used to calculate the activation map. After that, $V$ and the activation map are multiplied together, and an additional 1$\times$1 convolution is used to form the final attention. Finally, the attention multiplies a learnable scale parameter and adds back to the input feature map to form the output. Fig. 5(b) illustrates the structure of self-attention and non-local. Although self-attention and non-local are effective, they can be further improved. For example, the attention will be more efficient if $Q$ and $K$ are calculated based on $V$, because $V$ has fewer channels compared with input features. Leveraging this insight, we propose an OAM that consists of cascaded object attention blocks as shown in Fig. 5(a). Given an input feature map $F\in{R^{C\times N}}$, $V\in{R^{\frac{C}{2\alpha}\times N}}$ is first obtained using 1$\times$1 convolutions, where $C$ is the channel number, $N$ is the unit number, $\alpha$ is a factor to control the compression rate and the output channel number, and $V=W_{v}F$. $Q\in{R^{\frac{C}{2\alpha}\times N}}$ and $K\in{R^{\frac{C}{2\alpha}\times N}}$ are then calculated based on $V$, where $Q=W_{q}V$ and $K=W_{k}V$. The object attention is then calculated as follow:

$$\beta_{Q,K}=softmax(Q^{T}K)$$

$(3)$

$$Attn=concatenate(\gamma*V\beta_{Q,K},\ V)$$

$(4)$

Where $\gamma$ is a learnable scale parameter that can be updated gradually. The intuition for why we propose object attention block is straightforward. Firstly, $V$ is used to refine input information with fewer channels (512/$\alpha$) in object attention, calculating $Q$ and $K$ based on $V$ enables them to perceive refined input information with half the computational cost. Secondly, $\alpha$ is used to compress the block and control the output channel number, the connectivity of $Q$, $K$ and $V$ allows them to be automatically compressed proportionally with the change of $\alpha$. Thirdly, concatenation mechanism in object attention avoids using extra convolution and ensures the later block perceive the refined input information and attention features separately. Table I compares the computational cost of three methods. It can be seen that our object attention block is the most efficient of the three methods under the same input and output channel number configuration. Our object attention block is also more flexible because we can adjust $\alpha$ to reduce the computational cost arbitrarily, and the channel number of $Q$, $K$, $V$, and output will be changed automatically.

Aggregation of object features is essential for accurate scene recognition. To solve this problem, we propose a GRAM that consists of a strip depthwise convolution and a pointwise convolution as shown in Fig. 6. Given an input feature map $F_{in}\in{R^{C\times N}}$, the strip depthwise convolution first aggregates object features and relations at each channel, and converts the feature map into $F_{mid}\in{R^{C\times 1}}$, pointwise convolution is then used to generate scene representation vector $F_{out}\in{R^{C^{{}^{\prime}}\times 1}}$ that has higher semantic level. Depthwise convolution combined with pointwise convolution is much more efficient compared with conventional convolution kernel [27]. The conventional depthwise convolution has a 3x3 kernel to learn local information in each channel. However, we aim to aggregate the object features and relations into a scene representation vector, and learn the global relation at the same time. Therefore, the conventional depthwise convolution is not suitable for our case. To solve this problem, we proposed a strip depthwise convolution, which operates on a list of object features rather than a patch of spatial features. The intuition for why we use this is clear. Firstly, the strip depthwise convolution can convert object feature map into a representation vector, that is more suitable for the final recognition layer. Secondly, the convolution is also able to aggregate the relations between all objects in each channel. Finally, the convolution is more efficient not only because of the depthwise characteristic, but also the avoiding of the use of large amount of fully connected layers.

Our network is implemented in the Pytorch library [28], and a single RTX 2080Ti GPU is used in the experiments. We use the most common settings to implement the experiments [1]. The batch size for all experiments is set to 256 and cross-entropy loss is used in our method. We adopt Stochastic Gradient Descent (SGD) optimizer with the base learning rate of 0.1 while momentum and weight decay are set to 0.9 and 0.0001, respectively. The learning rate is divided by 10 every 10 epochs. Training is performed for 40 epochs. Meanwhile, object features are calculated off-line to increase the training speed.

The reduced Places365 dataset is used in this paper since it is the largest scene recognition dataset with various indoor environment categories [29]. To verify the effectiveness of our method, we used two different class settings of indoor scenes for a fair comparison with other state-of-the-art methods. The first one contains 7 classes: Bathroom, Bedroom, Corridor, Dining room, Kitchen, Living room, and Office. We extract both training data and test data from the target 7 classes in Places365 and form the Places365-7classes, the class setting is totally the same as [3]. The second one includes 14 indoor scenes in home environment: Balcony, Bedroom, Dining room, Home office, Kitchen, Living room, Staircase, Bathroom, Closet, Garage, Home theater, Laundromat, Playroom, and Wet bar. We also extract both training data and test data from the target 14 classes in Places365 and form the Places365-14classes, the class setting is totally the same as [19].

SUN-RGBD is one of the most challenging datasets for scene understanding [30], which includes images from various sources: 3784 images captured by Kinect v2; 1159 images captured by Intel RealSense cameras; 1449 images from NYU Depth V2 [31] and 554 manually selected realistic scene images from Berkeley B3DO [32] captured by Kinect v1; 3389 selected frames by filtering out significantly blurred frames from the SUN3D videos [33] captured by Asus Xtion. The diversity of categories and sources makes SUN-RGBD more suitable for verifying the generalization ability of methods. We only consider the RGB images in this work, and the 7 classes that chosen in Places365 are used to test our model’s generalization ability. Notably, we only used the official test images in SUN-RGBD, and these images are evaluated by the model trained using Places365-7 classes.

To evaluate the effectiveness of the proposed OTS, we compare it with other benchmark methods on the Places365-7classes, Places365-14classes and reduced SUN-RGBD datasets. As shown in Table II, OTS significantly outperforms the ResNet50 baseline with 7.4% on Places365-7classes which shows the effectiveness of OTS. Moreover, OTS has 2% higher accuracy than [3] with only one stream. [3] used a detection network to calculate one-hot object existence features, and added an additional stream to calculate auxiliary image features.

To verify the generalization ability of OTS, we further test OTS on SUN-RGBD. It is noteworthy that SUN-RGBD is just used as a test set to verify the generalization ability of models, and all models are trained using Plces365-7classes. As shown in Table III, OTS achieves the highest accuracy compared with other methods on SUN-RGBD. The results demonstrate the generalization ability of OTS.

We further compare OTS with [19] on the larger Places365-14classes. [19] used a segmentation network to calculate Word2Vec features, and added an additional stream to calculate image features. As shown in Table IV, our OTS is 2.2% higher than [19] without any additional streams. The results show the effectiveness of our method. Better segmentation models have the potential to improve the performance of OTS, but they are not the focus of this work. In addition to evaluating the benchmark datasets, we also inference OTS in a real-world office environment, which can be found in our video supplement files.

We run ablation experiments to analyze the results of our method. Unless specified otherwise, the dataset used in ablation experiments is Places365-14classes since it is larger and contains more categories. We firstly run an ablation experiment to show the effectiveness of each module. As shown in Table V, both OAM and GRAM significantly improve the scene recognition accuracy because of the ability to capture the long-range dependencies between all objects. Meanwhile, only adding OFAM degrades the performance of the model because ResNet50 uses the features of Conv5 layer but OFAM extracts the object features from Conv4 layer as shown in Fig. 3, which means the performance gains of our method rely on the object relation construction instead of the backbone features of the pre-trained segmentation model.