Semantic-embedded Similarity Prototype for Scene Recognition

Scene recognition is a pivotal research topic within the field of scene understanding [1]. Recently, several deep neural network architectures [2, 21, 3, 22, 23, 4, 24, 5] have been used to facilitate the development of computer vision. Accordingly, some methods attempt to extract deep features for scene recognition. Xie et al. [6] propose to combine CNN with dictionary-based representations for scene recognition. Lin et al. [25] propose a hierarchical coding algorithm to transform convolutional features into the final image representation for scene recognition. However, while neural network architectures have triumphed in classical domains like image classification, their efficacy in scene recognition often falls short. This disparity is mainly due to the high inter-class similarity resulting from the existing objects across scenes. Therefore, numerous methods utilize internal object information for scene recognition [26, 27, 28].

Specifically, Herranz et al. [26] propose a method for enhancing scene classification by dividing scene images into patches of different scales and merging them. Similarly, Song et al. [27] propose using neural networks to create discriminative patch representations and build a hierarchical architecture for scene recognition. These approaches aim to extract patch-level representations from dense grids to guide scene recognition. However, using dense grids can lead to semantic ambiguity, such as a complete object being distributed across multiple patches or multiple objects residing within a single patch. Considering this limitation, some other methods [8, 7, 9, 10] use object detection to extract more certain object regions from the scene. For instance, [8] employs detection techniques to extract a high-level deep representation of objects, which is combined with the backbone to create comprehensive representations for scene recognition. Song et al. [9] propose a framework that captures object-to-object relations for representing images in scene recognition. Furthermore, some researchers have recognized the background’s crucial role in scene classification. Consequently, several approaches [11, 12, 13] incorporate semantic segmentation to extract a scene’s foreground and background. SAS-Net [11] enhances scene classification by weighting feature representations based on semantic features obtained from semantic segmentation score tensors, allowing the network to focus on discriminative regions within images. ARG-Net [12] utilizes semantic segmentation to segment regions within the scene. It then combines the feature maps derived from the backbone to establish context between regional features.

However, either multi-branching (patch-level) or object-assisted approaches (object detection, semantic segmentation) impose heavy computational burdens on the model, rendering it unsuitable for practical deployment on resource-constrained edge devices. In contrast, the proposed similarity prototype improves the discriminative ability of networks by introducing semantic prior knowledge during training. This approach enables the achievement of higher recognition accuracy when applying the trained model in real-world scenarios. In this context, the model can rely solely on the exceptional feature extraction capabilities of the backbone network itself, without the need to shoulder additional computational burdens like semantic segmentation.

In recent years, it has been widely acknowledged that training DNNs with hard labels tends to lead to overfitting [29]. Consequently, label softening has been utilized in various applications. One such technique is Label Smoothing Regularization (LSR) [16], which involves taking an average between the hard labels and a uniform distribution over labels to soften labels. This approach prevents the network from rapidly overfitting due to overconfidence. Another method, Bootstrapping [30] introduces two label softening methods, Bootsoft and Boothard, to mitigate the negative impact of noisy labels. Bootsoft applies predictive distributions to smooth labels, while Boothard uses predictive categories to soften labels. The authors in [31] propose to embed images and labels into a latent space to capture their inherent relationships. By doing so, they leverage this latent space to regularize the network and improve classification performance. Online Label Smoothing (OLS) [32] suggests a strategy where soft labels are generated based on the model’s prediction statistics for the target category. This online label smoothing further enhances the regularization ability of soft labels. Similarly, label smoothing is introduced in [33] within a hybrid loss function to assist with variable hyperparameters and improve the accuracy of model loss calculations.

Unlike the mentioned approaches above, our main objective in label softening is to embed the inter-class correlation in the proposed similarity prototype into the labels. This embedding guides the network to fit in a more discriminative direction during training. Of course, due to the weakened confidence of the target classes, our softened labels still have the ability to mitigate the overfitting of the network.

Deep Metric Learning (DML) aims to use deep networks to map data into a nonlinear embedding space, where similar data are close together, and dissimilar data are far apart [34]. In general, DML can be divided into two main categories. The former is the Siamese network (contrastive loss) [17], a method based on one pair of samples at a time, which greedily increases the similarity of positive pairs and reduces negative pairs. The latter is the Triplet network (triplet loss) [35], which pulls the anchor sample closer to the positive sample than the negative sample by a fixed margin. Furthermore, PSDML [36] proposes the quadruple loss, which sets boundaries for anchor-positive and anchor-negative pairs, respectively. N-pair loss [37] and Structured loss [38] propose to explore the relationship between multiple pairs of positive and negative samples simultaneously within a training batch, utilizing more information to achieve a faster fit. However, none of the above optimization approaches are flexible enough as they apply the same distance threshold to all pairs, regardless of the potential variance in their similarities and dissimilarities. With this in mind, several methods [39, 40] attempt to adaptively distinguish similarity differences based on the network optimization state or the degree of difference between sample pairs. Nonetheless, these methods rely on the discriminative capacity of the network, thus limiting their effectiveness. In contrast, our approach utilizes a statistically derived similarity prototype as prior knowledge, allowing us to assign suitable boundaries to each class-level sample pair, thereby improving the metric’s performance.

Different from previous semantic-guided methods, we choose to use semantic knowledge to help networks achieve higher scene recognition performance without additional computational costs in practice. This means that computationally intensive and time-consuming object extraction tasks are no longer performed in real-world applications of the network. For this reason, instead of focusing on exploring the intra-scene object region context, we emphasize discovering the intrinsic similarity among scene classes with the help of semantic knowledge. As we introduced in the previous section, the inter-scene similarity can help the network to find a correct fitting direction during training, i.e., to devote more effort to discriminating similar classes.

Specifically, we first derive semantic representations for each scene category using semantic segmentation techniques, training data, and statistics. These semantic representations are then used to construct a similarity prototype, which contains detailed inter-class similarity knowledge. Furthermore, we propose two approaches for leveraging the similarity prototype to assist in network training: Gradient Label Softening (GLS) and Batch-level Contrastive Loss (BCL). In this way, we make the trained network achieve higher performance without adding any additional network parameters or computational costs in practice.

Such a plug-and-play technique can be easily incorporated into most scene representation learning approaches, all it takes is just a few lines of code with minimal computational overhead during training.

The reasoning process for the similarity prototype can be outlined in two steps: semantic representation and label correlation. As shown in Fig. 2, we initially use the semantic knowledge to derive a semantic representation for each scene class. These class-level semantic representations serve as the basis for investigating inter-class label correlations, which are subsequently embedded into our similarity prototype.

In this section, we propose to utilize the similarity prototype to assist model training from a label-softening perspective. Unlike Label Smoothing Regularization (LSR) [16] that only softens labels with human-defined smoothing values, with similarity prototypes providing prior knowledge, we quantify the degree of label softening in terms of the similarity among different scene classes.

To be concrete, given a dataset with $C$ scene classes, we can reason about its similarity prototype $S\in{\mathbb{R}^{C\times C}}$ (refer to Section 3.2 for details). We first normalize the row vectors of $S$ to obtain $S_{norm}$ in usage soft label form:

where $\oslash$ represents element-level division, ${E_{C\times C}}$ represents the All-ones matrix in $C\times C$ dimensions.

However, there are differences in the target category confidence of the individual soft labels in $S_{norm}$, which potentially leads to the class imbalance in model training. To solve this, we propose a further refinement to $S_{norm}$, ensuring that each soft label has the same target category confidence. We achieve this by modifying the diagonal elements of $S$.

Specifically, given ${S_{c,c}}$, its corresponding target category confidence $\sigma$ in the soft label can be calculated by:

After conversion, given the target confidence $\sigma^{\prime}$, its corresponding ${S_{c,c}}$ should be assigned as:

Next, we take the maximum target category confidence in $S_{norm}$ as the unity confidence, and update $S$ to obtain ${S^{1}}$:

Then, using Eq. 5 again, we can obtain the available normalized matrix $S_{norm}^{1}$. In $S_{norm}^{1}$, different class labels possess the same confidence for the target category, while maintaining a reasonable confidence difference between the target category and non-target categories.

Since the inference process of the similarity prototype overlooks specific details like colors within scenes, intuitively, the trained model’s ability to differentiate between various scene categories should surpass that of the pure knowledge statistics (similarity prototype). Consequently, instead of constraining the target category confidence $\sigma^{\prime}$ to $\sigma^{\prime}_{0}$, we gradually boost the target category confidence during training, thereby driving the network to progressively improve its discriminative ability in the appropriate direction. With the increase of training epoch, the confidence $\sigma^{\prime}$ of the target category can be formulated as:

where $STEP$ represents the number of epochs using soft labels. We use $\sigma^{\prime}$ in place of $\sigma^{\prime}_{0}$ in Eq. 8 and reapply Eq. 8 and Eq. 5 to update class labels during training, as outlined in Algorithm 2.

In the above process, we maintain the confidence difference among scene categories based on the similarity prototype, ensuring that the network fully uses semantic prior knowledge. After $STEP$ epochs (when $\sigma^{\prime}>0.99$), all the class labels are converted into a hard label. An example of the changes in $S_{norm}^{1}$ across epochs is shown in Fig. 4.

With the change of epoch, the soft labels supplied by $S_{norm}^{1}$ will be used as fitted endpoints for model training along with the cross-entropy loss function, as follows:

where $B$ represents the number of images in a minibatch, $p$ represents the prediction probabilities output by the network, and $target$ represents the corresponding ground truth of the mini-batch.

In this section, we use the similarity prototype to support model training from a metric learning perspective (i.e., contrastive loss). Contrastive loss aims to optimize the feature space by maximizing similarity within the same class and minimizing similarity between different classes. By integrating the similarity prototype into contrastive loss, we eliminate the need for manual threshold definition and instead establish more suitable boundaries based on statistical similarity. Fig. 5 visually represents the batch-level contrastive loss operation. Let $x_{i}^{m}$ and $x_{j}^{n}$ be a pair of inputs, the contrastive loss ${\mathcal{L}_{contrastive}}$ can be formulated as:

where ${p_{x_{i}^{m},x_{j}^{n}}}$ represents the feature similarity between $x_{i}^{m}$ and $x_{j}^{n}$. If a pair of inputs is from the same class, the value of $\gamma$ is 1; otherwise, its value is 0.

To ensure the plug-and-play nature, inspired by N-pair loss [37] and Structured loss [38], we compute contrastive loss based on mini-batch. Specifically, assume that the input to the network is a mini-batch of $B$ randomly sampled scene images. Denote the output prediction of the network as $p\in{\mathbb{R}^{B\times C}}$ and the ground truth as $target\in{\mathbb{R}^{B}}$. Using $target$ as the row index of similarity prototype $S$, we can calculate the similarity prototype variant ${S_{batch}}\in{\mathbb{R}^{B\times B}}$ among the $B$ samples in $target$ via

Next, we can calculate the similarity matrix ${p_{matrix}}\in{\mathbb{R}^{B\times B}}$ among the $B$ samples within the network’s prediction $p$ using cosine similarity or Euclidean distance.

Since the inference process of the similarity prototype overlooks details like colors with scenes, the discriminability of the trained model should surpass the statistically derived similarity prototype. Therefore, the similarity between scene classes predicted by the network should be lower than the derived inter-class similarity. Accordingly, an inter-class contrastive loss matrix ${\mathcal{L}_{inter\_matrix}}$ can be designed via

Note that we only need to consider the anomalous case, so the inter-class contrastive loss ${\mathcal{L}_{inter}}$ is formulated by

However, the above approach only considers samples from different scene classes, failing to leverage the feature information of samples from the same class within the minibatch. The reason is that the similarity between samples within a class cannot be effectively measured using the similarity prototype, and it is unrealistic to achieve a similarity of ”1” between different samples of the same class. However, in the scene classification task, we only need to ensure that the model outputs a higher similarity between samples of the same category than its similarity to other categories (as shown in Fig. 5). Therefore, we improve the contrastive loss by a layer of intra-class contrast measures to better utilize the feature information of samples from the same class.

To compute the intra-class contrastive loss, we first generate a variant of the similarity prototype $S$: the self-similarity matrix $S^{\prime\prime}$. This matrix solely focuses on intra-class similarity. It sets the intra-class similarity threshold to the maximum value obtained from the similarity between the scene class and other scene classes:

Similarly, we can obtain the intra-class similarity prototype variant ${S^{\prime\prime}_{batch}}\in{\mathbb{R}^{B\times B}}$ among the $B$ samples in $target$ via Eq. 12. Then, the intra-class contrastive loss matrix ${\mathcal{L}_{intra}}$ can be designed via

As for whether the measure of intra-class contrastive loss is effective, we conduct experiments for comparative discussion in Section IV.

Of course, the contrastive loss is still secondary after all, and the final loss is generated by the combination of the cross-entropy loss and the contrastive loss:

Semantic Segmentation: Vision Transformer Adapter [41] that is pretrained on the ADE20K dataset [42] is used as the semantic segmentation network. The network outputs a label map $M$, which has the same size as the input image. Each pixel $\left({w,h}\right)$ in $M$ is assigned a value ${M_{wh}}$, representing the semantic label of the corresponding pixel in the input image.

Hyperparameters: The proposed similarity prototype and its derivative strategies can be easily integrated into various models. We conduct a series of experiments to demonstrate the effectiveness and generalization of our methods using eight commonly used pretrained models: ResNet50-IN1k [2], VGG16-IN1k [21], MobileNetV3-IN1k [3], ShuffleNetV2-IN1k [22], MobileVIT_S-IN1k [23], ConvNextBase-IN22k [5], VITBase-IN22k [24], and SwinTransformerBase-IN22k [4]. The suffix “IN1k” indicates that the model is pretrained on the ImageNet 1k dataset [43], while “IN22k” indicates pretrained on the ImageNet 22k dataset [43].

We train all models using Adam optimizer [44]. For the validation on the MIT-67 and SUN397 datasets, we set the initial learning rate of the last fully connected layer to 0.001, and the learning rate of the other pretrained layers to 0.00001 (decayed by a factor of 0.1 at the 10th, 15th, and 20th epochs). The batch size is set to 32, and the weight decay to 0.00001. We train the models for a total of 100 epochs. On the Places365 dataset, due to its larger size and faster convergence, we modify the batch size to 64 and train all models for 30 epochs. When training the models pretrained on ImageNet 22k, considering their larger parameter sizes and faster convergence, we also train them for 30 epochs. For the MobileVIT model, we increase the initial learning rate by a factor of 10 to improve fitting efficiency. All experiments are conducted on a single NVIDIA 3090 GPU using the PyTorch and SAS-Net [11] open-source framework.

MIT-67 Dataset [18] comprises 67 indoor scene classes with a total of 15620 images. Each scene category contains a minimum of 100 images. In line with the recommendations of [18], each class has 80 images for training and 20 for testing.

SUN397 Dataset [19] is a comprehensive dataset covering indoor and outdoor scenes. It encompasses 397 scene categories, with 175 indoor and 220 outdoor categories, all comprising at least 100 RGB images. Following the evaluation protocol in the original paper, we randomly select 50 images from each scene class for training and another 50 for testing.

Places365 Dataset [20] is one of the largest scene-centric datasets, containing approximately 1.8 million training images and 365 scene categories. To visualize the effect of our proposed method, this paper uses a simplified version known as Places365-7 and Places365-14. Places 365-7 consists of seven indoor scenes: Bath, Bedroom, Corridor, Dining Room, Kitchen, Living Room, and Office. Places 365-14 contains 14 indoor scenes: Balcony, Bedroom, Dining Room, Home Office, Kitchen, Living Room, Staircase, Bathroom, Closet, Garage, Home Theater, Laundromat, Playroom, and Wet Bar. For the test set, we use the same setup as the official dataset.

This section applies the similarity prototype-based Gradient Label Softening (GLS) strategy to train multiple models. We evaluate GLS using multiple models on the MIT-67 and SUN397 datasets and compared them to the baseline (trained with hard labels) and LSR [16]. The results are presented in Table 1.

It is evident that the proposed GLS significantly improves the scene recognition performance on both lightweight and complex models, demonstrating its robustness across different networks. Specifically, when evaluating on the MIT-67 dataset, applying GLS to ResNet50 yields an accuracy of 79.403%, surpassing the baseline of 2.91%. In contrast, traditional LSR shows limited improvements and, in some cases, even negatively affects certain networks. For example, on the SUN397 dataset, ResNet50 with GLS can achieve 62.06% accuracy, which improves ResNet50 alone by 1.637% and ResNet50 with LSR by 3.032%, respectively. It is worth noting that the SUN397 dataset comprises 397 distinct scene classes, further highlighting the suitability and performance of our method on large-scale datasets.

Moreover, the benefits of GLS are more pronounced in lightweight models. For example, MobileVit’s accuracy on the MIT-67 dataset improved by 3.582% with GLS. In contrast, for SwinT with a large parameter count, GLS only improves its accuracy by 0.349%. The reason for this discrepancy is that our approach works mainly by improving the feature discrimination ability of the model itself. SwinT, due to its large number of parameters and advanced network architecture, has performed well (88.681%) under the baseline strategy. Thus, the potential for improvement is relatively limited. However, when faced with the more complex SUN397 dataset, SwinT has more potential for feature discrimination improvement. In this case, our GLS enables a significant improvement in both lightweight and complex models. This further validates the importance of utilizing semantic similarity knowledge to guide network training.

In this section, we apply the similarity prototype-based Batch-level Contrastive Loss (BCL) to assist model training. We compare its performance with baseline (i.e., training models by only cross-entropy loss based on hard-label) as well as the traditional Contrastive Loss (CL) strategy [17]. In CL, the similarity expectation within the same category is set to ”1” and ”0” for different categories. These strategies are evaluated using multiple models on the MIT-67 and SUN397 datasets, and the resulting outcomes are presented in Table 2.

The results in Table 2 indicate that, in most cases, BCL yields improved performance over the baseline and generally outperforms the traditional CL strategy. Specifically, when applied to the MIT-67 dataset, BCL-trained ShuffleNet exhibits a performance improvement of 2.014% compared to the baseline, while the traditional CL strategy only yields a 1.044% improvement. On the SUN397 dataset, BCL-trained ConvNext shows a performance boost of 0.917% over the baseline, while the traditional CL results in a 0.043% decrease. This implies that the traditional CL excessively emphasizes similarity within the same categories and promotes dissimilarity between different categories, consequently impairing recognition performance. These findings further emphasize the feasibility and necessity of incorporating the similarity prototype to provide prior knowledge during training.

In addition, comparing results in Tables 2 and 1, we can observe that the BCL has a relatively limited effect in assisting models in most cases. This phenomenon is expected, as contrastive loss algorithms [17, 35, 36, 37, 38] typically have high requirements on the within-batch images and require careful selection of appropriate training images to ensure effectiveness. In contrast, to maintain the plug-and-play nature, we apply the BCL directly to randomly extracted mini-batches without imposing specific constraints on the selection of images. This practice inevitably limits performance improvement but is not the focus of our study. We will further analyze the complementary performance of GLS and BCL in Section 4.6.

Meanwhile, we observe that BCL may lead to performance degradation in specific scenarios. For instance, when evaluating MobileNet on the MIT-67 dataset, all three BCL strategies, except for the ”cosine_mean_inter” strategy, negatively affect the model’s performance. However, it is important to note that the traditional CL resulted in an even greater performance degradation. This outcome could potentially be attributed to the inapplicability of the contrast loss strategy (batch level) to MobileNet’s evaluation on the MIT-67 dataset. Nonetheless, in situations where such a conflict occurs, BCL applied under the ”cosine_mean_inter” strategy manages to further improve the model performance, again demonstrating the significance of using the similarity prototype to provide prior knowledge during model training.

In this section, we employ both Gradient Label Softening (GLS) and Batch-level Contrastive Loss (BCL) strategies to assist model training. We evaluate their effectiveness on the MIT-67 and SUN397 datasets using multiple models, and the experimental results are presented in Table 3.

The result of Table 3 shows that the combination of GLS and BCL significantly improves the accuracy of the trained models compared to the baseline. For example, on the MIT-67 dataset, utilizing both GLS and BCL strategies leads to a 4.552% accuracy gain for MobileVIT, while on the SUN397 dataset, ConvNext achieves a 2.731% accuracy improvement. Importantly, this improvement is achieved without any increase in network parameters or computational cost, which makes this result particularly remarkable. This outcome strongly underscores the importance and generalization capability of the proposed similarity prototype in improving model performance.

In Table 3, we also explore the combination of LSR [16] and CL [17] for model training. It is observed that this combination offers only a marginal improvement in baseline accuracy compared to our GLS and BCL, and in some cases, it even negatively affects model performance. For example, when evaluating MobileNet on the MIT-67 dataset, combining GLS and BCL yields a 1.866% increase in baseline performance. In contrast, combining LSR and CL results in a 0.388% decrease. This contrast further confirms the significant role of our similarity prototype in providing prior knowledge during the model training process. Also, it supports our earlier observation (Section 4.5) that MobileNet is less compatible with general metric learning strategies on the MIT-67 dataset.

Next, we compare the best performance obtained using the GLS strategy, the best performance obtained using the BCL strategy, and the best performance obtained when the two strategies are applied simultaneously. These results are summarized in Table 4.

Table 4 shows that the improvement in model performance using the BCL strategy, compared to the GLS strategy, is somewhat limited. This is because, to maintain the plug-and-play nature, we apply the BCL directly to randomly extracted mini-batches without imposing specific constraints on the selection of images. BCL differs from many contrastive loss algorithms [17, 35, 36, 37, 38], which emphasizes the careful selection of appropriate training images for better performance results. Further analysis of Table 4 reveals that, in most cases, there are complementary effects between GLS and BCL strategies. By incorporating these two training strategies simultaneously, we typically achieve higher accuracy than using either strategy alone. This finding further validates the potential of the proposed similarity prototype, indicating that there is still room for further development of the similarity prototype.

To comprehensively comprehend the functioning of our method for model training, we select the visualizations produced by training ResNet50 with hard labels as a baseline. We then train it using the proposed strategy and evaluate its performance on two simplified versions of the validation set from the Places365 dataset, with the results shown in Table 5. In addition, we extract the feature maps generated by the trained model. To provide a visual representation of these feature embeddings, we employ t-SNE, a dimensional reduction technique. By plotting the 2-dimensional feature representation of the embeddings in Fig. 8, we can effectively visualize the distribution of the images in the validation set. Each point on the plot corresponds to an individual image, and points sharing the same color indicate images belonging to the same category. Note that the proposed strategy used in this part is based on the cosine-based similarity prototype and the BCL strategy is ”nonzero_inter_intra.”

Upon observing Fig. 8, it can be concluded that employing any of our proposed strategies for model training has a significant positive impact on the network’s discriminative ability compared to the baseline. The trained model demonstrates enhanced capability in distinguishing different scene categories and exhibits superior aggregation of scene instances under the same category. Regarding the effect of feature visualization, we can see that both the proposed GLS and BCL strategies yield similar outcomes. Moreover, intuitively, the simultaneous application of these two auxiliary strategies results in a more discriminative network with clearer boundaries between different scene classes. This observation aligns with the experimental results presented in Table 4 and 5, providing additional evidence of the potential of the proposed similar prototype. In summary, based on the qualitative visualization and analysis presented in this section, it can be confidently affirmed that our proposed method exerts a highly impressive aiding effect on model training.

To demonstrate the superior computational efficiency of our approach in real-world deployments, we conduct a comparative analysis to elucidate the difference in computational cost between local training and practical deployment. Specifically, we use ConvNext_Base [5] as the backbone network and VIT Adapter [41] as the semantic segmentation network, with the respective FLOPs and Inference throughput detailed in Table 6. The FLOPs values are sourced from the original papers. Regarding Inference throughput, considering that the two papers use different measurement devices, we run two modules on a 3090 GPU to measure Inference throughput for a fair comparison. Since GLS and BCL occupy very little FLOPs during training, we do not show them in Table 6. Note that since GLS and BCL are no longer implemented in the practical deployment, this omission does not affect our conclusions.

In Table 6, it can be observed that our approach relies only on the trained backbone network for real-time scene recognition. By effectively eliminating dependence on semantic segmentation, we achieve a significant decrease in the model’s FLOPs, resulting in a substantial enhancement of inference throughput for practical applications. This result aligns with our initial expectations and strongly demonstrates the clear advantages of our approach when implemented in practice.

Additionally, to the best of our knowledge, all existing object-assisted scene recognition methods require the object information extraction process during practical deployment. Given that the object information extraction process (Semantic Seg [41]) shown in Table 6 has a throughput far lower than that of a typical image classification network (ConvNext [5]), we can infer that our method will operate much faster in real-world deployments compared to other object-assisted methods. This further highlights the significant relevance of our proposed similarity prototype for object-assisted methods.

Furthermore, removing semantic segmentation in real-world applications alleviates the requirement about the Inference speed of the selected semantic segmentation technique. For instance, the VIT Adapter’s limitations in inference speed make it unsuitable for real-world deployments. However, our method only uses this segmentation technique for model training, allowing us to leverage its precision advantages while maintaining practical applicability.

In this part, we apply the proposed similarity prototype-related strategies to an established semantic-guided scene recognition method, DGN-Net [45]. To ensure a fair comparison, all experiments in this part use training hyperparameters identical to those of the original DGN-Net. Initially, we reproduce the baseline performance of DGN-Net. Subsequently, we integrate our similarity prototype-related strategies into the training process of DGN-Net. The experimental results are displayed in Table 7. Note that the proposed strategy used in this part is based on the Euclidean-based similarity prototype and the BCL strategy is ”nonzero_inter_intra.”

As shown in Table 7, simply adding the proposed strategies on top of DGN-Net improves state-of-the-art performance across all evaluated datasets: MIT-67 (+0.97%), SUN397 (+0.541%), Places365-7 (+0.571%) and Places365-14 (+0.5%). These results further demonstrate the superiority and generalization of the prior knowledge provided by our similarity prototype for scene recognition.

We conduct a comparative analysis between our approach and existing state-of-the-art methods. As shown in Table 8 and 9, these comparisons are conducted on several public datasets: MIT-67, SUN397, Places365-14, and Places365-7. We categorize these methods based on whether they require object information extraction techniques in practice (marked in the Semantic column). We use two models for comparison: one that does not require object information (ConcNext [5] or ResNet50 [2]) and another that does (DGN-Net [45]). By combining these models, we conduct a fair comparison between the proposed method and existing methods.

The results in Table 8 and 9 reveal that our method outperforms most previous methods. Specfically, methods [6, 25, 26, 27, 28] incorporating factors such as multi-scale information or multi-model combination for scene recognition. In contrast, our method, guided by inter-class correlation knowledge in the similarity prototype, achieves superior performance with only a single-branch and single-scale architecture. Notably, while CSDML [46] also utilizes inter-class correlation to aid scene recognition, it adopts a traditional metric learning perspective to build class-level knowledge, ignoring the use of object information within scenes, and thus performs sub-optimally compared to our method. These comparisons not only prove the effectiveness of the proposed similarity prototype but also highlight the importance of incorporating object knowledge for scene recognition.

The focus then shifts to object information-assisted methods. Methods [7, 47, 49] also use statistics to process object information within scenes to aid scene recognition. However, they rely on inter-object correlation to enhance feature discriminability, making them dependent on the precision of object information extraction techniques. In particular, their performance is more restricted when the precision of object information extraction is limited for a specific image. In contrast, our approach begins from a more generalized perspective, i.e., using statistics combined with object information to infer inter-class correlations. This knowledge can fundamentally improve the model’s discriminability during training, making it no longer limited by the precision of object information extraction techniques in practice, and thus achieving superior performance.

Overall, all previous object-assisted methods [7, 10, 11, 12, 13, 45, 47, 48, 49, 50] require substantial computational costs to extract object information within scenes in practical applications. In contrast, our method uses object knowledge only during training to improve the network’s feature extraction ability, avoiding the computational burden of object information extraction in practical applications. Nevertheless, despite no longer using object knowledge in practice, our method still achieves superior recognition performance than most object-assisted methods. This finding once again emphasizes the superiority and practicality of our similarity prototype.

Furthermore, since our approach relies solely on backbone networks for scene recognition, some semantic-guided methods, such as DGN-Net [45], inevitably achieve higher accuracy. However, incorporating our proposed strategies into DGN-Net can further improve its performance across all evaluated datasets. This result again demonstrates the potential of our similarity prototype, which can serve as an auxiliary strategy to further enhance state-of-the-art methods.

In summary, Table 8 and 9 provide significant evidence for the superiority and practicality of the proposed similarity prototype in advancing scene recognition.