Searching for internal symbols underlying deep learning

We note that ‘Second Thought Certification’ (STCert) proposed in an earlier study [22] can be used to identify ROIs related to visual objects with high accuracy. STCert is a two-step process (Fig. 1). First, it uses foundational segmentation models [23, 24] to detect ROIs related to DNNs’ predictions. For instance, if a DNN predicts a tiger, STCert asks the segmentation model to return the bounding box (i.e., ROI) of the tiger in the picture. Second, it crops ROIs and uses them to confirm or reject DNNs’ original predictions. STCert provides a more reliable ROI estimates than a single step segmentation, since its segmentation is crosschecked with a classifier. That is, STCert relies on ‘self-consistency’. As such, we used STCert to identify ROIs in this study to extract ROIs from randomly chosen 200 training examples for each class in Mixed_13; see Supplementary text for more detail. Since Mixed_13 contains 78 classes (Supplementary Table 1), the total number of training examples is 15,600. Among them, we first selected examples, on which DNNs’ predictions were correct to minimize the bias induced by the incorrect predictions/decisions of DNNs (i.e., incorrect mapping between ROI and the inputs’ labels). For some of the images, STCert did not find ROIs or found multiple ROIs, which means the actual number of ROIs is determined by a few factors, not just the number of training examples.

Once ROIs were identified, we sampled activations ($H_{i}$) of hidden neurons, whose receptive fields correspond to ROIs. To reduce noise and filter out non-essential information, we used ROI-pooling [21] to convert hidden layer activations into 3-by-3 activation maps. That is, the size of hidden layer activation maps can be arbitrary depending on the size of objects, but ROI-pooling always turns them into 3-by-3 activation maps. It should be noted that individual activations in 3-by-3 activation maps reflect pixels at different spatial locations. As we assumed that symbols are collectively encoded neurons that share the same receptive field, we built vector codes (referred to as activity vectors below) by collecting activations at the same location in the maps across channels. For instance, there are 512 channels in ResNet18’s layer 4, and we have 9 location-specific activation maps in each channel. Here, we aggregated activations from the same location in the activation map from all 512 channels and then created 9 activity vectors, each of which contains 512 components from the layer per image.

In this study, we analyzed 4 hidden layers of popular DNNs, ResNet18, ResNet50, VGG19, DenseNet121, and Vision Transformers (ViT) [17, 25, 26, 27]. ResNet and DenseNet consist of 4 composite blocks, and their outputs were analyzed. VGG19 contains 5 max pooling layers, whose outputs roughly correspond to ResNet’s composite block’s outputs. We selected the last 4 max-pooling layers and analyzed them, as they match the layers of ResNet and DenseNet. For ViT consisting of 12 layers, we chose the 4 layers (layer ids: 3, 6, 9, 12) . We used Pytorch [28] and Timm [29] python deep learning libraries to implement these models. A workstation with Intel Core9 CPU and RTX 4090 was used for all our experiments and analyses.

We turned to unsupervised clustering analysis to identify representative activity vectors equivalent to symbols in this study. We also note that many activity vectors contain low values. With these low-value activity vectors, clustering algorithms consider the zero-valued vector as the most prominent cluster center, which is not informative. Thus, we removed the vectors consisting of values lower than the mean activity value estimated on each layer. Next, we used UMAP analysis [30, 31] to further reduce them to 3-dimensional vectors; see Table 1 for the parameters used in the analysis. These three dimensional vectors were analyzed to detect symbols via clustering. It should be noted that the exact number of clusters existing in the activity vector space are unknown, and the result of clustering analysis strongly depends on the number of predefined clusters. To address this issue, we used X-means clustering [32] to automatically discover an optimal number of clusters (i.e., the number of symbols). We note that X-means is an extension of Kmeans clustering, which uses Bayesian Information Criterion (BIC) to determine the optimal number of clusters, and a python package ‘pyclustering’ [33] was used to implement X-means in this study. In our analysis, we restrict the maximum number of classes to 1,000. Table 2 shows the identified number of clusters in 4 layers of DNNs tested. As shown in the table, the optimal numbers of clusters in the early layers are smaller than 1,000, but those in the late layers hit the maximum value, suggesting that homogeneity of activity vectors decreases, as the layers go deeper.

Then, we asked if the identified symbols (i.e., cluster centers) can be linked to semantic meanings of inputs (i.e., labels) by correlating them with inputs’ labels. Specifically, for each image ($img_{k}$), we obtained ROIs ($ROI_{m}$) and 9 symbols ($S_{mn}$, where $n=1,...,9$). These 9 symbols $S_{mn}$ were correlated with the labels ($l_{m}$) of $ROI_{m}$, which were determined by the label of $img_{k}$ containing $ROI_{m}$. That is, Correlation Map ($CM(i,j)$) evolves over all identified ROIs obtained from the training examples of Mixed_13 according to Eq. 1.

$$CM(i,j)=\begin{cases}CM(i,j)+1&\text{if $i=S_{mn}$, $j=l_{m}$}\\ CM(i,j)&\text{otherwise}\par\end{cases}$$

(1)

, where $n=1,...,9$, and $CM(i,j)$ is the component of Correlation map at $i^{th}$ row and $j^{th}$ column; $S_{mn}$ denotes the $n^{th}$ (out of 9) symbol related to $m^{th}$ ROI; where $l_{m}$ is the label of $m^{th}$ ROI (i.e., $img_{k}$).

If a symbol is associated with a specific class, it would appear frequently when the class object is presented, but it would not appear when other class objects are presented. Fig. 2 shows the correlation $CM(i,j)$ between 100 randomly chosen symbols from 4 layers of ResNet18 and 78 classes. $Y$-axis denotes 100 randomly chosen symbols, and $x$-axis denotes the label (i.e., class). We found two types of symbols. First, some symbols were associated with a broad range of examples. Second, some symbols were selectively linked to specific classes. Symbols specific to classes were rare in early layers but became more common in late layers. We found equivalent results in ResNet50, VGG19 and DenseNet121 (Fig. 3). Interestingly, we note that many symbols can be merged together (Fig. 4), suggesting that many symbols can be redundant. Thus, we did not increase the maximum number of clusters above 1000. These results suggest that some symbols could be directly linked to the labels of inputs, and thus, one could infer the labels from their symbols in the hidden layers.

To test this possibility, we obtained symbols associated with test examples of Mixed_13 and asked if they could be used to predict the labels of inputs (i.e., ROIs). In this experiment, we collected ROIs from all test images whether or not DNNs make correct predictions. From these ROIs, we collected activity vectors and obtained the test symbols by identifying the nearest cluster centers. These ‘test symbols’ were used to predict the likelihood of each class by using $CM(i,j)$ (Eq. 1). As we aimed to compute the probability of the class, the correlation map $CM(i,j)$ was normalized along with a row (i.e., over classes) using SoftMax (2). By using the normalized Correlation Map $P(i,j)$, we calculated the probability $P(i,j)^{m}$ that $m^{th}$ ROI belonged to the class $j$ when the symbol $S_{i}$ was observed (Eq. 2). That is, symbol indices were used as hash keys for a look-up table.

$$P(i=S_{i},j)^{m}=\frac{e^{CM(i=S_{i},j)}}{\sum_{n=1}^{78}e^{CM(i=S_{i},n)}}$$

(2)

As we obtained 9 symbols from each ROI, we took average probability over 9 (Eq. 3) and then determined the most probable class as the prediction $Pr_{m}$ (Eq. 4).

$$P(j)^{m}=\frac{1}{9}\Sigma_{i=1}^{9}P(i,j)^{m}$$

(3)

$$Pr_{m}=argmax_{j}P(j)^{m}$$

(4)

Fig. 5 shows the accuracy of the symbol-based predictions for all 5 ImageNet models and ResNet50 trained Oxford-IIT PET dataset. We note that the prediction accuracy is around 80% in layer 4 (i.e., $5^{th}$ Max-pooling in VGG19 and $12^{th}$ embedding layer in ViT) and much lower in the earlier layers. These accuracies of ImageNet model cannot be directly compared to the models’ accuracy because the choice is limited to one out of 78 classes, not 1000 classes. However, the accuracy in layer 4 and the earlier layers were significantly higher than chance (1/78), supporting that the identified symbols were associated with semantic meanings. ResNet50 trained on Oxford-IIT PET dataset also show a similar level of accuracy.

If symbols are associated with semantic meanings, they could also be associated with DNNs’ predictions. We tested this possibility by correlating the symbols and DNNs’ predictions on Mixed_13 training examples (once again, 200 examples per class) instead of inputs’ labels. As stated above, we used the symbols to infer their predictions on test examples and found that the symbols are predictive of DNNs’ answers on test examples (Fig. 5B). We made equivalent observation from ResNet50 trained on Oxford-IIT PET dataset. These observations raise the possibility that DNNs could utilize symbols to make decisions.

We also tested the predictive power of symbols depending on the number of symbols. Specifically, we set a number of symbols from the 4 layers of ResNet18 to a specific number by using KMeans clustering instead of X-Means clustering. Then, we measured the predictive power of extracted symbols on labels of test examples by evaluating the accuracy of symbol-based predictions. Specifically, we tested 5 different numbers of symbols, which are 500, 1000, 1500, 2000 and 2500. As shown in Table 3, the predictive power (i.e., accuracy) of symbols does not monotonically increase, as the number of symbols increases, suggesting that the predictive power of symbols is not sensitive to the exact number of symbols.

In the analysis above, we used $CM(i,j)$ to infer the most probable class by using the symbol indice as a hash key and $CM(i,j)$ as a look-up table. However, this look-up table can be interpreted in a completely opposite direction. If we know the label $j$ of inputs, we could infer the likelihood of individual symbols $S_{i}$ appearing in DNNs. With this in mind, we defined ‘Expected Symbol Score (ESS)’ per layer with respective label/class $j$ (Eq. 5).

$$ESS_{j}=\frac{1}{9}\Sigma_{S_{1},...,S_{9}}P(S_{i},j)^{m}$$

(5)

, where $S_{i}$ denotes the indices of 9 symbols obtained from each ROI. That is, $ESS_{j}$ with respective class $j$ is the mean value of 9 probabilities from $P(i,j)^{m}$; see Eq. (5). If all observed symbols $S_{i}$s are exclusively correlated with class $j$, $P(i,j)=1$, $P(i,k\neq i)=0$, we can expect that $ESS_{j}=1$, and $ESS_{k\neq i}=0$. By contrast, if symbols contain no information related to classes, we can expect that $P(i,j)=ESS_{j}=1/78$, where 78 denotes the number of classes.

Detecting OOD examples has been a central challenge in computer vision models. During training, models are forced to choose one of the predefined choices. After training, they must choose the most probable answer on any given input, even when the inputs are not related to its training. By nature, predictions on OOD examples are random and should be ignored. To address this shortcoming, OOD detection algorithms have been proposed to inform DL models (and its users) when to ignore inputs; see [34] for a review.

We assumed that hidden layer responses (i.e., symbols), highly optimized for in-distribution examples during training, are in synchronization across layers during inference, when inputs are drawn from in-distribution but that they are out of synchronization when inputs are drawn from out-of-distribution. Further, we speculated that ESSs could be used to measure the level of synchrony between layers and thus detect OOD examples. To test this line of hypotheses, we extracted the symbols from NINCO dataset [35], which contain OOD examples carefully curated against ImageNet.

In our analysis, we first used $CM(i,j)$ of layer 4 to predict the most probable class, which were incorrect, as the inputs were drawn from OOD, and tested ESSs (once again, expected symbol scores) in layers 1, 2 and 3. If the symbols are truly out of synchronization due to OOD examples, we anticipated low expected symbol scores in all three layers. We chose 18 classes from NINCO dataset, and ROIs of NINCO images were determined by GroundingDINO (an open-set segmentation model[23]) using their real class labels. We evaluated the expected symbol scores and compared them with the ESSs of normal images from the test set of Mixed_13.

Fig. 7A visualizes ESSs ($ESS_{1},ESS_{2},ESS_{3}$) from layers 1, 2, 3 of ResNet18 in a 3D space, in which $x,y,z$ denote $ESS_{1},ESS_{2},EES_{3}$, respectively. The orange circles and the blue triangles denote in-distribution and OOD examples respectively. As shown in the figure, ESSs were substantially different between in-distribution and OOD examples, and we quantified this observation by estimating AUROC for all 6 models. The estimated AUROC from ESSs of all 3 layers and their norms $||(ESS_{1},ESS_{2},ESS_{3})||$ suggest that ESSs are effectively separated between normal and OOD examples and can be used to detect OOD examples.

Since the discovery of DL models’ vulnerability to adversarial examples [36, 37], adversarial perturbation has emerged as one of the greatest threats to DL safety. Given that all inputs are progressively processed in modern DNNs, it seems natural to assume that adversarial perturbation may change internal representation on layer-by-layer basis. An earlier study [38] found that the difference in representations in the early layers was not noticeable, but in the late layers it grew significantly. We asked if ESSs can be used to detect the influence on adversarial perturbation. Specifically, we crafted adversarial examples from validation images of Mixed_13 using AutoAttack [39], which is known to be one of the most effective attacks. In the experiment, we used a ‘standard’ mode with $eps=0.03$ and $norm=Linf$. For individual inputs, we used labels of clean images to detect ROIs and estimated the corresponding symbols; that is, ROIs are identified with original labels, not the manipulated predictions.

For ROIs, we estimated the norm of ESSs from all 4 layers $||(ESS_{1},ESS_{2},ESS_{3},ESS_{4})||$ and calculated AUROC by comparing the norms of ESSs between normal and adversarial inputs. We note that the AUROC estimated from the norms were close to 0.5 (see the orange bars in Fig. 8), suggesting that the symbols in DNNs were not significantly different between normal and adversarial examples. Thus, we tested an alternative method. So far, ESSs have been computed using the most probable class predicted by the layer 4, which is the closest to the logit/output layer in the model. Specifically, we used layer 4’ s symbols ($S_{i}$) and correlation map $CM(i,j)$ to predict a class label $j$, but ESSs can also be computed using DNNs’ predictions directly. That is, we can set $j$ to be the model’s prediction.

If ESSs are computed with DNNs’ predictions, we can effectively evaluate the consistency between models’ decisions and internal symbols, which we assume could be useful in detecting adversarial perturbation. Notably, we analyzed unmodified ImageNet models, whose output nodes correspond to 1,000 classes, not 78 of Mixed_13. As we did not set the target class during adversarial attacks, the predictions were not necessarily confined to 78 classes of Mixed_13. Thus, to build ESSs using their predictions, we selected adversarial and normal symbols when predictions are one of 78 classes in Mixed_13 (see Table 4 for the number of ROIs included in this analysis). For ResNet50 trained on Oxford-IIT PET dataset (shown as ‘PET’ in the $x$-axis), we used all perturbed inputs. Decision-based ESSs are compared between normal and adversarial inputs (see the blue bars in Fig. 8). As shown in the figure, ‘decision-based ESSs’ are significantly different between normal and adversarial inputs in all 6 models, suggesting that ESSs with respect to DNNs’ (manipulated) predictions can be used to detect adversarial perturbation.

Further, inspired by the results suggesting that symbols can be used to predict labels of inputs (Fig. 5), we made predictions on adversarial inputs using symbols in L4 (e.g., $12^{th}$ embedding layer for ViT). Fig. 9, compares the accuracy of symbol-based predictions and the accuracy of the models’ original predictions on adversarial inputs. We note that even though adversarial inputs effectively lowered the accuracy of DL models’ predictions (see orange bars), their impact on the symbol-based predictions was minimal. This result indicates that symbols can be used to address DL models’ vulnerabilities to adversarial perturbations.

In this study, we extract symbols from ROIs. This is based on the assumption that classifiers rely on internal features of visual objects. We tested this assumption by creating cropped versions of ROIs (containing objects) and evaluated DL models’ accuracy on full images, ROIs and cropped ROIs. Specifically, the three cropped versions, ROI81, ROI64 and ROI49%, contain 81%, 64% and 49% of pixels of ROIs obtained from STCert. All cropped ROIs and the full ROI are aligned to have the same centers, and we tested DL models’ accuracy on the test set of Mixed_13. The accuracy on ROI is normalized to the accuracy on full images. We note that the drop in the accuracy on ROIs is limited even at ROI:49% (Table 5), suggesting that DL models can indeed learn to use internal features of visual objects, which supports our assumption.

We should underline that the purpose of our study is not to provide a comprehensive survey of methods that can extract internal symbols and potential utilities of internal symbols. Instead, we focused on searching for positive evidence of the existence of internal symbols associated with DNNs’ decisions and exploring their applications.

The two versions of STCert were discussed in the original study [22]. The naive STCert uses the bounding boxes returned by the segmentation models as ROIs. The context-aware STCert adds some background to the bounding boxes to create ROIs. As the context-aware STCert is more robust, we adopted the context-aware STCert in this study. The workflow of STCert can be summarized as follows:

1. 1.

   Detect bounding boxes.

2. 2.

   Create 5 candidates of bounding boxes by enlarging ROIs. The first candidate is ROI itself.

3. 3.

   Crop pixels within the candidates and forward them into DNNs. That is, we have the list of 5 second thought predictions.

4. 4.

   Examine if the original prediction is included in the list of second predictions.

5. 5.

   Return the bounding box as ROI, if the original prediction is in the list. If not, ignore the bounding box and return Null.

Our analysis raises the possibility that internal symbols can be used to augment DNNs’ decision-making process. It may even be possible for a second network to use symbols to correct responses in each layer locally during inference. When urgent and immediate decisions are demanded, DNNs’ decisions can be used, as they are, but for more reliable answers, a second network may use symbols to modify hidden layers’ responses. This may remind some readers of thalamo-cortical loops in the brain. Our analysis has been inspired by thalamo-cortical loops and indicates that thalamo-cortical loop-like error correction could be constructed using internal symbols to build more reliable DNNs.