Experimental Observations of the Topology of
Convolutional Neural Network Activations^††thanks: Version including technical appendix can be found on ArXiv.

CNNs are a type of deep neural network that respects the spatial information existing in the input data. They use shared weights to provide translation invariant measures of correlation across an input, which makes them ideal for image classification tasks, where objects requiring identification might be found anywhere in an image.

Mathematically, a trained neural network used for classification is best described as the composition of linear and non-linear tensor maps called layers, where a tensor is a multi-dimensional real-valued array. The input to a neural network is a tensor, and the output of the network is a probability vector indicating the likelihood the input belongs to each class. The intermediate outputs from each layer of the composition are called feature maps or activation tensors. Linear layers use tensor maps that respect element-wise addition and scalar multiplication, and can be either fully connected or convolutional.

Convolutional layers use cross correlation, also known as a sliding dot product, to map 3D tensors to 3D tensors. If the activation tensor from a convolutional layer has dimensions $c\times n\times m$, we say the tensor has $c$ channels and $nm$ spatial dimensions. Activation tensors may be sliced into spatial and channel activations, as shown in Figure 1, and then reshaped to obtain vector representations of their values.

One of the two topological tools that we use in our work is persistent homology (PH). At a high level, PH is a method for understanding the topological structure of a space that data are sampled from. We typically have access only to the sample, in the form of a point cloud, and use PH to infer large-scale structures of the unknown underlying space. Here, we provide a brief overview of PH and point readers to Edelsbrunner and Harer (2008); Ghrist (2008) for more details.

The theoretical basis for persistent homology lies in the concept of homology from algebraic topology. Given a topological object, e.g., a surface or the geometric realization of a simplicial complex (a collection of finite sets, $\Sigma$, such that if $\tau\subset\sigma$ and $\sigma\in\Sigma$ then $\tau\in\Sigma$), its homology is an algebraic representation of its cycles in all dimensions. In dimensions 0, 1, and 2, the cycles have simple interpretations as connected components, loops, and bubbles, respectively. Higher dimensional interpretations exist but are less intuitive.

Given a single point cloud, $S\subset\operatorname{\mathbb{R}}^{k}$, we can construct a family of associated simplicial complexes on which to compute homology. In this paper, we use the Vietoris-Rips (VR) complex given a scale parameter $\epsilon$, $VR(S,\epsilon)$. In short, $VR(S,\epsilon)$ is a simplicial complex where each collection of points in $S$ whose pairwise distances are all at most $\epsilon$ is a set in $VR(S,\epsilon)$. We show examples of two VR complexes (just the 1-skeleton, the pairwise edges) of the same point cloud at two scale parameters in Figure 2.

Finally, we can describe the motivation and concept of PH. A single point cloud technically is a simplicial complex, but it is not interesting homologically. Whereas constructing a VR complex at a single scale parameter does provide an interesting topological object, it does not capture the multiscale phenomena of the data. PH is a method that considers all VR scale parameters together to identify at which $\epsilon$ a cycle is first seen (is “born”) and at which $\epsilon^{\prime}$ the cycle is fully triangulated (“dies”). This set of birth and death values for a sequence of simplicial complexes of a given point cloud provides a topological fingerprint for a point cloud often summarized in a persistence diagram (PD) as a set of $(b,d)$ coordinates. Figure 2 also shows the point cloud’s PD from the full sequence of $\epsilon$ thresholds.

PDs form a metric space under a variety of distance metrics. In this paper, we will use sliced Wasserstein (SW) distance introduced by Carrière, Cuturi, and Oudot (2017). Given two PDs, the SW distance is computed by integrating the Wasserstein distances for all projections of the PD onto lines through the origin at different angles.

The mapper algorithm was first introduced by Singh, Memoli, and Carlsson (2007). It is rooted in the idea of “partial clustering of the data guided by a set of functions defined on the data” (2007). On a high level, the mapper graph captures the global structure of the data.

Let $S\subset\operatorname{\mathbb{R}}^{k}$ be a high-dimensional point cloud. A cover of $S$ is a set of open sets in $\operatorname{\mathbb{R}}^{k}$, $\operatorname{\mathcal{U}}=\{U_{i}\}$ such that $S\subset\cup_{i}U_{i}$. In the classic mapper construction, obtaining a cover of $S$ is guided by a set of scalar functions defined on $S$, referred to as filter functions. For simplicity, we describe the mapper construction using a single filter function $f:S\to\operatorname{\mathbb{R}}$. Given a cover $\operatorname{\mathcal{V}}=\{V_{\ell}\}$ of $f(S)\subset\operatorname{\mathbb{R}}$ where $f(S)\subseteq\cup_{\ell}V_{\ell}$, we can obtain a cover $\operatorname{\mathcal{U}}$ of $S$ by considering as cover elements the clusters (for a choice of clustering algorithm) induced by $f^{-1}(V_{\ell})$ for each $V_{\ell}$.

Then, the 1D nerve of any cover $\operatorname{\mathcal{U}}$ is a graph and is denoted as $\operatorname{\mathcal{N}}_{1}(\operatorname{\mathcal{U}})$. Each node $i$ in $\operatorname{\mathcal{N}}_{1}(\operatorname{\mathcal{U}})$ represents a cover element $U_{i}$, and there is an edge between nodes $i$ and $j$ if $U_{i}\cap U_{j}$ is non-empty. If $\operatorname{\mathcal{U}}$ is constructed as above, from a clustering of preimages of a filter function $f$, then its 1D nerve, denoted as $\operatorname{\mathcal{M}}=\operatorname{\mathcal{M}}(S,f):=\operatorname{\mathcal{N}}_{1}(\operatorname{\mathcal{U}})$, is the mapper graph of $(S,f)$.

Consider the point cloud in Figure 3 as an example containing two nested circles. It is equipped with a height function $f:S\to\operatorname{\mathbb{R}}$. A cover $\operatorname{\mathcal{V}}=\{V_{1},\cdots,V_{5}\}$ of $f(S)$ is formed by five intervals (see Figure 3 middle). For each $\ell$ ($1\leq\ell\leq 5$), $f^{-1}(V_{\ell})$ induces a number of clusters that are subsets of $S$. Such clusters form the elements of a cover $\operatorname{\mathcal{U}}$ of $S$. As shown in Figure 3 (left), the cover elements of $\operatorname{\mathcal{U}}$ are contained within the 12 rectangles on the plane. The mapper graph of $S$ is shown in Figure 3c. For instance, cover $f^{-1}(V_{1})$ induces a single cover element $U_{1}$ of $S$, and it becomes node 1 in the mapper graph of $S$. $f^{-1}(V_{2})$ induces 3 cover elements $U_{2}$, $U_{3}$ and $U_{4}$, which become nodes 2, 3 and 4. Since $U_{1}\cap U_{2}\neq\emptyset$, an edge exists between node 1 and node 2. The two circular structures in Figure 3 (left) are captured by the mapper graph in Figure 3 (right).

The value of TDA to organize, understand, and interpret various aspects of ML and DL models has been recognized in several current research directions. Much of this research has focused on model parameters, structure, and weights. Guss and Salakhutdinov (2018) examine model architecture selection by defining the “topological capacity” of networks, or the ability for the network to capture the true topological complexity of the data. They explore the learnability of model architectures in the face of increasing topological complexity of data. Gabrielsson and Carlsson (2019) build the mapper graph of a point cloud of learned weights from convolutional layers within a simple CNN and find that the weights of different CNN model architectures trained on the same data set have topological similarities. “Neural persistence”, developed by Rieck et al. (2019), is a topological measure of complexity of a fully connected deep neural network that depends on learned weights and network connectivity. They find networks that use best practices such as dropout and batch normalization have statistically higher neural persistence, and define a stopping criterion to speedup the training of such a network.

Other studies, like that of Wheeler, Bouza, and Bubenik (2021) use TDA to study activation tensors of simple multi-layer perceptron networks to discover how the topological complexity, as measured by a property of persistence landscapes, changes through the layers. Gebhart, Schrater, and Hylton (2019); Lacombe, Ike, and Umeda (2021) investigate the topology of neural networks via “activation graphs,” which model the natural graphical structure of the network. Finally, most closely related to our work is that of Rathore et al. (2021), which describes TopoAct, a visual platform to explore the organizational principle behind neuron activations. TopoAct displays the mapper graph of activation vectors for a single layer at a time in a CNN to show how the model organizes its knowledge via the branching structures. The authors consider a point cloud formed by randomly sampling a single spatial activation in a given layer for each image in a corpus. We extend this work by using a larger and more data-driven sample of spatial activations to build our mapper graphs, quantifying the intuition of “pure” and “mixed” mapper nodes, considering the effect of noisy input on the resulting graph, and showing how our results generalize to multiple common model architectures.

The following two sections outline our experiments using PH and mapper graphs to study the standard benchmark dataset CIFAR-10 (Krizhevsky and Hinton 2009) on a ResNet-18 architecture (He et al. 2016). We perform standard preprocessing to normalize the images by the mean and variance from the full training set. Code for the models and additional details regarding the dataset, as well as the parameters and computing infrastructure specific to each set of experiments, are provided in the arXiv technical appendix.

In Figure 5, we observe a grid-like pattern in the SW distances between layers of ResNet-18 similar to the results found in Kornblith et al. (2019), which the authors attribute to the residual architecture. This observation supports our belief that meaningful qualities of the model and its architecture can be uncovered by studying the topology of the activation space with PH.

Metrics such as canonical correlation analysis (CCA) (Morcos, Raghu, and Bengio 2018; Raghu et al. 2017), centered kernel alignment (CKA) (Kornblith et al. 2019), and orthogonal Procrustes distance (Ding, Denain, and Steinhardt 2021) provide dissimilarity measures that can be used to compare layers of neural networks. Recent work has demonstrated the value of topological approaches to representation similarity such as Representation Topology Divergence (Barannikov et al. 2022). These methods operate on an $N\times cnm$ matrix representation of a convolutional layer, where the $c\times n\times m$ activation tensors produced by each of the $N$ inputs from the dataset are normalized and unfolded into vectors in $\operatorname{\mathbb{R}}^{cnm}$. Here we note this as a key difference from our $N\times c$ point cloud representation obtained through top $l^{2}$-norm sampling but leave a more thorough comparison to future work.

We apply the intuitive specificity and sensitivity tests outlined by Ding, Denain, and Steinhardt (2021) to probe the utility of the SW distance between layers as a representation similarity metric for neural networks. In comparison to the intuitive test results shown for CCA, CKA, and orthogonal Procrustes distance from Ding, Denain, and Steinhardt (2021), this metric exhibits some non-standard behavior, for which we provide some speculative explanations but further work is needed to fully understand such a metric.

In Figure 8, we compare the mapper graphs generated from a point cloud of random activations ($50\text{k}\times c$) against those generated from the full activations ($50\text{k}\cdot nm\times c$) across different convolutional layers, where $c$ is the number of dimensions of each activation, and $nm$ is the total number of spatial activation vectors per image. The glyph for each node of the mapper graph is a pie chart showing the composition of class labels in that node. It can be seen that at layer 16, the mapper graphs of the random and full activations clearly capture the separation among class labels; there is a central region in the graph where nodes with mixed labels (with lower $l^{2}$-norm) separate out into branches with single labels (with higher $l^{2}$-norm). As we move toward earlier layers, the ability of the mapper graphs to show class separation gradually deteriorates. In addition, both random and full activations show similar bifurcation patterns, indicating robustness with respect to the sampled activations.

Next, we study whether branching structures emerge at earlier layers if we use top foreground or background activations. Figure 9 shows the evolution of mapper graphs using the foreground and background activations across layers. We observe that the mapper graph of foreground activations at layer 15 already shows notable class bifurcations. Such early separations are less obvious for random and full activations. The mapper graphs of background activations also show clear class separations at layer 15 and 16, indicating that background pixels likely play an important role in class separation as well. Mapper graphs for the top 5 foreground and background activations are provided, along with similar observations in the technical appendix.

To explore the stability of mapper graphs to noise in the input data, we injected pixel-wise Gaussian noise to all 50k images with different standard deviations ($\sigma$). Examples of how the images change as the standard deviation increases are shown in Figure 10, and the corresponding mapper graphs at layer 16 are shown in Figure 11. It can be seen that the mapper graphs are stable for small perturbations ($\sigma=0.1$). As $\sigma$ increases, mapper graphs illustrate that the model’s ability to differentiate different classes decreases. This observation aligns with the intuition that increasing the noise level will decrease prediction accuracy.

For an image classification task, each point (i.e., a spatial activation) $x\in S$ is assigned a class label (inherited from the class label of its corresponding input image). We introduce three quantitative measures to quantify how well a mapper graph of the activation space separates the points from different classes.

In order to show that our mapper graph observations are not dependent on the ResNet-18 architecture or CIFAR-10 data set we also perform these experiments using a different model-data pair. To compare with the prior experiments which use the lower resolution CIFAR-10 data set, the experiments in this section use a subset of 10 classes from the ImageNet dataset (Deng et al. 2009), as shown in the legend of Figure 13. There are 1300 images per class, resulting in a set of $N=13\text{k}$ images. The images have varying resolutions with an average resolution of $469\times 378$. The data is pre-processed by first resizing each image to 256 pixels and center cropping to a patch of size 2$24\times 224$, followed by a normalization with mean and variance of the original ImageNet training set images. For foreground extraction, we apply a different strategy than previously used since cv2.grabCut does not work as well with the ImageNet dataset due to the large amount of high frequency details in the image backgrounds. Instead we use a pre-trained DeepLabV3 semantic segmentation model (Chen et al. 2017) to obtain the foreground mask which is then applied to the images to get the foreground pixels.

The models that we use for the generalization experiments include ResNet-18, Inception_v1 (Szegedy et al. 2015), Inception_v3 (Szegedy et al. 2016) and AlexNet (Krizhevsky, Sutskever, and Hinton 2012). The number of parameters of each model is 11.6M, 6.6M, 27.2M and 61.1M respectively.

Figure 13 shows the resulting mapper graphs generated from the last layer of each model. Through these experiments, we demonstrate that the structures and insights we observe on ResNet-18 applied to CIFAR-10 are applicable to a wide range of other image recognition models as well.