IC Networks: Remodeling the Basic Unit for Convolutional Neural Networks

The MP neuron McCulloch and Pitts (1990) is the most commonly used neuron model, which can be formulated as $\mathrm{f}(\sum_{i=1}^{n}\ w_{i}x_{i}+b)$, where a linear transformation and a non-linear activation function $\mathrm{f}(\cdot)$ are applied to the $n$-D input successively. $w_{i}$ and $b$ denote the learnable weight and the bias, respectively. To facilitate the non-linear representation of neural models, we propose a new neuron model by replacing the linear transformation with a non-linear one:

	$\displaystyle\centering y$	$\displaystyle=\mathrm{f}\left(\sum_{i=1}^{n}\ w_{i}x_{i}+\sigma\left(\sum_{i=1}^{n}\ (w_{i}-1)x_{i}+b_{1}\right)+b_{2}\right)$		(1)
		$\displaystyle=\mathrm{f}\left(\sum_{i=1}^{n}\ w_{i}x_{i}+\sigma\left(\sum_{i=1}^{n}\ w_{i}x_{i}-x_{sum}+b_{1}\right)+b_{2}\right),$

where $\mathrm{f}(\cdot)$ denotes a non-linear activation function and $\sigma(\cdot)$ is a rectified linear unit (ReLU) Nair and Hinton (2010). $b_{1}$ and $b_{2}$ are two independent biases used to adjust the center of the model distribution. $x_{sum}=\sum_{i=1}^{n}\ x_{i}$ denotes the summation of all the input features. We term the structure defined in eq. (1) the inter-layer collision (IC) neuron, since this idea is inspired by the physical elastic collision model where the speeds of two objects are $\frac{2m_{1}}{m_{1}+m_{2}}v$ and $(\frac{2m_{1}}{m_{1}+m_{2}}-1)v$ after collision (Details in Appendix C). We treat $\frac{2m_{1}}{m_{1}+m_{2}}$ as learnable weight $w_{i}$ and introduce some mathematical adjustments.

Through introducing the $\sigma(\cdot)$ operation, the neuron model can definitely increase the number of non-linear patterns. We use the term $H=\sum_{i=1}^{n}\ (w_{i}-1)x_{i}$ to represent a hyperplane ($H=0$) in an $n$-dimensional Euclidean space, which divides eq. (1) as follows:

$$y=\begin{cases}\mathrm{f}\left(2\sum_{i=1}^{n}\ w_{i}x_{i}-\sum_{i=1}^{n}\ x_{i}\right)&\text{if }H\geq 0\\ \mathrm{f}\left(\sum_{i=1}^{n}\ w_{i}x_{i}\right)&\text{if }H<0\end{cases}.$$

(2)

Here we omit the bias term. Intuitively, this IC neuron has a stronger representation ability than the MP neuron, since it can produce two different linear representations before the activation operation $\mathrm{f}(\cdot)$. To facilitate understanding, we use 2-D data as input and ReLU as the activation function to show how the two kinds of neurons generate non-linear boundaries. Figure 2(a)(b) shows that the single MP neuron and the IC neuron can divide the $2$-D Euclidean space into multiple subspaces, each of which can represent a fixed linear transformation. We observe that a single IC neuron can divide one more subspace to represent a different linear transformation. Furthermore, we map the XOR problem which is a typical linear inseparable problem onto a $2$-D plane to explain the difference between the two kinds of neurons. For the ReLU MP neuron, it is clear that at least two neurons are required to solve the XOR problem. However, the non-linear boundary of the IC neuron is a broken line, providing a single neuron possibility to solve the XOR problem. Figure 2(c) gives a solution with a single IC neuron, dividing the whole plane into three spaces where $(0,0),(1,1)$ are represented by zero and $(1,0),(0,1)$ are in two spaces with similar representation.

Although neuron using the hyperplane $H=0$ can increase the number of non-linear patterns, the calculation of the hyperplane is limited by weight $w_{i}$, making it inflexible to divide different subspaces. Hence, the subspaces divided by the hyperplane are usually not optimal thus the weights easily converge to a local minimum. To add more representation flexibility to the hyperplane $H=0$, we improve eq. (1) by introducing an adjustment weight $w^{\prime}$:

	$\displaystyle\centering y$	$\displaystyle=\mathrm{f}\left(\sum_{i=1}^{n}\ w_{i}x_{i}+\sigma\left(\sum_{i=1}^{n}\ w_{i}x_{i}-w^{\prime}x_{sum}+b_{1}\right)+b_{2}\right)$		(3)
		$\displaystyle=\mathrm{f}\left(\mathbf{w}^{T}\mathbf{x}+\sigma\left(\mathbf{w}^{T}\mathbf{x}-w^{\prime}\times\mathbf{1}^{T}\mathbf{x}+b_{1}\right)+b_{2}\right),$

where $\mathbf{w}$ and $\mathbf{x}$ represent weight vector and input vector, respectively. $\mathbf{1}$ is an all-one vector. $w^{\prime}$ can be regarded as the intrinsic weight of one neuron, which is different from the weight $w_{i}$ connecting two neurons. Then there are two independent parameters in the calculation of the hyperplane $H=0$: $w^{\prime}$ is used to change the direction of the hyperplane, and the bias $b_{1}$ is used to shift the hyperplane in the whole space. We give a theoretical analysis of the adjustable range of $w^{\prime}$:

Theorem 1 implies that $\sum_{i=1}^{n}\ (w_{i}-w^{\prime})x_{i}=0$ can almost represent all the hyperplanes parallel to the cross product of $\mathbf{w}$ and $\mathbf{1}$, providing flexible strategies for dividing spaces. Besides, the IC neuron keeps a significant advantage over the MP neuron in that the $\sum_{i=1}^{n}\ w_{i}x_{i}$ term can flexibly represent any linear combinations in the input space. We provide the theorem as below:

The proof of Theorem 1 and 2 are provided in the Appendix A. In summary, by adjusting the relationship between $\mathbf{w}$ and $w^{\prime}$, the IC neuron can retain the representation ability of the MP neuron and flexibly segment linear representation spaces for some complex distributions.

The convolutional kernel, a filter used to capture the latent features in input signals, can be regarded as a combination of the MP model and a sliding window. To simplify the notation, we omit the activation operator and bias term. The output feature $\mathbf{u}_{i}\in\mathbb{R}^{H\times W}$ of the standard convolutional layer is given by:

$\displaystyle\mathbf{u}_{i}=\mathbf{w}_{i}\otimes\mathbf{X},$

(4)

where $\mathbf{X}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times C^{\prime}}$ is the input feature map, $\mathbf{w}_{i}\in\mathbb{R}^{k\times k\times C^{\prime}}$ is a filter kernel that belongs to $\mathbf{W}_{i}=[\mathbf{w}_{1},\dots,\mathbf{w}_{C}]$ and $\otimes$ is used to denote the convolution operator. To apply the IC neuron model to eq. (4), we replace the kernel $\mathbf{w}_{i}$ with an IC kernel $[\mathbf{w}_{i},w^{\prime}_{i}]$,

	$\displaystyle\mathbf{u}_{i}$	$\displaystyle=[\mathbf{w}_{i},w^{\prime}_{i}]\otimes\mathbf{X}$		(5)
		$\displaystyle=\mathbf{w}_{i}\otimes X+\sigma(\mathbf{w}_{i}\otimes\mathbf{X}-w^{\prime}_{i}\times(\mathbf{1}\otimes\mathbf{X})),$

where $\mathbf{1}$ is an all-one tensor with the same size as $\mathbf{w}_{i}$. The input feature map may contain hundreds of channels and $\mathbf{1}\otimes\mathbf{X}$ term will mix features in all the channels with the same proportion. Therefore, we distinguish different features by a grouped convolutional trick:

$\displaystyle\mathbf{u}_{i}=\mathbf{w}_{i}\otimes\mathbf{X}+\sigma(\mathbf{w}_{i}\otimes\mathbf{X}-(\mathbf{1}\hat{\otimes}\mathbf{X})\cdot\mathbf{w}^{\prime}_{i}),$

(6)

where $\hat{\otimes}$ denotes the depthwise convolution Chollet (2017) which separates $\mathbf{1}$ and $\mathbf{X}$ into $C^{\prime}$ independent channels and performs channel-wise convolution. The adjustment weight $\mathbf{w}^{\prime}_{i}$ becomes a vector with size $C^{\prime}$, recalibrating the features of $\mathbf{1}\hat{\otimes}\mathbf{X}$ by channel-wise multiplication ($\cdot$). Note that all the convolution operators ($\otimes$ and $\hat{\otimes}$) in eq. (6) share the same stride and padding. The structure in eq. (6) which we term the IC layer has two main advantages compared to the traditional convolutional layer:

• •

  According to Section 3.1, the single kernel $[\mathbf{w}_{i},\mathbf{w}^{\prime}_{i}]$ can represent more linear patterns before activation, which enhancing the representation of convolutional layer.

• •

  The information $\mathbf{1}\hat{\otimes}\mathbf{X}$ contains some low-level features from the previous layer. It helps the filters to learn high-level features faster since it provides an approximate distribution of the pixels in the input feature maps.

The IC layer can be easily integrated into some existing models. Consider the ResNets He et al. (2016) which are commonly used networks as examples. The basic block used by ResNet-18 and ResNet-34 has two $3\times 3$ convolutional layers. The bottleneck block used by deeper ResNets has three convolutional layers ($1\times 1,3\times 3,1\times 1$). Note that the information $\mathbf{1}\hat{\otimes}\mathbf{X}$ equals $\mathbf{X}$ when kernel size is $1\times 1$, it will not provide extra features for the filter. Therefore, we prefer to replace all the $3\times 3$ layers in building blocks to build IC-ResNets. The combination with other popular models is introduced in Section 4, where we show the universality and superiority of the IC layer.

In order to further understand why the IC layer can capture more fine-grained features, we compare the Grad-CAM visualizations Selvaraju et al. (2017) of the IC models with their basic models. As shown in Figure 3, the IC networks tend to focus on more relevant regions with more object details. More importantly, the features of IC networks and basic networks have some similarities. We think that although the features captured by IC networks have finer texture information, their focus on the image is similar to features captured by basic networks.

Motivated by the observed similarity, we propose a method to guide the learning of IC networks by using the knowledge of pre-trained basic networks. First, we have a basic network B with a pre-trained set of parameters $\mathbf{\theta}^{*}$ and our goal is to train a corresponding IC network IC-B with better performance. According to Theorem 2 and the argument that a network has many configurations of parameters with the same performance Kirkpatrick et al. (2017), we assume that IC-B can have a configuration close to $\mathbf{\theta}^{*}$. Based on our hypothesis, we load the set $\mathbf{\theta}_{*}$ for IC-B and add a scaling factor $\alpha$ to control the influence of $\sigma(\cdot)$. The IC layer in IC-B can be initialized as:

$\displaystyle\mathbf{u}_{i}=\mathbf{w}_{i}^{*}\otimes\mathbf{X}+\alpha\times\sigma(\mathbf{w}_{i}^{*}\otimes\mathbf{X}-(\mathbf{1}\hat{\otimes}\mathbf{X})\cdot\mathbf{w}^{\prime}_{i}),$

(7)

where $\mathbf{w}_{i}^{*}$ is a weight loaded from $\mathbf{\theta}_{*}$ and we use random initialization for $\mathbf{w}^{\prime}_{i}$. Since the $\mathbf{w}_{i}^{*}\otimes\mathbf{X}$ term can capture the features independently, $\alpha$ is set to zero at the beginning to let IC-B have the same performance as B. After initialization, we fine-tune all the parameters. The $\sigma(\cdot)$ term benefits from the pre-trained information, making the adjustment weight converges quickly. We use the scaling factor $\alpha$ to gradually amplify the influence of $\sigma(\cdot)$ in the training process. There are two strategies to adjust $\alpha$. The first one is to gradually increase its value manually, but it need to provide accurate hyperparameters. The second is to treat $\alpha$ as a parameter trained with other parameters. Our experience shows that the best value of $\alpha$ is usually in $[0.01,0.04]$.

To further use the knowledge of B, we refer to the way in KD. We use the basic network B as a teacher model and the IC-B as a student model. The soft targets predicted by a well-optimized teacher model can provide extra information, compared to the ground truth. To obtain the soft targets of B, temperature scaling Hinton et al. (2015) is used to soften the peaky softmax distribution:

$\displaystyle p^{i}(x;\tau)=\frac{e^{s_{i}(x)/\tau}}{\sum_{k}e^{s_{k}(x)/\tau}},$

(8)

where $x$ is the data sample, $i$ is the category index, $s_{i}(x)$ is the score logit that $x$ obtains on category $i$, and $\tau$ is the temperature. The knowledge distillation loss $L_{kd}$ measured by KL-divergence as:

$\displaystyle L_{kd}=-\tau^{2}\sum_{x\in\mathcal{D}_{x}}\sum_{i=1}^{C}p_{t}^{i}(x;\tau)\log(p_{s}^{i}(x;\tau)),$

(9)

where $t$ and $s$ denote the teacher and student models. $C$ is the total number of classes, $\mathcal{D}_{x}$ indicates the dataset. The complete loss function $L$ of training IC-B is a combination of the standard cross-entropy loss $L_{ce}$ and knowledge distillation loss $L_{kd}$:

$\displaystyle L=(1-\lambda)L_{ce}+\lambda\max(L_{ce}-e,0),$

(10)

where $\lambda$ is a balancing weight. $e$ is a constant used to increase tolerance the for the gap between the soft targets of B and IC-B.

Different from the traditional KD method, our method does not extract the knowledge from a deeper network with better performance. Therefore, we do not need the output distributions of the teacher and student models to be exactly equal. The loss term $\max(L_{ce}-e,0)$ allows deviation between $p_{s}^{i}$ and $p_{t}^{i}$. Besides, during training, we gradually reduce the value of $\lambda$ to make the IC-B reduce the dependence on the teacher network. Eq. (10) provides a strategy to introduce some extra information for training IC networks. Combined with loading pre-trained parameters and fine-tuning, the IC networks achieve high performance after a few learning rounds. We term this learning method the weak logit distillation (WLD) since it weakens the impact of soft targets to reduce the dependence on the teacher networks.

For the standard convolutional layer with $k\times k$ receptive fields, the transformation

$\displaystyle\mathbf{X}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times C^{\prime}}\xrightarrow{conv}\mathbf{U}\in\mathbb{R}^{H\times W\times C}$

(11)

needs $k\times k\times C^{\prime}\times C$ parameters. In the IC layer, we calculate the sum of each channel without additional parameters. The weight $\mathbf{W^{\prime}}=[\mathbf{w^{\prime}}_{1},\mathbf{w^{\prime}}_{2},\cdots,\mathbf{w^{\prime}}_{C}]$ adds $1\times 1\times C^{\prime}\times C$ parameters. Therefore, the number of parameters added by the IC layer is only $\frac{1}{k\times k}$ of the original layer.

The IC layer adds a depthwise convolution and a learnable weight $\mathbf{W}^{\prime}$ which can be regarded as a $1\times 1$ convolution. The depthwise convolution is used to calculate the element-wise sum of the input by an all-one kernel. The increased computational complexity is the same as adding a convolutional filter because we only need to do this operation once. Therefore, the increased complexity is $\frac{1}{C}$ of the original layer. The weight $\mathbf{W^{\prime}}$ is a $1\times 1$ convolution which uses less than $\frac{1}{k\times k}$ computation of the $k\times k$ convolution Sifre and Mallat (2014). Therefore, the approximate extra computation is $\frac{1}{C}+\frac{1}{k\times k}$ of the original layer.

We use the ILSVRC 2012 classification dataset Russakovsky et al. (2015) which consists of more than 1 million color images in 1000 classes divided into 1.28M training images and 50K validation images. We use three versions of ResNets (ResNet-18, ResNet-34, ResNet-50) to build the corresponding IC networks. For a fair comparison, all our experiments on ImageNet are conducted with the same environment setting. The optimizer uses the stochastic gradient descent (SGD) LeCun et al. (1989) method with a weight decay of $10^{-4}$ and a momentum of $0.9$. The training process is set to $120$ epochs with a batch size of $256$. The learning rate is initially set to $0.1$ and will be reduced 10 times every $30$ epochs. Besides, all experiments are implemented with the Pytorch Paszke et al. (2019) framework on a server with NVIDIA TITAN Xp GPUs.

To compare with previous research He et al. (2016); Wang et al. (2019), we apply both the single-crop and 10-crop for testing. We build the IC-ResNet-18, IC-ResNet-34 and IC-ResNet-50 by replacing the whole $3\times 3$ convolutional layers in blocks with $3\times 3$ IC layers. The validation error and FLOPs are reported in Table 1. We observe that the IC-ResNet-18 and IC-ResNet-34 can obviously reduce the 10-crop top-1 error by $1.19\%,1.03\%$ and the top-5 error by $0.86\%,0.60\%$ with a small increase in the calculation ($10.44\%,10.60\%$), validating the effectiveness of the IC layer. The IC-ResNet-50 retains $1\times 1$ convolutional layers in bottleneck blocks. The 10-crop top-1 error is $21.90\%$ and the top-5 error is $6.08\%$, exceeding ResNet-50 by $0.95\%$ and $0.63\%$, respectively. Moreover, the extra FLOPs of the IC-ResNet-50 is only $5.10\%$ of the ResNet-50. For the deeper ResNets, we believe that the deeper IC-ResNets can get similar results, since they all use the same block as ResNet-50.

We further investigate the universality of the IC layer by integrating it into some other modern architectures. These experiments are conducted on the CIFAR10 dataset, which consists of 60K $32\times 32$ colour images in 10 classes divided into 50K training images and 10K testing images. Each model is trained in $200$ epochs with a batch size of $128$. The learning rate is initialized to $0.1$, which will be reduced $10$ times at the $60$th epoch and the $120$th epoch. The optimizer settings are the same as in the ImageNet experiment.

We integrate the IC layers into VGGNets (VGG-16 and VGG-19 versions), MobileNet Howard et al. (2017), SENets (SE-ResNet-18 and SE-ResNet-50 versions) and ResNeXt Xie et al. (2017) (2x64d version). Specially, the adjacent convolution layers (a depthwise convolution layer and a pointwise convolutional layer) in MobileNet are treated as one convolutional layer when integrating with the IC layer because there is a close relationship between adjacent layers. The results are listed in Table 4, we observe that the IC layers improve representation of all the basic models. This set of experiments show the universality of the IC layers. Besides, we observe that the IC networks have faster convergence speed than basic models in both the ImageNet and CIFAR-10 experiments. The training curves are shown in Appendix B.

We further assess the generalization of IC networks on the task of object detection using the PASCAL VOC 2007+2012 detection benchmark Everingham et al. (2010), This dataset consists of about 5K train/val images and 5K test images over 20 object categories. We use the IC-ResNet-50 as the backbone networks to capture the features. Weights are initialized by the parameters of the IC-ResNet-50 trained by WLD on the ImageNet dataset. The detection frameworks that we use are Faster R-CNN Ren et al. (2015) and Retinanet Lin et al. (2017). We use the same configuration for both the IC-ResNet-50 and the ResNet-50, which is described in Chen et al. (2019). We evaluate detection mean Average Precision (mAP) which is the actual metric for object detection. As shown in Table 5, IC-ResNet-50 outperforms ResNet-50 by $1.0\%$ and $0.9\%$ on the Faster R-CNN and Retinanet frameworks, respectively. The ResNet-50 result we report is the same as previous work. Our experiments demonstrate that the IC networks can be easily integrated into the object detection and achieve better performance with negligible additional cost. We believe that IC networks can show their superiority across a broad range of vision tasks and datasets.