Noise Modulation: Let Your Model Interpret Itself

Adversarial training was first proposed in [11], but the most popular and effective formulation is proposed by  [19]. The core idea of adversarial training is to add adversarial examples into training set, [19] even substitute the whole training set with adversarial examples.

Given a dataset $D=\{(x,y)|x\in\mathbb{R}^{N},y\in\mathbb{N}\}$ containing pairs of input $x$ and label $y$, a model $f$ parametrized with $\theta$ and a loss function $L$, adversarial training refers to solving the following minimax optimization problem. Following the formulation of [19],

$$\min_{\theta}\mathbb{E}_{(x,y)\sim\mathcal{D}}\left[\max_{\delta\in B(\varepsilon)}L(f(\theta;x+\delta),y)\right].$$

(1)

At each step, the inner maximization first searches for an optimal perturbation $\delta^{*}$ in a given attack space $B(\varepsilon)$, e.g. a $l_{\infty}$-norm ball $\{\delta|\delta\in\mathbb{R}^{N},\varepsilon\in\mathbb{R},||\delta||_{\infty}\leq\varepsilon\}$, such that the loss function is maximized; the outer minimization then optimizes the parameters $\theta$ to minimize the loss function just like standard training, but on perturbed examples $x+\delta^{*}$.

The inner optimization is commonly solved using Projected Gradient Descent (PGD) attack and its variants. It is an iterative method that at each epoch $t\in[T]$ ($T$ epochs in total), the perturbation is moved towards the direction of input-gradient for a certain step and projected back to the feasible space, i.e.

	$\displaystyle\delta^{t+1}=P_{B(\varepsilon)}(\delta^{t}+\alpha\text{sign}(\nabla_{x+\delta^{t}}L(f(\theta;x+\delta^{t}),y)))$		(2)
	$\displaystyle\delta^{0}\sim\mathcal{U}(-\varepsilon,-\varepsilon),$

where $P_{B(\varepsilon)}$ denotes projecting the perturbations back to the attack space, $\alpha\in\mathbb{R}$ denotes the step size at each iteration. After $T$ epochs, the adversarial example $x^{*}=x+\delta^{*}$ perturbed with the optimal perturbations $\delta^{*}$, is further fed for training. It only changes the training strategy, at inference, the model still gives its predictions on clean inputs.

The heavy computations of adversarial training mainly stems from the computation of input-gradients at each iteration of PGD. For adversarial training with $T$ epochs of PGD, there are $T$ more back-propagation iterations computed than a standard training. Other sideeffects of adversarial training, e.g. requiring larger model and more data comes from the adversarial nature of these perturbations. These perturbations are designed to move the original examples towards the decision boundary, making the resulted adversarial examples much more difficult to fit for model [30]. The increased difficulty requires a larger model to fit given the same amount of data [32], this further increases the computational overheads as the computation of back-propagation also grows with the scale of model.

In this section, we will draw an approximation of the adversarial perturbations in an efficient way. The first step, as initially proposed by [11], is to substitute the iterative multi-step PGD attack with its one step approximation, i.e.

$\displaystyle\delta^{*}\sim P_{B(\varepsilon)}(\delta^{0}+\alpha\text{sign}(\nabla_{x+\delta^{0}}L(f(\theta;x+\delta^{0}),y))).$

(3)

The perturbation now becomes a projected linear combination of a uniform noise and the sign of input-gradient. Our second step is to find a way to approximate the input-gradient. It has been observed that adversarial training makes the input-gradients similar to the corresponding inputs [6]. This can be verified by visualizing the input-gradients of adversarially trained model, as shown in the fourth column in Figure 1.

This similarity drives us to hypothesize that the sign of input-gradient can be approximated roughly by a multiplication of the corresponding input and some noise $\delta^{\prime}\in\mathbb{R}^{N}$ ¹¹1Apparently, the sign of input-gradient is not the same as corresponding input., i.e. $\text{sign}(\nabla_{x}L)\sim\delta^{\prime}\cdot x$. Take it back to Equation 3, the perturbation now becomes

$\displaystyle\delta^{*}\sim P_{B(\varepsilon)}(\delta+\alpha\cdot\delta^{\prime}\cdot x).$

(4)

This approximation of Equation 4 actually estimates the adversarial perturbation with a modified input. It is approximately the case after adversarial training, as reported in [2]. If we take this approximation back into adversarial training, the example $x^{*}$ fed for training becomes

	$\displaystyle x^{*}$	$\displaystyle=x+\delta^{*}$		(5)
		$\displaystyle\sim P_{x+B(\varepsilon)}(x\cdot(1+\alpha\cdot\delta^{\prime})+\delta).$

For our purpose, the perturbations are not intended to be adversarial, therefore, we further remove the components that are irrelated to input. The final approximation will be

$$x^{*}\sim x\cdot(1+\alpha\cdot\delta^{\prime}).$$

(6)

This formulation reminds us of the technique of amplitude modulation. In communication, amplitude modulation refers to having the amplitude of a carrier signal, generally of high-frequency, vary in proportion to that of the message signal, i.e. the signal that contains the information we wish to transmit, before transmission. At the receiver, the modulated signal is first demodulated with carrier of the same frequency, and then filtered through a low-passing filter to recover the original signal. From this perspective, adversarial training works as a dynamic modulator that tunes the model to filter out the informative components of frequencies by itself.

It motivates us to rethink this formulation from the perspective of modulation, and leads us to noise modulation we will propose in the next section.

For a digital signal like an image, we can also modulate it over a digital carrier of the same size and then demodulate the modulated signal using the same carrier. If we feed this demodulated signal for training, the model will have to learn to filter out informative components of frequencies related to its purpose. Based on this digital analogy of amplitude modulation, we propose modulational training as formulated below.

It seems quite different from Equation 6, but we will show that they are the same in essence. For the second power of a non-zero carrier $c^{2}=\{c_{n}^{2}\},c_{n}\in\mathbb{R}$, the $k$-th component of its Discrete Fourier Transform (DFT) $C=\{C_{k}\},C_{k}\in\mathbb{C}$ is

$$C_{k}=\sum_{n=0}^{N-1}c_{n}^{2}\text{e}^{-j\frac{2\pi kn}{N}}.$$

(8)

The constant component $C_{0}=\sum_{n=0}^{N-1}c_{n}^{2}$ is positive, as long as $c\neq 0$. Thus the second power of this non-zero carrier can be rewritten as a combination of a constant $\lambda=\frac{C_{0}}{N}$ and some noise $\delta^{\prime\prime}\in\mathbb{R}^{N}$,i.e.

$$c^{2}=\lambda+\delta^{\prime\prime}\implies x\cdot\frac{N\cdot c^{2}}{C_{0}}=x\cdot(1+\delta^{\prime\prime}\cdot\frac{N}{C_{0}}).$$

(9)

This formulation links back to our approximation of adversarial training (Equation 6). It indicates that it is possible to recover the original signal by filtering out the extra noises. Since the filtering of a certain frequency is a linear operation, in fact, a convolution, it is learnable by DNNs and even better for Convolutional Neural Networks(CNNs).

We still do not know which carrier can completely recover the interpretable input-gradients brought by adversarial training. But as we strives for efficiency, we propose to use noise as carrier and name this special case of modulational training as noise modulation.

The process of noise modulation is illustrated in Figure 2. For each input, a noise is sampled and then multiplied with the input twice. The demodulated input is rescaled to keep the constant components stasis and then fed for further training. The modulated input hides the discriminative features on the amplitude and these features are then brought back by demodulation. At inference, there is no processing for inputs, just like adversarial training.

As shown in Definition 2, noise modulation requires no extra back-propagation iterations than standard training. Since the noise is sampled independently from model, it is model-agnostic by design and can be further accelerated by prepossessing the dataset before training. Its computation grows with the dimension of data rather than the scale of model. Theoretically, it is much more efficient than adversarial training.

In terms of interpretable input-gradients, we mean that the input-gradient of model serves as an attribution map telling us how each dimension of an input influences the model’s decision. As proved by the existence of adversarial examples  [28], moving the input towards the direction of its corresponding input-gradients increases the loss function. It indicates that input-gradients contain information crucial to the decision of model while not necessarily making sense for human.

Qualitatively, the input-gradient is more interpretable if it is more visually aligned with input [9]. Given the fact that numerically the input-gradient is much more smaller than input, we will use the absolute cosine similarity over their signs, to define a Visual Interpretability of Input-gradient(VII) as follows.

A higher VII indicates that the model has a more interpretable input-gradient aligned with input over the test set under the certain loss. In the following sections, we will use cross-entropy as the loss function to calculate this metric. The input-gradients will be scaled to $[0,1]$ for visualization.

In this section, we will demonstrate the effectiveness of noise modulation over input-gradients. We will evaluate our method on MNIST [18] and CIFAR-10 [10]. We will use two simple models, a three layer Multi-Layer Perceptron (MLP) and a six layer Fully Convolutional Neural Networks(FCNN) on all of these datasets. Besides, we will also evaluate LeNet [17] on MNIST, VGG [25] and ResNet [12] on CIFAR-10.

Since the focus of this paper is NOT the state-of-art performance of these models, for fair comparison, we will use a basic setting without fine-tuning over each models. All of these models are trained from scratch using a mini-batch size of 64 for 50 epochs. The optimizer is Adam [15] with a fixed learning rate of 0.001.

All of the inputs are first scaled to $[0,1]$ and prepossessed with normalization, no extra data augmentation is used. For noise modulation, the ratio of constant component $\beta$ is set to $0.5$ and the noise used as carrier is a standard Gaussian noise. The model with highest validating accuracy during training is saved for comparison.

The experiment results are presented in Figure 3. We observe that standard models already produce basically interpretable input-gradients for human on MNIST, but on CIFAR-10, these standard models produce input-gradients with irregular noises that make no sense for human. Noise modulation significantly increases the interpretability of input-gradients with a few accuracy cost, except for MLP on MNIST. The MLP trained standardly on MNIST already has an input-gradient with most of its non-zero components centered around the digits, noise modulation focuses its gradients around some critical points on digits but leaves a more noisy background that reduces the value of VII metric.

We also observe that a larger model is more capable of producing interpretable input-gradients, but a simple FCNN is also able to produce a reasonable input-gradient for human. It is clear that noise modulation is effective on increasing the interpretability of input-gradients. More results can be found in Figure 6 at the end of this paper.

While the interpretability increases, noise modulation still brings a drop of accuracy. The design of constant component in Definition 2 is meant to deal with this problem.

On the same ResNet-18 we use in Section 3.2, we will change the ratio of constant component $\beta$ in carrier and check its influences on accuracy and interpretability. Theoretically, a larger $\beta$ indicates a noisier input-gradient and a higher accuracy as noise modulation becomes standard training when $\beta=1$,

The experiment results are presented in Figure 4. As expected, a larger $\beta$ yields a higher accuracy and a lower visual interpretability of input-gradients. From human perception, the input-gradients become roughly interpretable when $\beta$ is smaller than $0.8$. Reducing $\beta$ makes the interpretability increase until it plateaus and resonates once the constant component of carrier is smaller than $0.5$ and the noise dominates carrier.

Given these experiment results, we recommend to set the constant component ratio $\beta$ as $0.8$ for a basically interpretable input-gradient and $0.5$ to exploit the best interpretability over a compromisement with accuracy.

The choice of noises is another hyper-parameter in noise modulation. Intuitively, a Gaussian noise spanning the whole spectrum is the proper choice, as we intend to have the model learn to filter out the informative features by itself,

Besides standard Gaussian noise, we will test other five noises on the same ResNet-18 in Section 3.2 with every condition fixed but the type of noise. These noises include Uniform noise, Laplacian noise, Gamma noise, Exponential noise and Rayleigh noise. The Uniform noise is sampled from $[0,1]^{N}$. The distribution of Laplacian noise peaks at $0$, and the exponential decay of it is set as $1$. The shape of the distribution of Gamma noise is set to $1$, as well as its scale. For Exponential noise and Rayleigh noise, the scale is also set to $1$.

The experiment results are presented in Figure 5. All of these noises yield a much more interpretable input-gradients. As expected, Gaussian noise has the best overall performance. The accuracy loss brought by Rayleigh noise is the least, as well as the increasement of interpretability. Laplacian noise brings the highest increasement of interpretability but sacrifies most of the accuracies too.