Isolation and Induction: Training Robust Deep Neural Networks against Model Stealing Attacks

Model stealing attack, also referred to as model extraction, aims at inferring hyper-parameters (Wang and Gong, 2018; Oh et al., 2019), extracting model parameters (Lowd and Meek, 2005; Tramèr et al., 2016; Milli et al., 2019), or copying functionalities of a certain machine learning model. Our work focuses on stealing the classification accuracy of the model, which is the most prevailing and universal stealing attack in deep learning. Tramèr et al. (Tramèr et al., 2016) proposed the concept of model stealing that attackers could “steal” the property of a machine learning model by queries without prior knowledge of the victim model. Generally speaking, there are many properties that can be stolen, e.g., model parameters, training data, or functionality. Papernot et al. (Papernot et al., 2017) proposed a partial data approach named Jacobian-Based Dataset Augmentation (JBDA), which generates synthetic data by adding small perturbations on a small set of in-distribution samples. Orekondy et al. (Orekondy et al., 2019) proposed KnockoffNets, employing samples from a surrogate dataset as query inputs of the victim model. The stealing performance of partial data and surrogate data approaches would degrade when the available data are different from the original training set. In recent years, some data-free stealing methods have been proposed. Kariyappa et al. (Kariyappa et al., 2021a) and Truong et al. (Truong et al., 2021) are motivated by the framework of data-free knowledge distillation (Hinton et al., 2015; Micaelli and Storkey, 2019) and proposed data-free model stealing methods, where they use zeroth-order gradient estimation to calculate the victim gradient in black-box settings. Moreover, Sanyal et al. (Sanyal et al., 2022) proposed a model stealing attack in the hard-label setting, They utilize some unrelated proxy data to get a pre-trained data generator, while the stealing process is data-free.

Currently, most model stealing defenses tend to add perturbations to the model outputs, thus disturbing the optimization of the adversary. Lee et al. (Lee et al., 2019) proposed an accuracy-preserving defense against model stealing attacks by adding deceptive perturbations to the model outputs while preserving its top-1 label, while it yields to the hard-label stealing. Other defenses like Maximizing Angular Deviation (MAD) (Orekondy et al., 2020) perturbs the model outputs with controllable intensity, defending against model stealing attacks at the expense of benign accuracy. Another approach of defense takes advantage of the data limitation of adversaries, making the victim model produce dissimilar output between in-distribution inputs and out-of-distribution inputs. Kariyappa et al. (Kariyappa and Qureshi, 2020) proposed Adaptive Misinformation (AM) defense that detects the OOD inputs and misleads adversaries with modified outputs. Kariyappa et al. (Kariyappa et al., 2021b) then proposed Ensemble of Diverse Models (EDM) defense, which introduces randomness into the model output by using an ensemble of diverse models. Models in the ensemble are trained to perform diversely for OOD inputs, making the functionality of the model hard to be stolen. In addition, there are other types of countermeasures towards model stealing, such as digital watermarking (Jia et al., 2021; Li et al., 2022b; Charette et al., 2022), which inject an extractable watermark into the victim model and can distinguish whether a model is from stealing.

Existing defensive approaches take advantage of some limitations of model stealing attacks to mitigate the knowledge leakage, while they are suffering from high computational costs and unfavorable trade-offs. In this paper, we are devoted to defending against stealing attacks by incorporating the countermeasure with the victim’s parameters, inherently enhancing the robustness against model stealing attacks.

In this paper, we mainly discuss the functionality stealing towards DNNs on image classification tasks. Specifically, an adversary aims at stealing the functionality of a victim model $\mathcal{V}$ by training a clone model $\mathcal{C}$. These attacks usually follow the framework of knowledge distillation (Hinton et al., 2015), where the victim model $\mathcal{V}$ plays the role of “teacher”, and the knowledge of the teacher is distilled into the “student” model $\mathcal{C}$. The objective of the adversary is to maximize the classification accuracy of clone model $Acc(\mathcal{C}(x;\bm{\theta}_{\mathcal{C}}),y)$ on the victim’s target distribution $\mathcal{D}_{tar}$. Define $\bm{\theta}_{\mathcal{V}}$ to be the parameter of $\mathcal{V}$ and $\bm{\theta}_{\mathcal{C}}$ to be the parameter of $\mathcal{C}$, and the adversary’s goal could be formulated as Eqn. 1.

In most real-world settings, the adversary has no knowledge about the victim’s structure, parameters, or training set. The only interaction between the adversary and the victim is the black-box query process: the adversary inputs an image $x$ and the victim returns a softmax probability or logits. Though the original training set is unavailable, the adversary can use synthetic data or surrogate data to query the victim model. For example, JBDA (Papernot et al., 2017) synthesizes data from a small part of in-distribution samples, and KnockoffNets (Orekondy et al., 2019) uses surrogate datasets to query the victim model. Therefore, the adversary’s learning objective is a surrogate goal based on the distribution of the query dataset $\mathcal{D}_{que}$, which can be formulated as follows:

In the ideal scenario, if $\mathcal{D}_{tar}$ and $\mathcal{D}_{que}$ are close enough and the query budget is sufficient, the model stealing is generally inevitable since the victim $\mathcal{V}$ must guarantee the performance for benign users on the target distribution. However, in practice, $\mathcal{D}_{tar}$ and $\mathcal{D}_{que}$ are dissimilar due to the knowledge limitation of the adversary, and the query budget is limited by the adversary’s financial cost. It is these limitations that support existing defensive methods.

In defenses against model stealing, the defender aims at preventing the functionality of the victim model from being stolen with an acceptable impact on its benign accuracy. To be more practical, the accuracy degradation should be constrained with a minimum threshold $T$. The objective of the defender is to minimize the classification accuracy of the clone model $Acc(\mathcal{C}(x;\bm{\theta}_{\mathcal{C}}),y)$ on victim’s target distribution $\mathcal{D}_{tar}$, which could be formulated as Eqn. 3.

Considering the limitations of the adversary, existing defense methods either add adaptive perturbations to multiply the adversary’s query cost, or differentiate the victim’s behavior on ID and OOD data to deceive the adversary. In this paper, we propose a defensive method against model stealing attacks, which gets rid of the extra computational costs and ameliorates the trade-off between benign accuracy and stealing resistance.

Existing defensive methods are suffering from extra computational costs during inference since they often employ auxiliary inference-time modules. Studies have revealed that DNNs are heavily over-parameterized (Zhang et al., 2017) and can be trained to fit and generalize across diverse data distributions, e.g., adversarial examples for adversarially-trained models (Goodfellow et al., 2015; Madry et al., 2018; Zhang et al., 2021; Liu et al., 2021, 2023b; Guo et al., 2023; Tang et al., 2021) and real-world disturbance for reinforcement agents (Pinto et al., 2017; Guo et al., 2022; Li et al., 2023b, a). Inspired by them, we propose a defensive training framework to directly train a robust model, so that the model can generate deceptive outputs towards stealing attack queries without extra modules.

Generally, as Eqn. 1 shows, the adversary’s expected goal is to minimize the disagreement with the ground truth on the target distribution. This objective is not directly related to the victim’s parameter $\bm{\theta}_{\mathcal{V}}$, but the adversary must extract knowledge from the victim. As a consequence, the real objective of the adversary is illustrated in Eqn. 2. There exists a gap between the expected goal and the real objective, and the defense can be achieved by isolating the adversary’s real objective from the expected goal. As the adversary usually updates its parameters by gradient descent, we propose gradient isolation to isolate the real gradient from the expected gradient, thereby incorporating the robustness within the victim’s parameters to mislead the adversary.

To achieve gradient isolation, we need to estimate the above two gradient terms during training. Therefore, we introduce a surrogate white-box clone model $\mathcal{C}$ into the victim’s training process to represent the stealing role of the adversary. To isolate the update gradient from the expected gradient, we maximize the directional divergence between them. Specifically, for a certain batch of data $\bm{x}$, assuming the target of the adversary is $\bm{y}$, the update gradient can be written as:

The goal of gradient isolation is to maximize the directional divergence of these gradients, which can be quantified by the cosine similarity. Therefore, the objective of gradient isolation can be written as follows:

Some existing defenses add perturbations to the output over all samples, which harm the benign accuracy and cause low trade-offs. To improve the trade-off between clean accuracy and stealing robustness, we further design an adversary induction approach to train victim models. Thus, the victim model will behave normally for benign users but produce inductive outputs that induce the adversary to optimize without learning too much useful knowledge.

Successfully inducting the adversary stands on the assumption that benign and malicious queries can be distinguished through the distribution (Kariyappa and Qureshi, 2020; Kariyappa et al., 2021b). Practically, the adversary has limited knowledge of the distribution of the victim’s training set $\mathcal{D}_{tar}$, and consequently uses a surrogate dataset $\mathcal{D}_{que}$ to query and steal the victim model. Following this assumption, we regard the query samples of the adversary as out-of-distribution and apply an OOD dataset $\mathcal{D}_{out}$ during defense to substitute them. The over-parameterization property of DNNs ensures the generalization capability across diverse distributions and thus enables the acquisition of the victim that exhibits divergent behavior on benign in-distribution (ID) queries and malicious out-of-distribution (OOD) queries. Thus, our InI aims to guarantee the victim’s benign performance on ID queries, while inducing the adversary to attain little knowledge with uninformative outputs on OOD queries.

In particular, on ID samples, we should guarantee the victim’s benign performance. This can be achieved by applying a cross-entropy loss to train the classification model, which is shown as follows:

On OOD samples, we should induce the adversary to acquire minimal knowledge. Intuitively, the adversary induction can be realized by reducing the adversary’s information gain on OOD queries. The information gain can be quantified by the KL divergence between the output probabilities of the clone and the victim model, which can be formulated as below:

Noted that $\mathcal{L}_{ig}$ is the optimization objective of the adversary during model stealing. Therefore, when $\mathcal{L}_{ig}$ is minimized before the stealing process, the adversary can only gain little information via optimization. Thus, the adversary would only extract little knowledge from the victim model. Since the adversary often uses gradient descent to update their parameters and attain knowledge, minimizing the first-order approximation of the KL divergence can also help to reduce the information gain. Consequently, we can minimizing the norm of $\nabla_{\bm{\theta}_{\mathcal{C}}}\mathcal{L}_{ig}$, i.e., the gradient of the information gain. In summary, the objective of adversary induction can be formulated as:

$\mathcal{L}_{ind}$ is a function related to $\bm{\theta}_{\mathcal{V}}$, which can be integrated with our training framework. During the victim’s training, we apply a surrogate white-box clone model $\mathcal{C}$ and an OOD dataset to estimate the adversary’s information gain and minimize $\mathcal{L}_{ind}$ by updating the victim’s parameter $\bm{\theta}_{\mathcal{V}}$, thus reducing the knowledge leakage from the victim. The OOD dataset used in training comes from another classification task and is different from the query dataset in stealing attacks.

Though previous defenses (Kariyappa and Qureshi, 2020; Kariyappa et al., 2021b) utilize OOD datasets to train the victim, they only constrain the victim to produce meaningless or diverse outputs and did not take the adversary’s role into consideration. In contrast to them, InI produces inductive outputs that trigger the adversary to learn less, reducing the knowledge leakage from the victim.

Figure 2 illustrates our overall framework. To train a robust victim $\mathcal{V}$, we introduce a surrogate clone model $\mathcal{C}$ into the training process. During training, the victim has white-box access to the surrogate clone model. To jettison the auxiliary modules for low computational costs, InI isolates the adversary’s optimization gradient from the expected gradient during training via $\mathcal{L}_{iso}$ in Eqn. 5. To further improve the trade-off between benign performance and stealing robustness, InI produces uninformative outputs on malicious queries through $\mathcal{L}_{iso}$ in Eqn. 8 to induce the adversary to acquire less useful knowledge. By incorporating the robustness within the victim’s parameters, the victim will acquire how to resist the model stealing attacks in a favorable trade-off without any auxiliary modules during inference.

During training, all objectives are simultaneously calculated and updated. We use some hyper-parameters $\gamma_{1}$, $\gamma_{2}$ to control the trade-off between each loss, and the total loss can be written as:

However, during training, there may exist some conflicts between the optimization directions among the above losses. As the blue bars shown in Figure 3, we extract the gradients at the first 3 epochs of training and calculate their cosine similarities, where we observe that the cosine similarities between different objectives exist conflicts. To mitigate these conflicts, we leverage PCGrad (Yu et al., 2020) to deal with the gradient conflicts. Before the gradient descent update, PCGrad tries to find the conflict among these gradients and projects one gradient to the orthogonal direction of the others as follows:

After PCGrad stabilization, the conflicts are better mitigated and the optimization process is improved (see the orange bars in Figure 3). The overall pseudo-algorithm of our InI can be found in Supplementary Material.

In this part, we elaborate on our experimental settings about datasets, model architectures, defenses, attacks, and evaluation metrics.

Datasets and architectures. We evaluate our proposed InI on the most commonly-adopted image classification datasets for model stealing including MNIST (LeCun et al., 1998), FashionMNIST (Xiao et al., 2017), CIFAR-10 (Krizhevsky et al., 2009), and CIFAR-100 (Krizhevsky et al., 2009). We choose ResNet-18 (He et al., 2016) as the backbone of all victim models. We also evaluate results on VGG networks (Simonyan and Zisserman, 2014) which show similar observations (c.f. Supplementary Materials).

Implementation details. For the training of InI, we use an SGD optimizer with momentum 0.5 and a weight decay of $1\times 10^{-3}$. For MNIST and FashionMNIST datasets, we train 50 epochs with a learning rate annealing of 0.1 every 20 epochs, and for CIFAR-10 and CIFAR-100 datasets, we train 150 epochs with a learning rate annealing of 0.1 every 50 epochs. The initial learning rate is 0.1.

Defenses. To demonstrate the effectiveness of InI, we compare our method with the commonly-adopted defensive approaches: MAD (Orekondy et al., 2020), AM (Kariyappa and Qureshi, 2020), EDM (Kariyappa et al., 2021b). We also report the results of an undefended model denoted by “Vanilla”. We referred to the official implementation of these methods. The batch size of all defensive methods is 128. For the auxiliary OOD datasets used by AM, EDM, and InI, we choose KMNIST (Clanuwat et al., 2018) for MNIST and FashionMNIST, and choose TinyImageNet (Le and Yang, 2015) for CIFAR-10 and CIFAR-100. For the hash dataset used by EDM, we choose KMNIST for MNIST and FashionMNIST, and use SVHN (Netzer et al., 2011) for CIFAR-10 and CIFAR-100.

Attacks. Following the previous works(Orekondy et al., 2020; Kariyappa and Qureshi, 2020; Kariyappa et al., 2021b), to evaluate the performance of InI against model stealing, we use the commonly-used stealing attacks including KnockoffNets (Orekondy et al., 2019) and JBDA (Papernot et al., 2017), and evaluate the integration of InI with other defensive methods. For each attack, we conduct soft-label and hard-label attacks, which means the adversary learns according to the victim’s output probability and the top-1 label, respectively. The detailed settings of attacks are listed as follows:

• •

  KnockoffNets: The budget of attack is 50000. As for the surrogate datasets, we use EMNISTLetters (Cohen et al., 2017), EMNIST (Cohen et al., 2017), CIFAR-100, CIFAR-10 as the surrogate dataset for MNIST, FashionMNIST, CIFAR-10, CIFAR-100.

• •

  JBDA: We choose 150 images from the victim’s training set as the seed samples. We use 6 rounds of augmentation and a noise rate of 0.1 to synthesize the query data. The clone model is trained for 10 epochs every augmentation round.

Evaluation metrics. Following (Kariyappa et al., 2021b), we evaluate the performance of defenses by comparing the clone accuracy achieved by attacks on the victim’s test set. To take the victim’s benign accuracy into consideration, we further compare the relative performance of the defenses, which is the ratio of the adversary’s clone accuracy and the victim’s benign accuracy. For methods that have mutable parameters during inference (i.e., MAD and AM), we follow the setting in (Kariyappa et al., 2021b) and adjust the parameters of the defense to have similar benign accuracy on the test set. The benign accuracy of these defenses on classification tasks is shown in Table 1. For all the above metrics, the lower the better defenses.

In this part, we compare the performance of our InI with other model stealing defenses. We report the clone accuracy and the relative performance of existing defenses and InI in Tables 2 and 3. To further improve our defensive performance, we integrate the model trained by InI with MAD and AM and evaluate the performance. AM needs to jointly train the victim, yet we do not retrain our model in their framework but load the parameters from InI to replace the victim’s parameter for simplicity.

KnockoffNets attacks. Table 2 presents the defense results against KnockoffNets attacks. We can draw some observations listed below:

• •

  By inducing and isolating the adversary, InI achieves the best defense performance with similar benign accuracy on most of the results, which demonstrates our better trade-offs.

• •

  Noted that, on CIFAR-100 dataset, InI achieves an extraordinary defense performance against the soft-label attack, but behaves poorly against the hard-label attack.

JBDA attacks. Table 3 shows the defense performance against JBDA attacks. Different from KnockoffNets which applys surrogate datasets, JBDA uses a set of seed examples from the victim’s training set, and thus the results differ from KnockoffNets. From the results, we can observe that:

• •

  The OOD-based defenses (AM, EDM, and InI) behave poorly on the soft-label JBDA. This mainly results from that the samples of JBDA come from the seed examples in the training set and are close to the target distribution.

• •

  Instead, MAD, the perturbation-based method, could achieve the best performance on the soft-label attack on FashionMNIST, CIFAR-10, and CIFAR-100 datasets. However, its performance rapidly drops on the hard-label attacks, as it hardly modifies the top-1 label of the output.

• •

  JBDA achieves a good stealing performance on simpler datasets like MNIST and FashionMNIST, but cannot behave well on more complex datasets like CIFAR-10 and CIFAR-100. The JBDA stealing results on CIFAR-100 (around 0.05× on all defenses) is generally unusable.

Integration with other defenses. We also provide the experimental results of the integration of our InI with MAD and AM in Table 2 and 3. We can draw some observations that:

• •

  The integration of our InI with MAD and AM can further improve the trade-off, as they achieve further defense performance with invariant benign accuracy than the original MAD and AM against all the attacks. Additionally, the integration of our InI with AM mostly achieves more robustness than that with MAD.

• •

  On soft-label JBDA attacks, though the integration achieves improvement in Vanilla and InI, it still cannot surpass the performance of MAD in similar benign accuracy. We attribute this phenomenon to the robustness trade-off between soft-label and hard-label attacks, as InI has traded its benign accuracy off the robustness against hard-label attacks during training, and similar robustness can only be achieved at the cost of more benign performance.

In this section, we provide the inference speed analysis of each defensive method and evaluate our analysis through experiments. In practice, the model for MLaaS would only train once but would receive millions of inference queries from users, which the service provider would charge for. Therefore, boosting the inference speed has a significant impact on the practical use of defensive methods. Evaluations and discussions about training-time speed are provided in Supplementary Materials.

We first provide some analyses of the inference process of each method. As shown in Table 4, our InI can achieve the fastest inference speed among all defensive methods. On the contrary, existing methods employ auxiliary modules during inferences, which would introduce extra computational operations, or even harm the DNN’s parallel capability. Detailed analyses are given below.

Time cost of Vanilla and InI. We define the time cost of a forward operation as $M_{f}$ for a single query. The time cost of a model without defense should be $M_{f}$, and the same as InI since InI employs no extra modules.

Time cost of MAD. MAD computes the Jacobian matrix $G=\nabla\log f(x,\bm{\theta})$ to maximize the angular deviation between the perturbed gradient and the original gradient. The official code needs $C$ backward passes which costs $CM_{b}$ ($C$ is the number of classes of the target classification task) to calculate the Jacobian matrix. After getting $G$, MAD needs to heuristically search a perturbed probability $y^{*}$, which costs time of $S$. When the victim receives a batch of data with a mini-batch size of $B$, the backward pass and heuristic search would cost $BCM_{b}$ and $BS$, as these operations cannot perform in parallel.

Time cost of AM. AM employs an adaptive misinformation module to perturb the victim’s output probability. The adaptive misinformation is generated by a DNN with the same architecture as the victim’s backbone. As a consequence, the time cost of AM mainly comes from the forward passes of the victim’s backbone and the misinformation model, which cost around $2M_{f}$ in sum.

Time cost of EDM. EDM jointly trains an ensemble of $n$ victim models and selects a result from them according to a hash function. The hash function of EDM is a DNN with a simpler architecture, which costs $M_{h}$ for a forward pass. For a single query, the time cost of EDM should be $M_{f}+M_{h}$. However, in the official implementation of EDM, the time cost increase to $nM_{f}+M_{h}$ when EDM receives a batched query. The reason lies in that the forward pass of a single model and a batch of data can run in parallel, but the forward pass of different models cannot run in parallel. Therefore, all models in the ensemble would be accessed, and the time cost would increase.

Empirical evaluation. We perform empirical experiments to evaluate the speed of each defense. The experiment is conducted on an RTX 3080 GPU and the architecture of the victim’s backbone is ResNet-18. We randomly generate input images, execute the inference, and record the time cost. For fair comparisons, we conduct the process repeatedly for 2000 times and record the total time. As shown in Figure 4, other defenses consume significantly more time during inference compared to our method (1.98$\times$-25.4$\times$).

We then conduct ablation studies to understand the contributions of gradient isolation and adversary induction. Specifically, we conduct experiments by training the victim with (1) no extra defense (denoted by “Vanilla”); (2) only isolation loss $\mathcal{L}_{iso}$; (3) only induction loss $\mathcal{L}_{ind}$; (4) InI without $\nabla\mathcal{L}_{ig}$; and (5) the full InI ($\mathcal{L}_{iso}$+$\mathcal{L}_{ind}$). We record the defense performance against KnockoffNets and JBDA on the CIFAR-10 dataset. As shown in Table 5, we can draw several observations:

• •

  Model trained with $\mathcal{L}_{iso}$ has an apparent drop in stealing performance (e.g., 4.34 on soft-label KnockoffNets), which indicates that gradient isolation can bring robustness against model stealing.

• •

  Model trained with $\mathcal{L}_{ind}$ shows limited defenses against model stealing attacks, as it only induces the adversary to learn less instead of directly misleading the adversary.

• •

  With the two methods combined, InI can achieve the best defense performance (69.54 / 60.33 on KnockoffNets and 24.16 / 24.14 on JBDA), implying that the cooperation of gradient isolation and adversary induction plays an important role during training.