Synthetic data generation method for data-free knowledge distillation in regression neural networks

As neural networks become increasingly large in number of parameters, the deployment of such models faces a difficult challenge for applications such as mobile devices and embedded systems due to limitations in computational resources and memory [Cheng et al., 2018, Deng et al., 2020]. To address such problems, model compression through knowledge distillation has become an active area of research in recent years [Liu et al., 2022, Wang et al., 2022]. Knowledge distillation is the technique where knowledge learned by a larger teacher model is transferred to a smaller student model [Gou et al., 2021, Hinton et al., 2015, Wang and Yoon, 2022]. The main idea is that the student model mimics the teacher model to achieve a similar or even a superior performance.

Various methods of knowledge distillation define and focus on different forms of knowledge. Following the nomenclature in [Gou et al., 2021], these can be largely grouped as response-based knowledge, feature-based knowledge, and relation-based knowledge. For response-based knowledge, outputs of the teacher model are used to supervise the training of the student model. For example, [Hinton et al., 2015] uses soft targets from the logits output of the teacher model to train the student. For feature-based knowledge, outputs of intermediate layers, or feature maps learned by the teacher model can be to supervise the training of the student model. For example, [Romero et al., 2014] trains the student model to match the feature activations of the teacher model. For relationship-based knowledge, the relationships between different layers or data samples are used. For example, [Yim et al., 2017] uses the inner products between features from two layers to represent the relationship between different layers, while [Chen et al., 2021] trains the student model to learn to preserve the similarity of samples’ feature embeddings in the intermediate layers of the teacher models.

In some situations, access to the original data used to train the teacher model is not available due to issues such as privacy and legal reasons. Data-free knowledge distillation methods have been proposed to allow model distillation in the absence of original training data [Chawla et al., 2021, Chen et al., 2019, Hu et al., 2020, Lopes et al., 2017, Micaelli and Storkey, 2019, Nayak et al., 2021, Ye et al., 2020, Yoo et al., 2019]. This is achieved by generating synthetic data for training. Many methods achieve this by using generative adversarial networks (GAN) [Chen et al., 2019, Hu et al., 2020, Micaelli and Storkey, 2019, Ye et al., 2020, Yoo et al., 2019]. For example, [Micaelli and Storkey, 2019] train a generator model to generate synthetic images that maximizes the difference in prediction (measured by KL divergence) between the teacher and student models. The student model is then trained to minimize the difference on these synthetic images. When applied to image classification datasets (SVHN and CIFAR-10 datasets), they were able to train student models with performance close to distillation using actual full training set.

Other methods such as [Lopes et al., 2017] make use of metadata collected during training of the teacher model, in the form of the layer activation records of the teacher model to reconstruct dataset for training the student model.

Most of the methods currently existing in knowledge distillation literature deal with classification problems. These methods generally are not immediately applicable to regression problems where the predictions are unbounded real values. For regression problems, [Chen et al., 2017] uses a teacher bounded regression loss where the teacher’s predictions serve as an upper bound for the student model instead of using it directly as a target. This method was the first successful application of knowledge distillation to object detection problem and achieved strong results.

[Saputra et al., 2019] uses the teacher loss as a confidence score that assigns relative importance on the teacher prediction. This confidence score is then used in the main training task via an attentive imitation loss which adaptively down-weight the loss between the teacher and student model predictions when a particular teacher model’s prediction is not reliable. This method successfully performed knowledge distillation from a deep pose regression network, achieving a student model prediction close to the teacher model while reducing number of parameters by more than 90%.

[Takamoto et al., 2020] uses a teacher outlier rejection loss, that rejects outliers in training samples based on the teacher model predictions. This method was applied to predict gaze angles from images of human face and was demonstrated to be effective even with noisy labels.

[Xu et al., 2022] proposed the Contrastive Adversarial Knowledge Distillation (CAKD) approach for time-series regression tasks. CAKD uses adversarial adaptation to align feature distributions between student and teacher models. To achieve better alignment on fine-grained features, contrastive learning is also used to increase similarity between the features extracted by teacher and student models for the same sample. The teacher model prediction is used as soft labels to guide the training for the student.

[Kang and Kang, 2021] introduced the first work, and to our knowledge the only work that addresses data-free knowledge distillation for regression, by using a generator model that generates synthetic datapoints that is trained adversarially together with the student model. The method was able to substantially outperform the baseline methods of random sampling and data impression on various benchmark regression datasets. However, it was found that both the baseline methods and the generator method proposed here did not work well for deeper neural networks.

Given a trained teacher model T, and a student model $S_{\theta}$ parameterized by $\theta$, we generate synthetic data $x$ via some data generation method. The student model is trained by minimizing the student loss $L_{S}(x)$ defined in equation 1 using gradient descent. This generic method is illustrated in Figure 1.

The performance of the student model in mimicking the performance of the teacher is dependent on the representation strength $\theta$ of the student model, and the data $x$ used to train it, and the optimization process of minimizing student loss. Hence for a fixed student model architecture and training process, the synthetic data generation process plays the key role in determining the performance of the student model.

Three types of synthetic data generation methods are investigated in this study: random sampling, generative model, and direct optimization.

To facilitate comparison with the previous work by [Kang and Kang, 2021], the experiments were conducted on the same datasets. These 7 datasets are regression problem sets available from UCI machine learning repository [Dua and Graff, 2017] and KEEL dataset repository [Alcala-Fdez et al., 2010]. ‘longitude’ was selected as the output variable for Indoorloc. Details of the datasets are provided in the Table 1.

The data are split into training and test set. The training set consists of 5000 samples for each dataset. 10% of the remainder samples are placed into the validation set, and the remaining 90% is the test set. The validation set is used to periodically evaluate the training of the student model.

For data processing step, all values were standardized to a mean of 0 and a standard deviation of 1. Two processing workflows were tested where the scaling factors were calculated for the training set only and then applied to the test set, and where the scaling was done on the whole dataset prior to splitting of training and testing data. No significant differences were observed for both workflows, with differences in the RMSE of teacher models differing by less than 1.5%. Therefore, the second workflow was used for the results for simplicity.

To facilitate comparison, we used the same experiment setup for the neural networks as was used in [Kang and Kang, 2021]. The teacher model is a fully connected feed forward network containing 1 hidden layer of 500 units with Tanh activation function. The student model is also a fully connected feed forward network containing 1 hidden layer of either 25, or 50 units with Tanh activation function.

The teacher model is trained with the training data, while student models are trained without access to any real data from the training set. RMSProp optimizer is used for gradient descent, with a learning rate of $10^{-3}$ and weight decay regularization of $10^{-5}$. Batch size $m$ is set to be 50, and the number of batches in each epoch, $n_{s}$ is set to be 10. $\beta$ and $\gamma$ are selected to be $10^{-5}$. The number of epochs is selected as 2000. Models that performed the best on the validation loss was used to evaluate on the test set. For the direct optimization method to generate synthetic data, RMSProp optimizer with a learning rate of $10^{-1}$, and 2 epochs were used, how these two hyperparameters were selected are elaborated in the results section 4.1.

To further test the applicability of our method on different types of inputs, and on deeper and more complex neural network architectures, we designed an experiment for data-free knowledge distillation for regression on the MNIST handwritten digits dataset.

The MNIST dataset is originally intended to be used for classification task, following the method presented in [Wang et al., 2020], we adapt it for regression task by making the neural network to predict a continuous number that represent the class value of the digit label of the input image. The performance of the model is measured in mean absolute error (MAE) between the predicted value and the actual value of the digit. For e.g. for perfect performance, the model should predict a value of 3.0 for an image with the handwritten digit 3. A prediction of 2.9 will result in a MAE of 0.1.

The input image in MNIST is a single channel image of size 28 by 28 pixels, each pixel taking a value between 0 – 1. The mean $\mu$ and standard deviation $\sigma$ of each pixel position is calculated for the entire dataset and is used to generate random datapoints with a normal distribution $\mathcal{N}(\mu,\sigma)$ clipped between 0 – 1. This is done to achieve as much as possible a similar distribution as real data.

As proposed by [Wang et al., 2020], we used a multi-layer convolutional neural network with the architecture specified in Table 2. The teacher and student network follow the same architecture, except that the number of filters, $f$ for each convolutional layer in the teacher network is higher than that in the student network. $f$ is chosen to be 10 for the teacher network and 5 for the student network. Log hyperbolic cosine (Log-Cosh) loss was used instead of mean squared error as the loss function to improve training following method proposed by [Wang et al., 2020].

Due to the different nature of the input, which are images rather than standardized tabular data in the regression datasets, and the output which are natural number, we designed a different loss function for generating synthetic data. This loss function differs from equations 3 and 5 by replacing the mean-squared error loss with Log-Cosh loss and by changing the regularization terms to better capture the distribution of real data. Firstly, instead of penalizing the $L_{2}$ norm of $x_{g}$, we penalize the $L_{1}$ norm of $x_{g}$ because the handwritten digits image tends to be sparse. Secondly, instead of penalizing the student prediction on $x_{g}$, we randomly sample a whole number from 0 – 9 and penalize the distance of the teacher’s prediction to the random whole number. The purpose of this penalty is to allow the synthetically generated sample $x_{g}$ to match more closely with the actual data distribution, as real data should generally not be predicted too far away from whole number by the teacher model for this task. This strategy may also be helpful for other regression tasks where the output lies in a fixed set of discrete values.

To train the student model, RMSProp optimizer was used for gradient descent, with a learning rate of $10^{-3}$ and weight decay regularization of $10^{-5}$. Batch size $m$ is set to be 50, and the number of batches in each epoch, $n_{s}$ is set to be 10. The number of epochs is selected as 1000.

For the direct optimization method to generate synthetic data, RMSProp optimizer with a learning rate of $10^{-3}$, and 20 epochs were used. For the generator network, the number of rounds for training the generator per epoch was also set to 20.

A bioinformatics problem, predicting continuous protein solubility value with the constituent amino acids [Han et al., 2019], was used as a case study to test the effectiveness of data-free knowledge distillation for regression on a real-world scientific problem. Predicting continuous solubility value is useful for in-silico screening and design of proteins for industrial applications [Han et al., 2020].

We also want to test how the method can be used when the gradients of the teacher model are not available. For example, many bioinformatics tools such as protein solubility prediction are hosted on servers that allow users to query proteins and obtain predictions. However, both the model and data used to train the model are not available to the user. To recreate the model, data-free knowledge distillation without gradient access to the teacher model is required. If gradient information of the teacher model is unavailable, it is not possible to train the generative network as described in 3.2.2 directly. However, for direct optimization, it is possible to use metaheuristics optimization that does not require gradients instead of gradient descent.

The dataset used contains 3148 proteins with solubility represented as a continuous value between 0 – 1 from the eSol database [Niwa et al., 2009]. The input features are the proportion of each of the 20 amino acids within the protein sequence. 2500 proteins are selected for the training set, and the remaining as test set. The teacher model used is a support vector machine, which represents the black-box teacher model that contains no gradient information and only output prediction value is available.

As in the MNIST example, we introduce diversity in the predicted value by the teacher model on $x_{g}$ with a penalty term on distance away from a random $y$ value sampled for every batch.

The student model is made up of a fully connected Gaussian kernel radial basis function layer with output size of 100, followed by a fully connected linear layer that outputs the prediction. For training the student models in both baseline and direct optimization method, RMSProp optimizer is used for gradient descent, with a learning rate of $10^{-3}$ and weight decay regularization of $10^{-6}$. Batch size $m$ is set to be 50 with decreasing $\alpha$ schedule.

Random sampling was used for training baseline model and providing initial points for direct optimization method. The mean $\mu$ and standard deviation $\sigma$ of each amino acid feature is calculated from the training dataset and is used to generate random datapoints with a normal distribution $\mathcal{N}(\mu,\sigma)$ clipped between 0 – 1. The feature values are then normalized such that the value sums to 1. This is done as the features which are proportion of each of the 20 amino acids within the protein sequence must sum to 1. For the direct optimization method to generate synthetic data, differential evolution algorithm [Storn and Price, 1997] with 25 iterations of best2bin strategy was used, with initial points generated with the random sampling just described.

We first investigate the properties of the synthetic data generated by various methods, namely the student loss value of the synthetic data, and the distribution of the synthetic data.

Table 3 and Table 4 shows the comparison of root mean squared error (RMSE) for 5 methods of data-free distillation using student size of 25 and 50 respectively. For the generator method and direct optimization method, both a decreasing $\alpha$ schedule and $\alpha$ value of 1 are tested. The $\alpha$ value of 1 means that the training uses the generated synthetic data $x_{g}$ entirely without any randomly sampled datapoints.

Comparing the results for student model size of 25 and 50 hidden units, it is observed that with an increase in student model size, the RMSE is lower for all datasets due to the greater representation power of the student model. For most of the datasets tested, the direct optimization method achieves the lowest RMSE and most closely matches the performance of the teacher model. Compared against random sampling, direct optimization with decreasing alpha achieves lower RMSE on 6 out of 7 datasets. Compared against generator method with decreasing $\alpha$, direct optimization with decreasing $\alpha$ achieves lower RMSE on 5 out of 7 datasets.

When $\alpha$ is set to 1, we observe a substantial increase in RMSE for the generator method. However, for the direct optimization method, setting $\alpha$ to 1 generally does not lead to much worse performance. This matches our hypothesis that a decreasing $\alpha$ schedule is much more important for the generator method as the synthetic datapoint generated tends to deviate more from the underlying distribution at a later stage of training. Compared with generator method with decreasing $\alpha$, direct optimization $\alpha=1$ (i.e. training on $x_{g}$ only) achieves lower RMSE on 5 out of 7 datasets.

Figure 7 shows the RMSE on the validation set over the course of training of the student model of size 50. The direct optimization method shows a faster decrease in RMSE and a generally more stable learning behavior than the generator method. (Plot values have been smoothed with a Savitzky–Golay filter of window size of 15 epochs to reduce noise for better visualization)

We also examine the student loss on $x_{g}$ for the two models where $\alpha=1$. As seen in Figure 8, the generator method often produce unexpectedly large losses during training that could results in negative learning for the model, whereas the direct optimization method generally produces a stable and consistently decreasing loss.

We experimented with different settings of $\beta$, $\gamma$ and $\epsilon$ weights for the various components in the loss function (equation 8) and found that a low $\beta$ value ($10^{-6}$), high $\gamma$ value (set to 1), low $\epsilon$ value (set to $10^{-6}$) and provides good regularization that encourages synthetic data generated to be diverse and resemble real data distribution more closely.

Table 5 below shows the comparison of mean absolute error (MAE) as well as the RMSE achieved by training the student model with synthetic data sampled randomly, generated by the generator method and by the direct optimization method. MAE was used as the evaluation metric following the original study [Wang et al., 2020]. Results are averaged over 5 runs. Note that the best performing random model that outputs a constant value of 4.5 would give a MAE of approximately 2.5 for a class balanced test set.

We can observe that the generator method performs only slightly better than random prediction. The direct optimization method was able to provide a substantial improvement in performance compared to the other methods. We examine samples of the synthetic data generated by each method in Figure 9. It is observed that the direct optimization method generates synthetic data closer to what appears to be handwritten digits compared to the other methods. We also examine the histograms of predicted values by teacher networks on a batch of 50 synthetically generated data in Figure 10. It is observed that direct optimization method generates samples with the most diversity while maintaining closeness to integer values. The closer resemblance to real data distribution is likely the reason the student model trained on those synthetic data distills more useful knowledge from the teacher model and outperforms the other methods.

This experiment demonstrates that the direct optimization method can be easily and effectively adapted for data-free knowledge distillation for regression tasks on image inputs, different types of student or generator loss functions and for multilayer networks with non-MLP architectures which potentially addresses the limitation raised in [Kang and Kang, 2021] on poor applicability of the generator method for data-free knowledge distillation of multilayer networks for regression.

We experimented with different settings of $\epsilon$ value and found that a value of 0.05 encourages synthetic data generated to be diverse and provides the best training results.

Table 6 shows the comparison of root mean squared error (RMSE) for the teacher model. The performance obtained for the teacher model is comparable with those obtained in the original study [Han et al., 2019] on regression predictive model for protein solubility. Results are averaged over 5 runs. Note that for this dataset, a random model that outputs values uniformly drawn from 0 – 1 will give a RMSE of approximately 0.43.

It is observed that direct optimization method with differential evolution outperforms random sampling significantly and approaches the RMSE of the teacher model. This case study demonstrated that direct optimization can be easily and effectively applied to cases where gradient information from teacher model is not available, or if the teacher model is not a differentiable neural network at all, such as the support vector machine teacher model in this case. This is achieved by simply swapping the gradient descent with a metaheuristics algorithm for direct optimization. Whereas, using a conventional neural network generative model for synthetic data generation is not possible as the training for the generative model relies on gradients of both the teacher and student model.

The limitation however is that metaheuristics optimization methods tend to be much slower than gradient based optimization and thus incur and significant increase in runtime over the baseline method during training. This may be improved by using a faster metaheuristics algorithm, or one that has been optimized to run on GPU, but that is beyond the scope of this paper.

Assuming a $d$-dimensional $K$-Lipschitz continuous loss function, direct optimization of $x$ with gradient descent in fixed number of steps $t$ on the loss function results in $x_{t}$ that is bounded in distance away from $x$.

Proof:

Given a $d$-dimensional $K$-Lipschitz function, by definition, for all real $x_{1}$ and $x_{2}$:

Let $x_{1}=x_{2}+\delta x$ and taking the limits of $\delta x\to 0$:

For gradient descent, the update formula is:

The squared distance moved in the first step of gradient descent is bounded:

Therefore, the distance moved in $t$ steps of gradient descent where $t\geq 1$ on a $d$-dimensional $K$-Lipschitz function is bounded by:

Assuming a $d$-dimensional $K$-Lipschitz continuous loss function, generating synthetic data with a generator network trained with the $L_{2}$ regularized loss function [equation 3] will result in $x_{g}$ that is bounded in distance away from 0, i.e. $L_{2}$ norm of $x_{g}$.

Proof: Given a $d$-dimensional $K$-Lipschitz continuous loss function, the gradient at any point is bounded by $K$:

The generator network is trained to map random noise vector $z$ to $x_{g}$ to minimize the loss function with a $L_{2}$ regularization on $x_{g}$, i.e

As we are minimizing the loss function, we only focus on the lower bound on $f(x_{g})$

Then assuming sufficiently small learning rate, the solution $x_{g}$ will converge upon a point where gradient of the loss function is 0:

Therefore, the bound for the $L_{2}$ norm of $x_{g}$ is: