Learning Kernel for Conditional Moment-Matching Discrepancy-based Image Classification

The principal problem that pattern classification concerns is whether conditional distribution equation $P(Y|X)=P(\widehat{Y}|X)$ holds or not. Intuitively, CMMD is suitable for solving the pattern classification task since it can capture the discrepancy between two conditional distributions. Suppose that $\phi$ and $\psi$ denote the nonlinear mapping functions for $X$ and $Y$, respectively. Then cross-covariance $C_{XY}:\mathcal{G}\rightarrow\mathcal{F}$ is defined as [33]:

where $\otimes$ is the tensor product operator. The embedding of conditional distributions $C_{Y|X}$ can be defined as [34]:

Given a dataset $\mathcal{D}_{XY}$ drawn $i.i.d$ from $P(X,Y)$, $C_{Y|X}$ can be estimated by:

where $\Phi$ denotes the embedding of $X$, $\Psi$ denotes the embedding of $Y$, $\mathcal{K}=\Phi^{\top}\Phi$ is the Gram matrix, and $\lambda$ is a positive regularization parameter.

We randomly sample two batches, e.g., $\mathcal{D}_{XY}^{s}=\{(\textbf{x}_{i}^{s},y_{i}^{s})\}_{i=1}^{n}$ and $\mathcal{D}_{X\hat{Y}}^{t}=\{(\textbf{x}_{i}^{t},\widehat{y}_{i}^{t})\}_{i=1}^{n}$, from $P(Y|X)$ and $P(\widehat{Y}|X)$, respectively. Let $\textbf{X}_{s}=[\textbf{x}_{1}^{s},\cdots,\textbf{x}_{n}^{s}]$ and $\textbf{X}_{t}=[\textbf{x}_{1}^{t},\cdots,\textbf{x}_{n}^{t}]$, and the superscripts or subscripts $s$ and $t$ represent the two sample sets, respectively. $\hat{y}_{i}^{t}$ is the predicted label of $x_{i}^{t}$, and it is derived by some specified prediction method such as softmax. It is worth noting that $\textbf{X}_{s}$ and $\textbf{X}_{t}$ may be non-overlapping, because in some applications, e.g., semi-supervised classification tasks, there are some unlabeled input data. The empirical estimation of CMMD is

where $\Psi_{s}=[\psi(y_{1}^{s}),\cdots,\psi(y_{n}^{s})]$, $\Phi_{s}=[\phi(\textbf{x}_{1}^{s}),\cdots,\phi(\textbf{x}_{n}^{s})]$, $K_{s}=\Phi_{s}^{\top}\Phi_{s}$, $\widetilde{K}_{s}=K_{s}+\lambda\textbf{I}$, $\mathcal{L}_{s}=\Psi_{s}^{\top}\Psi_{s}$. Other variables relating to subscribe $t$ are defined in a similar way on dataset $\mathcal{D}_{X\hat{Y}}^{t}$. By minimizing this CMMD-based loss function, the difference between $P(Y|X)$ and $P(\widehat{Y}|X)$ is reduced to a reasonable range. In particular, according to Theorem 3 shown in [22], when CMMD approaches its minimum value of 0, the predicted distribution converges to the true distribution. It means that the CMMD criterion can be used to learn the category distribution.

However, CMMD fails to measure the similarity between two conditional distributions if the kernel function is selected improperly. It is well known that the performance of MMD is limited by the quality of the kernel function [35, 36]. When dealing with complex distributions, it is difficult to effectively represent similarity between samples using the RBF kernel and the Laplacian kernel, which lead to the degradation of MMD-based algorithms. The same problem exists in the CMMD criterion. We present a demonstrative instance here. We randomly select a sample batch and plot the heat map of the matrix $\tilde{K}^{-1}K\tilde{K}^{-1}$, which is the weight matrix of $\mathcal{L}$ in Equation (3) and is called H here for convenience. Figure 1 shows the H matrix by using the samples with random sampling. The off-diagonal elements of H are almost the same. Besides, we randomly select a sample batch from one single class, and then plot the heat map of matrix H in Figure 1. There is almost no difference between the two matrices. It indicates that the kernel function used in CMMD criterion fails to distinguish the between-class similarities and the within-class similarities when dealing with complex distributions. The heat maps of H matrix on the SVHN and CIFAR-10 datasets show very similar comparison results, and they are presented in Appendix A.

In summary, kernel function learning module plays an important role in characterizing the loss function $L_{\rm{CMMD}}$. If the kernel functions are not learned well, then CMMD loses its intrinsic discriminative representation ability. In contrary, well-learned kernel functions are useful to extract robust and discriminative features for classification. Therefore, a better kernel function is expected to improve the classification performance of the CMMD criterion.

In classification tasks, it is desirable that distances between samples with the same label are small in the learned metric space, while distances between those with different labels are large. However, as shown in Figure 1, when the nonlinear kernel functions are directly applied to input features $\textbf{X}_{s}$ and $\textbf{X}_{t}$, the RBF kernel function tends to derive a similar Gram matrix for the high-dimensional and normalized data. In this case, CMMD cannot characterize the weights of errors well, thus the classification performance degenerates rapidly.

By Equation (3), the distance $L_{\rm{CMMD}}$ of the two conditional distributions $P(Y|X)$ and $P(\tilde{Y}|X)$ can be estimated by sampling finite samples. Due to sampling bias, $L_{\rm{CMMD}}$ may not be zero (but should be close to 0) when $P(Y|X)=P(\tilde{Y}|X)$. When $L_{\rm{CMMD}}$ is a large value, the pre-specified kernel function may be inappropriate. Intuitively, if a kernel $K$ gets a small $L_{\rm{CMMD}}$ when $P(Y|X)=P(\tilde{Y}|X)$, it is likely to describe the similarity better between samples. So, instead of directly using a pre-specified kernel $K$ for the input features X, we expect to obtain a more suitable kernel function via

In Equation (4), all characteristic kernels should be considered. However, it is difficult to choose the optimal solution from all the characterized kernels. According to the theory proposed by [35], if function $h$ is injective and the kernel function $K$ is characteristic, then the compound kernel function $\mathcal{K}=K\circ h$, which satisfies to $\mathcal{K}(x,x^{{}^{\prime}})=K(h(x),h(x^{{}^{\prime}}))$, is still characteristic. In this paper, the optimal solution of Equation (4) is determined via selecting from a series of kernel functions $\mathcal{K}$. In other words, the kernel function is not applied directly to the raw data $X$, but instead applied to the embedded feature $Z$ with an $h$ transformation, which is represented by a deep network. Here an injective function $h_{\textbf{w}}$ with parameter w is considered. Let $\textbf{Z}=h_{\textbf{w}}(\textbf{X})$. We use subscript $\_z$ to denote the features referring to Z and to distinguish them from raw features X. Then the kernel learning loss by the CMMD criterion in the new model can be defined as:

in which $\Psi_{s}=[\psi(y_{1}^{s}),\cdots,\psi(y_{n}^{s})]$, $\mathcal{L}_{s}=\Psi_{s}^{\top}\Psi_{s}$, $\Phi_{s\_z}=[\phi(\textbf{z}_{1}^{s}),\cdots,\phi(\textbf{z}_{n}^{s})]=[\phi(h_{\textbf{w}}(\textbf{x}_{1}^{s})),\cdots,\phi(h_{\textbf{w}}(\textbf{x}_{n}^{s}))]$, $K_{s\_z}=\Phi_{s\_z}^{\top}\Phi_{s\_z}$, and $\widetilde{K}_{s\_z}\!\!=\!\!K_{s\_z}\!+\!\lambda\textbf{I}$. Other variables relating to subscribe $t$ are defined in a similar way on $\mathcal{D}_{XY}^{t}$. $K_{ts\_z}=\Phi_{t\_z}^{\top}\Phi_{s\_z}$, $\mathcal{L}_{st}=\Psi_{s}^{\top}\Psi_{t}$. By minimizing $L_{\rm{CMMD\_Z}}(\textbf{w})$, the similarity value of positive pair (i.e., in the same class) becomes larger, while the similarity value of negative pair (i.e., in different classes) becomes smaller.

In addition, to ensure that $h_{\textbf{w}}$ is injective or approximately injective, an auto-encoder (AE) module is designed in KLN. For an injective function $h$, there exists an inverse function $h^{-1}$ such that $h^{-1}(h(\textbf{x}))=\textbf{x}$, $\forall\textbf{x}\in\mathcal{X}$, which can be approximated by an AE. In the following, we use $\textbf{w}=\{\textbf{w}_{e},\textbf{w}_{d}\}$ to denote the network parameters of AE, in which $\textbf{w}_{e}$ denotes the encoder parameters and $\textbf{w}_{d}$ the decoder parameters. Correspondingly, $h_{\textbf{w}_{e}}$ is the encoder and $h_{\textbf{w}_{d}}$ is the decoder. The reconstruction loss of AE can be defined as:

In this AE architecture, decoder satisfies to $h_{\textbf{w}_{d}}\approx h_{\textbf{w}_{e}}^{-1}$ when the reconstruction loss is minimized, which ensures that $h_{\textbf{w}_{e}}$ is approximately injective.

Taking the AE regularization module into account, the objective function of KLN is formulated as

The first term is the CMMD criterion based on the new kernel $\mathcal{K}=K\circ h_{w_{e}}$, and the second term is the expected error from the AE, which can be viewed as a regularization term to ensure that the learned kernel function is characteristic. The trade-off positive parameter $\beta$ is predefined by users.

In the previous section, KLN is proposed based on the CMMD criterion and the AE framework. In order to deal with supervised classification tasks, a classification layer can be attached onto top of the hidden layer. As shown in Figure 2, a fully connected layer with softmax activation is connected to the hidden layer. The label prediction distribution and the kernel function are learned simultaneously. In particular, the CMMD criterion is adopted to measure prediction errors. The CMMD between the prediction distribution $P(\widehat{Y}|X)$ and the true distribution $P(Y|X)$ can be estimated as:

Comparing Equations (5) and (8), $L_{\rm{CMMD\_Z}}(\widehat{Y})$ is an effective approximation of $L_{\rm{CMMD\_Z}}(\textbf{w}_{e})$ when $P(\widehat{Y}|X)\approx P(Y|X)$. In order to reduce the computational complexity, $L_{\rm{CMMD\_Z}}(\widehat{Y})$, instead of $L_{\rm{CMMD\_Z}}(\textbf{w}_{e})$, is used to learn the kernel function. In this case, the CMMD distance is calculated only once. Let $\textbf{w}_{fc}$ represent the network parameters of the fully connected layer, the objective of KLN for supervised classification tasks is modified as follows,

We call the first term as CMMD loss, and the next term as AE regularization term. In Equation (9), the CMMD loss is used to learn the label prediction distribution and the kernel function simultaneously, and the AE regularization term is used to ensure that the learned kernel function is characteristic.

Figure 2 shows the flowchart of our KLN method. KLN uses the encoder to map the high-dimensional raw data X to the latent feature Z, which can be passed through the decoder to reconstruct X. To predict label $Y$, the latent variable Z is propagated to the loss layer via a non-linear transformation with FC-layers. Main steps of KLN for supervised classification tasks are summarized and then shown in Algorithm 1.

When the training process shown in Algorithm 1 is completed, we can obtain the prediction distribution $P(\hat{Y}|X)$, which is represented by the encoder of AE and the fully connected layer. When testing a new sample $\textbf{x}_{test}$, it is first input to the encoder to get the latent feature $\textbf{z}_{test}$, which is then input to the fully connected layer to get the predicted label $\hat{\textbf{y}}_{test}$.

We now consider adapting the formulation from Section III-C to the semi-supervised setting. Let $\textbf{X}=[\textbf{x}_{1},\cdots,\textbf{x}_{N}]$ be a set of all $N$ examples and $\textbf{X}^{l}=[\textbf{x}_{1}^{l},\cdots,\textbf{x}_{L}^{l}]$ be a set of $L$ labeled examples. In particular, $\textbf{X}^{l}\subset\textbf{X}$. For every $\textbf{x}_{i}\in\textbf{X}^{l}$, the ground-truth label $y_{i}\in{1,\cdots,C}$ is known, where $C$ is the number of classes.

In Equation (8), $\textbf{X}_{s}$ needs its true label and $\textbf{X}_{t}$ can use the predicted label, so $\textbf{X}_{s}$ can be sampled from $\textbf{X}^{l}$ and $\textbf{X}_{t}$ from X to estimate the CMMD loss. It means that in semi-supervised classification tasks, the CMMD loss can simultaneously use a small amount of labeled data and a large amount of unlabeled data to learn a more suitable kernel function. In addition, to make better use of the unlabeled data, we also introduce the confidence loss [27] for these data, which is commonly used in the semi-supervised learning models. The confidence loss $L_{con}$ minimizes the conditional entropy of $P(\widehat{Y}|\textbf{X})$ and the cross entropy between $P(Y)$ and $P(\widehat{Y})$. Accordingly, the objective of KLN for semi-supervised classification tasks is presented as follows,

where $\beta_{1}$ and $\beta_{2}$ are trade-off parameters. Comparing with the training process for supervised tasks, the optimization process and the sampling process are different. Main steps of KLN for semi-supervised classification tasks are summarized and then shown in Algorithm 2.

KLN learns discriminative features in an end-to-end manner. The loss functions (9) and (10) can be optimized by the Back-Propagation algorithm, which can be implemented automatically and efficiently by several deep learning frameworks including TensorFlow and PyTorch. Thus, the derivative details are omitted here. More implementation details will be discussed in the next section.

Four benchmark datasets, i.e., MNIST [37], SVHN [38], CIFAR-10 and CIFAR-100 [39], were used for evaluating the image classification performance in this section. Some exemplar instances are shown in Figure 3.

MNIST handwritten digits set is one of the most popular datasets used in the deep learning literature. It has 70,000 images of ten classes (from 0 to 9). Each sample is a 28$\times$28 gray image. The whole dataset was divided into three non-overlapping parts: one training set of 50,000 samples, one validation set of 10,000 samples, and one test set of 10,000 samples.

SVHN is a more complex dataset than MNIST as it has more samples and is closer to real scenes. Each sample is a $3\times 32\times 32$ color image. All the images were captured from house numbers in Google street views. The whole set is also composed of three different subsets, namely, one training set consisting of 73,257 samples, one testing set consisting of 26,032 samples, and one extra training set consisting of 531,131 samples.

CIFAR-10 is another image dataset. It consists of ten-class images including airplanes, cars, cats, and so on. Each class has 6,000 color images of size $3\times 32\times 32$. The whole set was divided into two separate parts, i.e., one training set of 50,000 images, and one test set of 10,000 images.

CIFAR-100 contains 60,000 color images of size $3\times 32\times 32$ drawn from 100 classes. The dataset was split into 50,000 training images and 10,000 test images.

All experiments were implemented by PyTorch Toolbox, and ran on a PC equipped with a NVIDIA TiTan X GPU and 32G RAM.

Data processing: For MNIST, no preprocessing was applied to the data except for scaling to the numerical range $[0,1]$. SVHN and CIFAR datasets require more techniques for image preprocessing. In addition to the global normalization, a series of data augmentation methods including random cropping and horizontal flipping were applied.

Network architecture: On the MNIST dataset, we followed the architecture of DCGAN [25] to design the network architecture of KLN. We replaced the pooling layers with strided convolutions, and used batch-wise normalization and LeakyReLU activation in the auto-encoder. On the SVHN, CIFAR-10 and CIFAR-100 datasets, we followed the architecture in [40]. More specific parameter settings are placed in Appendix B.

Hyper-parameters: For model optimization, the batch size was set to 100 for all methods. For supervised classification tasks, SGD algorithm with initial learning rate 0.02, momentum 0.9 and weight decay 0.0005 was used. On CIFAR and MNIST, the learning rate reduces by 0.2 at 50, 100, 130 epochs, and all the networks were trained by a total of 150 epochs. On SVHN, the learning rate reduced by 0.2 at 10, 20, 30 epochs, and the training procedure finished in 40 epoches. For semi-supervised classification tasks, ADAM algorithm [41] with learning rate 1$e$-3 and the two moment terms $\gamma_{1}=0.9$, $\gamma_{2}=0.99$ was used. The dimensionality $d$ of the latent variable Z is $d=128$. The CMMD loss was calculated by using multiple Gaussian kernel functions with bandwidth parameters [1, 3, 5, 7, 9]. For the weight hyper-parameters, we simply set $\beta=0.1$ in supervised classification tasks and $\beta_{1}=0.1,\beta_{2}=1$ in semi-supervised classification tasks.

This section compares KLN with some state-of-the-art supervised classification methods, which include VA+Pegasos [42], MMVA [42], Stochastic Pooling [43], Network in Network (NIN) [6], Maxout Network [44], DSN [45], ResNet [46] and so on. For a fair comparison, the same network architecture was used to evaluate the performance of the most recent method CGMMN-CNN. For the rest methods, the best results are from the original work.

Table I shows the classification results of MNIST. KLN was trained with a total of 150 epochs. The encoder and decoder of AE used a 4-layer fully convolutional network and a 4-layer de-convolutional network, respectively. The results show that KLN is competitive with various state-of-the-art competitors, e.g., CMMVA and DSN. DSN obtains the minimum error rate as it benefits from using more supervised information in every hidden layer, and KLN gets the same classification performance as DSN. In particular, we pay our main attention on comparing the results of CGMMN and KLN. Two types of architectures of CGMMN, as stated in [22], i.e., CGMMN-MLP and CGMMN-CNN, are evaluated and then analyzed here. The results of CGMMN-MLP and CGMMN-CNN are 0.97% and 0.45%, respectively, while the accuracy of KLN is 0.39%. Thus, KLN outperforms CGMMN-MLP and CGMMN-CNN.

Table II shows the classification results of SVHN. KLN was trained with a total of 40 epochs. The encoder and decoder of AE used a 9-layer convolutional network and a 4-layer de-convolutional network, respectively. The results of CNN and CMMVA are 4.9% and 3.09%, respectively. DSN still obtains a lower error rate as it benefits from using more supervised information in every hidden layer. The results of DSN, CGMMN-CNN and KLN are 1.92%, 2.01% and 1.56%, respectively. Thus, KLN outperforms both DSN and CGMMN, and the effectiveness of algorithmic improvement is validated again.

Table III shows the classification results of CIFAR-10. KLN was trained with a total of 150 epochs. The encoder and decoder of AE used a 9-layer convolutional network and a 4-layer de-convolutional network, respectively. All the error rates of CNN, NIN, Maxout Network, and DSN are larger than 7.0%. The results of FitNet4-LSUV, ResNet-110 and CGMMN-CNN are 6.06%, 6.43% and 6.61%, respectively, while that of KLN is 5.15%. Therefore, KLN outperforms both DSN and CGMMN-CNN on the CIFAR-10 dataset, and the effectiveness of algorithmic improvement is validated again.

Table IV shows the results of CIFAR-100. KLN was trained with the same parameter settings as those of CIFAR-10. All the error rates of CNN, NIN, Maxout Network, and DSN are larger than 30.0%. The results of FitNet4-LSUV, ResNet-110 and CGMMN-CNN are 27.66%, 25.16% and 24.84%, respectively, while that of KLN is 22.63%. Compared with the results of CGMMN method, KLN has more than two percent increments.

In Equation (9), the weight hyper-parameter $\beta$ is introduced to control the weight of the AE regularization term. We further evaluated the susceptibility of $\beta$. Several optional $\beta$ values were pre-specified in the set $\{10,1,0.1,0.01,0.001\}$. Ablation experiments were conducted to understand real impact of the AE regularization term, which corresponds to $\beta=0$. Figure 4 shows the classification results on SVHN and CIFAR-10 for demonstration. The results of KLN are insensitive to the value of $\beta$, and the model achieves slightly better performance when $\beta$ is set to 0.1. It is inferred that KLN usually learns the injective mapping with high probability when learning a more discriminative kernel function and parameterizing with deep neural networks.

The training time of KLN and CGMMN in one epoch is shown in Table V. In principle, the optimization procedure of CMMD contains two separate but related modules, one from the output layer to the latent layer, and the other from the latent layer to the input layer. Both of them can be learned efficiently by the standard Back-Propagation process. Compared with CGMMN, KLN has an additional AE module. Fortunately, the decoder of the AE is mainly composed of de-convolutional operations, which correspond to the convolutional layers used by the encoder. In other words, the AE does not take much time in optimization. It is worth noting that two different subsets were used in each epoch in KLN, while just one subset was used by CGMMN. Thus, KLN costs twice time as much as CGMMN. All the results of the above experiments validate this.

In order to evaluate the importance of kernel learning module in KLN, we set up three groups of controlled experiments, which corresponded to three commonly used mapping functions, i.e., identity mapping, PCA and AE respectively, to construct new kernel functions.

A new kernel $\mathcal{K}=k\circ h$, which composes of a mixture of Gaussian kernels and different embedding functions $h$, is considered in this section. When a kernel function is composed with the identity mapping, it means that the kernel matrix is calculated directly from the input data. For the PCA+CMMD and AE+CMMD methods, PCA and AE should be trained additionally, and then the PCA subspace and the encoder of AE are used as the embedding function separately. The critical difference between these methods and KLN is the network learning manner. KLN needs to learn an appropriate kernel function by simultaneously learning a classification model in an end-to-end manner. For a fair comparison, the same network architecture was used to implement these experiments on three datasets. Classification results evaluated on the test sets are shown in Table VI.

The classification accuracies of KLN on the three datasets are 99.61%, 98.44%, and 94.85%, respectively. As is shown above, these results are comparable to and even better than those of the state-of-the-art methods. In contrary, corresponding accuracies of using identity mapping are just 81.62%, 19.59% and 14.25%. Obviously, results of KLN are far better than those obtained from identity mapping. It indicates that KLN characterizes the similarity well. The classification accuracies of PCA+CMMD and AE+CMMD on the MNIST are 88.64% and 89.32%, respectively, while there are only about 30% classification accuracies on more complex datasets. It indicates the importance of learning a more effective kernel under complex data distributions.

In Section III-A, we have shown that distributions of $\langle\phi(\textbf{x}_{i}),\phi(\textbf{x}_{j})\rangle$ on samples with the same label or with different labels have no significant difference, thus, the kernel functions used in the CMMD criterion cannot effectively represent similarity. In this paper, a new approach is proposed to extract features via a different kernel utilization manner. Specifically, the new kernel $\mathcal{K}=k\circ h$, in which $k$ is a pre-specified characteristic kernel function and $h$ is the encoder of an auto-encoder network, is exploited in our method. It is expected that the distributions of $\langle\phi(\textbf{z}_{i}),\phi(\textbf{z}_{j})\rangle$ can be well distinguished.

We conducted experiments on three datasets, and then presented histograms of kernel values with the same or different labels in Figure 5. Taking the MNIST dataset for example, Figure 5(a) shows histogram for the samples with different labels while Figure 5(d) shows histogram for the samples with the same label. It can be seen that the kernel values, i.e., $\langle\phi(\textbf{z}_{i}),\phi(\textbf{z}_{j})\rangle$ of samples with different labels, are mainly concentrated in the range of 0.1-0.3, while those with the same label are concentrated between 0.7 and 1.0. There is a clear distinction between these two distributions. It indicates that the kernel values learned by KLN have better separating ability. Thus, the effectiveness of KLN in learning more expressive kernel functions is validated.

Results on SVHN and CIFAR-10 present a similar performance. In particular, for SVHN, the kernel values of samples with different labels are mainly concentrated on a sharp neighborhood of 0.0, while those with the same label are concentrated on the range of 0.6-0.9. For CIFAR-10, the kernel values of samples with different labels are mainly concentrated on a sharp neighborhood of 0.1, while those with the same label are concentrated on the range of 0.6-0.9.

We also conducted a group of contrastive experiments to display the histograms of kernel values computed by using the raw input features. The histograms are shown in Appendix C. Specifically, Figures 9, 10 and 11 display the histograms of kernel values on the MNIST, SVHN and CIFAR-10, respectively. All these results indicate that the kernel values, which are directly computed by using the raw features, have little discriminative power in practice.

This section evaluates KLN on semi-supervised classification tasks on MNIST, SVHN and CIFAR-10, and compares the performance with some state-of-the-art semi-supervised methods, including DGN [50], CatGAN [27], Ladder network [51], ADGM [52], SDGM [52], Improved-GAN [53], Triple-GAN [54].

Following the widely used settings, we used 100, 1000 and 4000 labeled samples on these datasets, respectively. The data preprocessing approaches and network architectures are the same with those used in the supervised learning scenario.

We repeated KLN over 10 runs with different random initializations and random subsets of labeled data, and calculated the mean error rates with the standard deviations following [53]. For a fair comparison, all the results of the baselines are cited from original papers. Table VII summarizes the quantitative results. KLN achieves the state-of-the-art predictive accuracies on all these datasets. Note that for a fair comparison with other algorithms, unlike the supervised learning scenario, the extra unlabeled data on SVHN were not used in training.

KLN has two important hyper-parameters, i.e., $\beta_{1}$ and $\beta_{2}$, in the semi-supervised learning algorithm. We further evaluated the susceptibility of KLN to these hyper-parameters. These two trade-off parameters were selected from sets $\{10,1,0.1,0.01\}$. Specifically, ablation experiments were conducted to understand the impact of different components in Equation (10), which corresponds to $\beta_{1}=0$ and $\beta_{2}=0$, respectively. Figure 6 shows the experiment results on SVHN and CIFAR-10. Parameter $\beta_{1}$ is the weight of AE regularization term, and the results shown in Figure 6(a) indicate that KLN is insensitive to the value of $\beta_{1}$. The same phenomenon exists in supervised classification tasks. It can be inferred that KLN usually learns the injective mapping with high probability while learning a more discriminative kernel function and parameterizing with deep neural networks. Parameter $\beta_{2}$ is the weight of the confidence loss and it could impact the performance of semi-supervised classification significantly. The comparison results are shown in Figure 6(b). The best choice of $\beta_{2}$ is 1. When the confidence loss was removed, i.e., $\beta_{2}=0$, the classification performance dropped but still had competitive results with [53]. It indicates the advantages of KLN and importance of the confidence loss.