Restricted Generative Projection for One-Class Classification and Anomaly Detection

The Deep SVDD proposed by [10] uses a neural network to learn a minimum-radius hypersphere to enclose the normal training data, i.e.,

where $\mathbf{c}\in\mathbb{R}^{d}$ is a predefined centroid and $\mathcal{W}=\{\mathbf{W}_{1},\ldots,\mathbf{W}_{L}\}$ denotes the parameters of the $L$-layer neural network $\phi$, and $\lambda$ is a regularization hyperparameter. In (1), to avoid model collapse, bias terms should not be used and activation functions should be bounded [10]. There are also a few variants of Deep SVDD proposed for semi-supervised one-class classification and anomaly detection [17, 19].

Both our method and Deep SVDD as well as its variants aim to project the normal training data into some space such that a decision boundary between normal data and unknown abnormal data can be found easily. However, the sum-of-square minimization in Deep SVDD and its variants only ensures that the projected data are sufficiently close to the centroid $\mathbf{c}$ in the sense of Euclidean distance and does guarantee that the data are sufficiently or evenly distributed in the hypersphere centered at $\mathbf{c}$. Thus, in the hypersphere, there could be holes or big empty regions without containing any normal data and hence it is not suitable to assume that the whole space enclosed by the hypersphere is completely a normal space. In other words, the optimal decision boundary between normal data and abnormal data is actually very different from the hypersphere. An intuitive example is shown in Fig.1. We see that there is a large empty space in the hypersphere learned by Deep SVDD. In contrast, the transformed data of our method are sufficiently distributed.

Our method is similar to but quite different from the variational autoencoder (VAE) [21]. Although our model is an autoencoder, the main goal is not to represent or generate data; instead, our model aims to convert distribution to find a reliable decision boundary for anomaly detection. More importantly, the latent distribution in VAE is often Gaussian and not bounded while the latent distribution in our model is more general and bounded, which is essential for anomaly detection. In addition, the optimizations of VAE and our method are also different: VAE involves KL-divergence while our method involves maximum mean discrepancy [39].

It is worth noting that similar to our method, Perera et al.[32] also considered bounded latent distribution in autoencoder for anomaly detection. They proposed to train a denoising autoencoder with a hyper-cube supported latent space, via adversarial training. The latent distribution and optimization are different from ours. In addition, the latent distributions of our method, such as uniform on hypersphere, are more compact than the multi-dimensional uniform latent distribution of their method.

Compared with the autoencoder based anomaly detection method NAE [40] that uses reconstruction error to normalize autoencoder, our method pays more attention to learning a mapping that can transform the unknown data distribution into a simple and compact target distribution. The ideas are orthogonal.

In statistics, maximum mean discrepancy (MMD)[39] is often used for Two-Sample test and its principle is to find a function that assumes different expectations on two different distributions:

where $p,q$ are probability distributions, $\mathcal{F}$ is a class of functions $f:\mathbb{X}\rightarrow\mathbb{R}$ and $\mathcal{H}$ denotes a reproducing kernel Hilbert space. Using the kernel trick, MMD can be represented as a simple loss function to measure the discrepancy between two distributions by finite samples, which is easy to apply to deep learning and can be efficiently trained by gradient descent. Based on the aforementioned advantages of MMD, Li et al.[41] proposed generative moment matching networks (GMMNs), which leads to a simpler optimization objective compared to the min-max optimization of GAN [37].

Although both our method and GMMNs [41] minimize the MMD between data distribution and prior distribution, our goal is not generating new data but detecting anomalies. In addition, we consider a few bounded target distributions and analyze their sampling properties. More importantly, our method has very competitive performance when compared with SOTA methods of anomaly detection and one-class classification.

Suppose we have a set of $m$-dimensional training data $\mathbf{X}=\{\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{n}\}$ drawn from an unknown bounded distribution $\mathcal{D}_{\mathbf{x}}$ and any samples drawn from $\mathcal{D}_{\mathbf{x}}$ are normal data. We want to train a model $\mathcal{M}$ on $\mathbf{X}$ to determine whether a test data $\mathbf{x}_{\text{new}}$ is drawn from $\mathcal{D}_{\mathbf{x}}$ or not. One may consider estimating the density function (denoted by ${p}_{\mathbf{x}}$) of $\mathcal{D}_{\mathbf{x}}$ using some techniques such as kernel density estimation [42]. Suppose the estimation $\hat{p}_{\mathbf{x}}$ is good enough, then one can determine whether $\mathbf{x}_{\text{new}}$ is normal or not according to the value of $\hat{p}_{\mathbf{x}}(\mathbf{x}_{\text{new}})$: if $\hat{p}_{\mathbf{x}}(\mathbf{x}_{\text{new}})$ is zero or close to zero, $\mathbf{x}_{\text{new}}$ is an abnormal data point; otherwise, $\mathbf{x}_{\text{new}}$ is a normal data point ¹¹1Here we assume that the distributions of normal data and abnormal data do not overlap. Otherwise, it is difficult to determine whether a single point is normal or not.. However, the dimensionality of the data is often high and hence it is very difficult to obtain a good estimation $\hat{p}_{\mathbf{x}}$.

We propose to learn a mapping $\mathcal{T}:\mathbb{R}^{m}\rightarrow\mathbb{R}^{d}$ to transform the unknown bounded distribution $\mathcal{D}_{\mathbf{x}}$ to a known distribution $\mathcal{D}_{\mathbf{z}}$ while there still exists a mapping $\mathcal{T}^{\prime}:\mathbb{R}^{d}\rightarrow\mathbb{R}^{m}$ that can recover $\mathcal{D}_{\mathbf{x}}$ from $\mathcal{D}_{\mathbf{z}}$ approximately. Let $p_{\mathbf{z}}$ be the density function of $\mathcal{D}_{\mathbf{z}}$. Then we can determine whether $\mathbf{x}_{\text{new}}$ is normal or not according to the value of $p_{\mathbf{z}}(\mathcal{T}(\mathbf{x}_{\text{new}}))$. To be more precise, we want to solve the following problem

where $\mathcal{M}(\cdot,\cdot)$ denotes some distance metric between two distributions and $\lambda$ is a trade-off parameter for the two terms. Note that if $\lambda=0$, $\mathcal{T}$ may convert any distribution to $\mathcal{D}_{\mathbf{z}}$ and lose the ability of distinguishing normal data and abnormal data. Based on the universal approximation theorems [43, 44] and substantial success of neural networks, we use deep neural networks (DNN) to model $\mathcal{T}$ and $\mathcal{T}^{\prime}$ respectively. Let $f_{\theta}$ and $g_{\phi}$ be two DNNs with parameters $\theta$ and $\phi$ respectively. We solve

where $f_{\theta}$ and $g_{\phi}$ serve as encoder and decoder respectively. However, problem (4) is intractable because $\mathcal{D}_{\mathbf{x}}$ is unknown and $\mathcal{D}_{f_{\theta}(\mathbf{x})}$, $\mathcal{D}_{g_{\phi}(f_{\theta}(\mathbf{x}))}$ cannot be computed analytically. Note that the samples of $\mathcal{D}_{\mathbf{x}}$ and $\mathcal{D}_{g_{\phi}(f_{\theta}(\mathbf{x}))}$ are given and paired. Then the second term in the objective of (4) can be replaced by sample reconstruction error such as $\tfrac{1}{n}\sum_{i=1}^{n}\|\mathbf{x}_{i}-g_{\phi}(f_{\theta}(\mathbf{x}_{i}))\|^{2}$. On the other hand, we can also sample from $\mathcal{D}_{f_{\theta}(\mathbf{x})}$ and $\mathcal{D}_{\mathbf{z}}$ easily but their samples are not paired. Hence, the metric $\mathcal{M}$ in the first term of the objective of (4) should be able to measure the distance between two distributions using their finite samples. To this end, we propose to use the kernel maximum mean discrepancy (MMD)[39] to measure the distance between $\mathcal{D}_{f_{\theta}(\mathbf{x})}$ and $\mathcal{D}_{\mathbf{z}}$. Its empirical estimate is

where $X=\{\mathbf{x}_{1},\dots,\mathbf{x}_{m}\}$ and $Y=\{\mathbf{y}_{1},\dots,\mathbf{y}_{n}\}$ are samples consisting of i.i.d observations drawn from $p$ and $q$, respectively. $k(\cdot,\cdot)$ denotes a kernel function, e.g., $k(\mathbf{x},\mathbf{y})=\exp(-\gamma\|\mathbf{x}-\mathbf{y}\|^{2})$, a Gaussian kernel.

Based on the above analysis, we obtain an approximation for (4) as

where $\mathbf{Z}_{\theta}=\{f_{\theta}(\mathbf{x}_{1}),f_{\theta}(\mathbf{x}_{2}),\ldots,f_{\theta}(\mathbf{x}_{n})\}$ and $\mathbf{Z}_{T}=\{\mathbf{z}_{i}:\mathbf{z}_{i}\sim\mathcal{D}_{\mathbf{z}},~{}i=1,\ldots,n\}$. The first term of the objective function in (6) makes $f_{\theta}$ learn the mapping $\mathcal{T}$ from data distribution $\mathcal{D}_{\mathbf{x}}$ to target distribution $\mathcal{D}_{\mathbf{z}}$ and the second term ensures that $f_{\theta}$ can preserve the main information of observations provided that $\lambda$ is sufficiently large.

Now we introduce four examples of simple and compact $\mathcal{D}_{\mathbf{z}}$ for (6). The four distributions are Gaussian in Hypersphere (GiHS), Uniform in Hypersphere (UiHS), Uniform between Hyperspheres (UbHS), and Uniform on Hypersphere (UoHS). Their 2-dimensional examples are visualized in Fig.2.

GiHS (Fig.2.a) is actually a truncated Gaussian. Suppose we want to draw $n$ samples from GiHS. A simple approach is drawing $(1+\rho)n$ samples from a standard $d$-dimensional Gaussian and discarding the $\rho n$ samples with larger $\ell_{2}$ norms. The maximum $\ell_{2}$ norm of the remaining $n$ points is the radius of the hypersphere. One may also use the inverse transform method of [45]. We have the following results.

Inequality (8) means a hypersphere of radius $r$ can include all the $n$ samples with a high probability if $r$ is sufficiently large. On the other hand, according to (7), if we expect to get $n$ samples in a hypersphere of radius $r$, we need to sample about $n/(1-\exp(-0.5\alpha))$ points from $\mathcal{N}(\mathbf{0},\mathbf{I}_{d})$. If $d$ is larger, we need to sample more points.

UiHS (Fig.2.b) is a hyperball in which all the samples are distributed uniformly. To sample from UiHS, we first need to sample from $\mathcal{U}(-r,r)^{d}$. Then we discard all the data points outsides the radius-$r$ hyperball centered at the origin. The following proposition (the proof is in Appendix) shows some probability result of sampling from a $d$-dimensional uniform distribution.

Inequality (10) means a hypersphere of radius $rt$ can include all the $n$ samples with probability at least $1-nd/(3t^{2})$. On the other hand, inequality (10) indicates that if we draw $n/(1-d/(3t^{2}))$ samples from $\mathcal{U}(-r,r)^{d}$, the expected number of samples falling into a hypersphere of radius $rt$ is at least $n$. Actually, sampling from UiHS is closely related to the Curse of Dimensionality and we need to sample a large number of points from $\mathcal{U}(-r,r)^{d}$ if $d$ is large because only a small volume of the hypercube is inside the hyperball. To be more precisely, letting $V_{\mathrm{hypercube}}$ be the volume of a hypercube with length $2r$ and $V_{\mathrm{hyperball}}$ be the volume of a hyperball with radius $r$, we have

where $\Gamma$ is the gamma function. Therefore, we need to draw $n/\eta$ samples from $\mathcal{U}(-r,r)^{d}$ to ensure that the expected number of samples included in the hyperball is $n$, where $\eta$ is small if $d$ is large.

UbHS (Fig.2.c) can be obtained via UiHS. We first sample from UiHS and then remove all samples included by a smaller hypersphere. Since the volume ratio of two hyperballs with radius $r$ and $r^{\prime}$is $(\frac{r}{r^{\prime}})^{d}$, where $r^{\prime}<r$, we need to draw $n/(1-(r^{\prime}/r)^{d})$ samples from UiHS to ensure that the expected number of samples between the two hyperspheres is $n$. Compared with GiHS and UiHS, UbHS is more compact and hence provides larger abnormal space for abnormal data to fall in.

UoHS (Fig.2.d) can be easily obtained via sampling from $\mathcal{N}(\mathbf{0},\mathbf{I}_{d})$. Specifically, for every $\mathbf{z}_{i}$ drawn from $\mathcal{N}(\mathbf{0},\mathbf{I}_{d})$, we normalize it as $\mathbf{z}_{i}\leftarrow{r\mathbf{z}_{i}}/{\|\mathbf{z}_{i}\|}$, where $r$ is the predefined radius of the hypersphere. UoHS is a special case of UbHS when $r^{\prime}=r$.

To quantify the compactness of the four target distributions, we define density $\rho$ as the number of data points in unit volume, i.e., $\rho=n/V$. Consequently, the densities of the four target distributions are reported in Table I. UoHS is more compact than UbHS as well as GiHS and UiHS, it should have better performance in anomaly detection. Indeed, our numerical results show that UoHS outperforms others in most cases.

In the test stage, we only use the trained $f_{\theta}^{*}$ to calculate anomaly scores. For a given test sample $\mathbf{x}_{\text{new}}$, we define anomaly score $s$ for each target distribution by

There are clear decision boundaries according to (12) and they can be regarded as ‘hard boundaries’ between normal samples and abnormal samples. However, these ‘hard boundaries’ only work in ideal cases where the projected data exactly match the target distributions. In real cases, due to the noise of data or the non-optimality of optimization, the projected data do not exactly match the target distributions. Therefore, we further propose a ‘soft boundary’ for calculating anomaly scores. Specifically, for a given test sample $\mathbf{x}_{\text{new}}$, we define anomaly score $s$ for all four target distributions as

where $\mathbf{x}_{i}$ denotes a single sample with index $i$ in the training data and $N_{k}$ denotes the index set of the $k$ nearest training (projected) samples to $f_{\theta}^{*}(\mathbf{x}_{\text{new}})$.

Empirically, in the experiments, we found that (13) has better performance than (12) in most cases. Table II, III, VI only report the results from (13). The comparison results between (12) and (13) are provided in Section V-E.

We call our method Restricted Generative Projection (RGP), which has four variants, denoted by RGP-GiHS, RGP-UiHS, RGP-UbHS, and RGP-UoHS respectively, though any bounded target distribution applies.

In the objective function of RGP defined by (6), the second term is the reconstruction error for $\mathbf{X}$, which is only a special example of approximation for the second term in the objective function of (4), i.e., $\mathcal{M}\left(\mathcal{D}_{g_{\phi}(f_{\theta}(\mathbf{x}))},\mathcal{D}_{\mathbf{x}}\right)$. Alternatively, we can use MMD to approximate $\mathcal{M}\left(\mathcal{D}_{g_{\phi}(f_{\theta}(\mathbf{x}))},\mathcal{D}_{\mathbf{x}}\right)$, which yields the following Double-MMD RGP:

Compared to the sum of squares reconstruction error used in (6), $\text{MMD}^{2}(g_{\phi}(\mathbf{Z}_{\theta}),\mathbf{X})$ is a weaker approximation for $\mathcal{M}\left(\mathcal{D}_{g_{\phi}(f_{\theta}(\mathbf{x}))},\mathcal{D}_{\mathbf{x}}\right)$, because it does not exploit the fact that the samples in $\mathbf{Z}_{\theta}$ and $\mathbf{X}$ are paired. Thus, the projection of Double-MMD RGP cannot preserve sufficient information of $\mathbf{X}$, which will reduce the detection accuracy. Indeed, as shown by the experimental results in Section V-F, our original RGP outperforms Double-MMD RGP.

Besides MMD, the optimal transport theory can also be used to construct a notion of distance between pairs of probability distributions. In particular, the Wasserstein distance [46], also known as “Earth Mover’s Distance”, has appealing theoretical properties and a very intuitive formulation

where $\mathbf{C}$ denotes a metric cost matrix and $\gamma*$ is the optimal transport plan. Finding the optimal transport plan $\gamma^{*}$ might appear to be a really hard problem. Especially, the computation cost of Wasserstein distance can quickly become prohibitive when the data dimension increases. In order to speed up the calculation of Wasserstein distance, Cuturi [47] proposed Sinkhorn distance that regularizes the optimal transport problem with an entropic penalty and uses Sinkhorn’s algorithm [48] to approximately calculate Wasserstein distance.

Now, if replacing the first term in (6) with the Sinkhorn distance[47], we can get a new optimization objective

where $\mathcal{M}(\mathbf{Z}_{\theta},\mathbf{Z}_{T})$ denotes the metric cost matrix between $\mathbf{Z}_{\theta}$ and $\mathbf{Z}_{T}$, $\epsilon$ is the coefficient of entropic regularization term, $\mathbf{a}$ and $\mathbf{b}$ are two probability vectors and satisfy $\mathbf{a}^{T}\mathbf{1}=1$ and $\mathbf{b}^{T}\mathbf{1}=1$ respectively. We call this method Sinkhorn RGP.

Compared to MMD, Sinkhorn distance is more effective in quantifying the difference between two distributions using their finite samples. Therefore, the Sinkhorn RGP usually has better performance than our original RGP (6), which will be shown by the experimental results in Section V-F.

We compare the proposed method with several state-of-the-art methods of anomaly detection on five tabular datasets and three widely-used image datasets for one-class classification. The datasets are detailed as follows.

• •

  Abalone²²2http://archive.ics.uci.edu/ml/datasets/Abalone[49] is a dataset of physical measurements of abalone to predict the age. It contains 1,920 instances with 8 attributes.

• •

  Arrhythmia³³3http://odds.cs.stonybrook.edu/arrhythmia-dataset/[50] is an ECG dataset. It was used to identify arrhythmic samples in five classes and contains 452 instances with 279 attributes.

• •

  Thyroid⁴⁴4http://odds.cs.stonybrook.edu/thyroid-disease-dataset/[50] is a hypothyroid disease dataset that contains 3,772 instances with 6 attributes.

• •

  KDD⁵⁵5https://kdd.ics.uci.edu/databases/kddcup99/[51] is the KDDCUP99 10 percent dataset from the UCI repository and contains 34 continuous attributes and 7 categorical attributes. The attack samples are regarded as normal data, and the non-attack samples are regarded as abnormal data.

• •

  KDDRev is derived from the KDDCUP99 10 percent dataset. The non-attack samples are regarded as normal data, and the attack samples are regarded as abnormal data.

• •

  MNIST⁶⁶6http://yann.lecun.com/exdb/mnist/[52] is a well-known dataset of handwritten digits and totally contains 70,000 grey-scale images in 10 classes from number 0-9.

• •

  Fashion-MNIST⁷⁷7https://www.kaggle.com/datasets/zalando-research/fashionmnist[53] contains 70,000 grey-scale fashion images (e.g. T-shirt and bag) in 10 classes.

• •

  CIFAR-10⁸⁸8https://www.cs.toronto.edu/ kriz/cifar.html[54] is a widely-used benchmark for image anomaly detection. It contains 60,000 color images in 10 classes.

We compare our method with three classic shallow models, four deep autoencoder based methods, three deep generative model based methods, and some latest anomaly detection methods.

• •

  Classic shallow models: local outlier factor (LOF)[55], one-class support vector machine (OC-SVM)[5], isolation forest (IF)[7].

• •

  Deep autoencoder based methods: denoising auto-encoder (DAE)[22], DCAE[56], E2E-AE, DAGMM[12], DCN [57].

• •

  Deep generative model based methods: AnoGAN[58], ADGAN[29], OCGAN [32].

• •

  Some latest AD methods: DeepSVDD[10], GOAD [59], DROCC [33], HRN [60], SCADN [35], NeuTraL AD [14], GOCC [26], PLAD [61], MOCCA [62].

In this section, we introduce the implementation details of the proposed method RGP and describe experimental settings for image and tabular datasets. Note that our method neither uses any abnormal data during the training process nor utilizes any pre-trained feature extractors.

For the five tabular datasets (Abalone, Arrhythmia, Thyroid, KDD, KDDRev), in our method, $f_{\theta}$ and $g_{\phi}$ are both MLPs. We follow the dataset preparation of  [12] to preprocess the tabular datasets for one-class classification task. The hyper-parameter $\lambda$ is set to 1.0 for the Abalone, Arrhythmia and Thyroid. For the KDD and KDDRev, $\lambda$ is set to 0.0001.

For the three image datasets (MNIST, Fashion-MNIST, CIFAR-10), in our method, $f_{\theta}$ and $g_{\phi}$ are both CNNs. Since the three image datasets contain 10 different classes, we conduct 10 independent one-class classification tasks on both datasets: one class is regarded as normal data and the remaining nine classes are regarded as abnormal data. In each task on MNIST, there are about 6,000 training samples and 10000 testing samples. In each task on CIFAR-10, there are 5,000 training samples and 10,000 testing samples. In each task on Fashion-MNIST, there are 6,000 training samples and 10,000 testing samples. The hyper-parameter $\lambda$ is chosen from $\{1.0,0.5,0.1,0.01,0.001,0.0001\}$ and varies for different classes.

In our method, regarding the radius $r$ of GiHS and UiHS, we first generate a large number (denoted by $N$) of samples from Gaussian or uniform, sort the samples according to their $\ell_{2}$ norms, and set $r$ to be the $pN$-th smallest $\ell_{2}$ norm, where $p=0.9$. For UbHS, we need to use the aforementioned method to determine an $r$ with $p=0.95$ and a $r^{\prime}$ with $p=0.05$. We see that $\{r,r^{\prime}\}$ are not related to the actual data, they are determined purely by the target distribution. In each iteration (mini-batch) of the optimization for all four target distributions, we resample $\mathbf{Z}_{T}$ according to $r$. For UoHS, we draw samples from Gaussian and normalize them to have unit $\ell_{2}$ norm, then they lie on a unit hypersphere uniformly. The procedure is repeated in each iteration (mini-batch) of the optimization. For hyper-parameter $k$ on the testing stage, we select $k=3$ for Thyroid, Arrhythmia, KDD, KDDRev, and select $k=5$ for Abalone dataset. For three image datasets, the hyper-parameter $k$ is chosen from $\{1,3,5,10\}$ and varies for different classes. We use Adam [63] as the optimizer in our method. For MNIST, Fashion-MNIST, CIFAR-10, Arrhythmia and KDD, the learning rate is set to $0.0001$. For Abalone, Thyroid and KDDRev, the learning rate is set to $0.001$. Table IV shows the detailed implementation settings of RGP on all datasets. All experiments were run on AMD EPYC CPU with 64 cores and with NVIDIA Tesla A100 GPU, CUDA 11.6.

To evaluate the performance of all methods, we follow the previous works such as  [10] and  [12] to use AUC (Area Under the ROC curve) for image datasets and F1-score for tabular datasets. Note that when conducting experiments on the tabular datasets, we found that most of the strong baselines, like DROCC [33], NeuTral AD [14], GOCC [26], used the F1-score and we just followed this convention. In our method, we get the threshold via simply calculating the dispersion of training data in latent space. Specifically, we first calculated the scores $s(\mathbf{X})$ on training data $\mathbf{X}$ using (12) or (13), and then sorted $s(\mathbf{X})$ in ascending order and set the threshold to be the $pN$-th smallest score, where $p$ is a probability varying for different datasets.

Tables II and III show the comparison results on Fahsion-MNIST and CIFAR-10 respectively. We have the following observations.

• •

  Firstly, in contrast to classic shallow methods such as OC-SVM [5] and IF [7], our RGP has significantly higher AUC scores on all classes of Fashion-MNIST and most classes of CIFAR-10. An interesting phenomenon is that most deep learning based methods have inferior performance compared to IF [7] on class ‘Sandal’ of Fashion-MNIST and IF [7] outperforms all deep learning based methods including ours on class ‘Deer’ of CIFAR-10.

• •

  Our methods outperformed the deep autoencoder based methods and generative model based methods in most cases and have competitive performance compared to the state-of-the-art in all cases.

• •

  RGP has superior performance on most classes of Fashion-MNIST and CIFAR-10 under the setting of UoHS (uniform distribution on hypersphere).

Table V shows the average performance on MNIST, Fashion-MNIST, and CIFAR-10 over all 10 classes to provide an overall comparison. We see that RGP achieves the best average AUC on Fashion-MNSIT and CIFAR-10 among all competitive methods. Four variants of RGP have relatively close average performance on all three image datasets. The experimental results of a single class on MNIST are reported in Appendix.

In Table VI, we report the F1-scores of our methods in comparison to ten baselines on the five tabular datasets. Our four variants of RGP significantly outperform all baseline methods on Arrhythmia, thyroid, and Abalone. Particularly, RGP-GiHS has $23.25\%$, $12.22\%$, and $19.58\%$ improvements on the three datasets in terms of F1-score compared to the runner-up, respectively. It is worth mentioning that Neutral AD [14] and GOCC [26] are both specially designed for non-image data but are outperformed by our methods in most cases. Compared with image datasets, the performance improvements of RGPs on the three tabular datasets are more significant. One possible reason is that, compared to image data, it is easier to convert tabular data to a compact target distribution. Furthermore, we also report the AUC scores on Abalone, Thyroid and Arrhythmia datasets and the results are provided in Appendix.

In addition to the quantitative results, we choose Thyroid (with 6 attributes) as an example and transform the data distribution to 2-dimensional target distributions, which are visualized in Figure 3. Plots (a), (b), (c), (d) in Figure 3 refer to GiHS, UiHS, UbHS, UoHS, respectively. The blue points, orange points, green points, and red points denote samples from target distribution, samples from training data, normal samples from test set, and abnormal samples from test set, respectively. For much clearer illustration, the left figure in each plot of Figure 3 shows all four kinds of instances and the right figure shows two kinds of instances including normal and abnormal samples from test set. We see that RGPs are effective to transform the data distribution to the restricted target distributions, though the transformed data do not exactly match the target distributions (it also demonstrates the necessity of using the ‘soft boundary’ defined by (13)).

We further explore the performance of two different anomaly scores. Specifically, we compare the ‘hard boundaries’ (12) and ‘soft boundary’ (13) as anomaly scores during the test stage on image datasets and tabular datasets. The results are showed in Figures 4, 5, 6. It can be observed that using ‘soft boundary’ (13) to calculate anomaly score has better performance than using ‘hard boundaries’ (12) on most classes of image and tabular datasets. Nevertheless, using ‘hard boundaries’ to calculate anomaly scores still achieves remarkable performance on some classes. For example, on the class ‘Ankle-boot’ of Fashion-MNIST and the class ‘Trunk’ of CIFAR-10, the best two results are both from RGPs using ‘hard boundaries’ (12) to calculate anomaly score.

We use Double-MMD RGP (14) to conduct experiments and the results are reported in Table VII, VIII. On image datasets, we just consider the target distribution UoHS (Uniform on HyperSphere) for simplicity. On tabular datasets, we conduct experiments on the proposed four different target distributions.

From the experimental results of Table VII, VIII, we found that Double-MMD RGP and original RGP have similar performance on the three tabular datasets, whereas on image datasets including Fashion-MNIST and CIFAR-10, the performance has apparent gap in spite of a large range of adjustment of $\lambda\in\{10.0,5.0,1.0,0.5,0.1,0.01\}$ for Double-MMD RGP (14). Note that Table VII reports the average AUC(%) on all classes of Fahion-MNIST and CIFAR-10, the results on single class are provided in Appendix.

For the phenomenon, we consider that the tabular datasets in our implementation have fewer features (no more than 279) than the image datasets and second term of (14) is a much weaker constraint for preserving data information than that of (6). As a consequence, Double-MMD RGP (14) is able to preserve the enough key information on the tabular data but loses a lot of important information on the image data than original RGP (6). Meanwhile, we know that the generalization error of MMD for high-dimensional samples or distribution is often larger than that for low-dimensional samples or distribution. To ensure that MMD is able to accurately measure the distance between two high-dimensional distributions, the sample sizes should be sufficiently large.

We use Sinkhorn RGP (16) to conduct experiments on Abalone, Arrhythmia, and Thyroid datasets and the results are reported in Table VIII. In all implementations, $\epsilon$ is set to $0.01$ and the a, b are uniform. In keeping with our expectation, the performance of Sinkhorn RGP (16) is similar to or better than the original RGP (6) for all four objective distributions, whereas the time cost of Sinkhorn RGP (16) is much higher. We do not experiment with Sinkhorn RGP for the image dataset since the time cost is too higher.