SORTAD: Self-Supervised Optimized Random Transformations for Anomaly Detection in Tabular Data

Anomaly detection [1] is the process of detecting samples that do not align with the majority distribution of the studied population. In real applications of fault or fraud detection [2, 3], these anomalies can point to defects or fraudulent samples. Therefore, there is an ever growing need for improving anomaly detection algorithms. Anomalies are often not well represented in the training data, even if they are labeled, due to a small number of samples from each possible anomaly source. In these cases it is not possible to train supervised methods to classify normal vs. abnormal samples. Unsupervised methods that only learn the normal samples distributions address these challenges and are preferred in anomaly detection.

Many anomaly detection methods have been proposed. Most methods rely on statistical properties like the density of the samples [4], or finding a general space that encapsulates the normal samples as in [5, 6]. Another promising method is self-supervision. Self-supervision is an unsupervised learning paradigm, in which a supervised model solves one or more secondary tasks, based on constructed labels that are derived from the features themselves. The trained model is then used to tackle the primary task. An intuitive example is using Auto-Encoders for anomaly detection [7, 8]. In this case the primary task is finding anomalies, while the secondary task is learning to compress the data, using the features themselves as the labels. The compression reconstruction error of each sample is used as the sample’s anomaly score. The samples with the highest scores are then considered anomalies. Self-supervision doesn’t rely on the sample labels, and usually works best when using only the normal samples for training, and was shown to improve model robustness [9].

Recent advancements in self-supervised learning for anomaly detection, including GEOM [10], GOAD [11], $SLA^{2}P$ [12], $E^{3}Outlier$ [13] and RecTrans [14], all used one new approach with the same main assumption. They randomly transform the samples, and then predict for each transformed sample which transformation was applied to it. The assumption is that anomalous samples will display some irregular behavior following transformations, while "normal" samples will behave similarly following such transformations. Thus, a classifier that is trained to detect from transformed data samples their corresponding transformation will have better accuracy and higher confidence for normal samples, as opposed to abnormal samples.

This approach was first introduced in GEOM [10] and in [15] for images. GEOM uses the images natural symmetry for rotations to randomly rotate images, and then predicts the applied rotation with a neural network. It then scores each sample using Dirichlet distribution approximation normality score.

Anomaly detection based on random transformations in tabular data faces some obstacles that don’t exist in images. The first obstacle, which we name "Unrelated", is that tabular data has no underlying relation between the features. Therefore, there are no symmetries that can be considered independent of differences between samples, like rotations in images as in GEOM [10]; [15]. These symmetries were used in the selection of transformations in order to avoid mixing the transformations outputs.

The second obstacle, "Non-Sparse", is that for tabular data the samples tend to be concentrated in small parts of the feature space (a relatively small L2 distance between them). This is clearly the case when normalization is used, making samples typically reside within the unit ball. This creates a need for transformations to be very different from one another, to enable a classifier to distinguish between the transformations applied. Otherwise transformations will map samples to the same space. This is unlike the case of images, where mapping different images to the exact same image by rotation is unlikely, even if they contain the same object, but with different backgrounds for example.

The third obstacle in tabular data, "Easy Detection", is that anomalies can be separated from normal samples by a certain way that amplifies the transformations detect-ability. This creates a problem for methods that rely on the difficulty of identifying the transformation as a measure for anomaly (e.g. RecTrans [14]). An intuitive explanation is with the simple transformations of $t_{1}(x)=x^{3}$ and $t_{2}(x)=x+0.3$. For values larger than 1, $t_{1}$ will usually produce larger values than $t_{2}$. If most normal values are around 1.2, for example, then the classifier may learn to classify high values to $t_{1}$. This may cause a problem if abnormal values are much higher. Even though these values are abnormal, the classifier will be very successful in classifying them, since it learned to classify high values to $t_{1}$. These cases will cause the summation scoring method in RecTrans, which sums just the correct transformations predictions, to fail.

In the tabular data domain both GOAD [11] and RecTrans [14] use random transformations. GOAD uses affine transformations, and then feeds the results to a neural network [16] trained with triplet loss [17]. This increases the outer variance, i.e., the mean distance between samples transformed by different transformations, while decreasing the inner variance, the variance of the resulting samples with the same transformation. It then proceeds to score each sample with the sum of the normalized Gaussian probabilities predicted for each transformation. RecTrans, on the other hand, uses random polynomial transformations. It then predicts the applied transformation with a neural network. It scores each sample with the sum of the correct predictions of the sample (the Summation Scoring method). The randomization of RecTrans introduces high variance in the results. GOAD is less susceptible to this variance since it optimizes the transformations using the neural network.

To address the obstacles described above, we propose SORTAD, a novel algorithm that is tailor maid to solve these challenges. Unlike the previous methods, it selects from the random transformations generated the ideal transformations that would help the classification process. The selected transformations are those that better separate the transformed data from the data transformed by the previously selected transformations. In addition, SORTAD proposes a scoring function that is especially sensitive to the changes in the transformation classifiers predictions encountered in tabular data, as described below. We show that SORTAD outperforms previous methods on commonly used anomaly detection benchmarks.

We summarize the new contributions in SORTAD compared to previous work:

• •

  We present a method to select transformations from randomly generated transformations, to improve overall anomaly detection and reduce performance variance, while using less transformations (leading to less computational overhead). We thus address the Unrelated, Non-Sparse and Easy Detection obstacles described above.

• •

  We present a modified version of the scoring method used in GEOM, specifically designed for anomaly detection in tabular data. This addresses the Easy Detection challenge.

• •

  We present a modified version of the reversible polynomial transformation, introduced in RecTrans, which is more numerically stable.

The remainder of this paper is structured as follows: Section 2 provides a detailed description of the proposed algorithm. Section 3 provides a description of the model evaluation method used which is specifically designed for anomaly detection in production environments, that doesn’t rely on knowing the amount of anomalies in the dataset. We also highlight its benefits over F1 score. Section 4 specifies the experimental setting and presents and discusses the results. Lastly, Section 5 is an overall discussion.

The underlying assumption of our method, as in previous methods mentioned in the Introduction, is that anomalous samples will display irregular behavior following transformations, while normal samples will behave similarly following such transformations.

Other than the empirical evidence in the results previous methods, we believe the main assumption has theoretical intuition as well. Under the assumptions that the features present are enough to detect anomalies, a reasonable assumption, there is an underlying reason that these samples are anomalies. Meaning in some underlying way they are different than the normal samples. Camparing the behavior of normal samples to newly seen samples should show which samples are anomalous.

Based on this assumption, SORTAD randomly transforms the data using reversible polynomial transformations, introduced in RecTrans [14]. These reversible polynomial transformations are as follows:

$$y_{1}=x_{p_{1}}+G(y_{p_{2}})$$

(1)

$$y_{2}=x_{p_{2}}+F(x_{p_{1}})$$

(2)

Where $x\in\mathbb{R}^{p}$ is the input; $p_{1}$ and $p_{2}$ are randomly non intersecting feature lists of the same length upholding $|p|=|p_{1}|+|p_{2}|$; $x_{p_{1}}$ and $x_{p_{2}}$ are $p_{1}$ and $p_{2}$ corresponding sample feature values; $(G)_{i}:\mathbb{R}\xrightarrow{}\mathbb{R}$ and $(F)_{i}:\mathbb{R}\xrightarrow{}\mathbb{R}$ are polynomial transformations, which are applied element wise to each feature value $i$. The final transformed sample is a concatenation of $y_{1},y_{2}$. In cases where there is an uneven amount of features, a zeros vector is added to the feature space. The reversibility of the transformations is proven in [14], thus, guarantees that with an optimal classifier it is possible to classify each transformation by reversing the input.

Similarly to RecTrans, $F$ and $G$ use different polynomial bases, Chebyshev and Legendre to increase their uniqueness. In addition, applied one after another, limited the amount by a hyper-parameter. These transformations help address the Non-Sparse and Unrelated obstacles.

Applying several transformations in a row introduces two crucial problems: underflow and overflow of the feature space. Overflow happens when a value gets too high for the current computer representation to handle, making the algorithm crash. Underflow happens when two values of very different magnitudes are summed, causing the smaller value to not be represented in the resulting number. This causes several samples to have the same prediction. Both cases are devastating for the detection performance. To address this problem two main elements were added to the transformations:

• •

  Divide factor – transformations were added a constant divider that is $10^{(d-h)}$, where $d$ is the degree of the polynomial, and $h$ is a hyper-parameter (usually 2). This change helps prevent overflow, especially in data sets normalized with robust scaling [18]. Robust scaling is a scaling technique widely used for anomaly detection data sets, which scales the data using the median value and the IQR of the feature, thus making it less sensitive to outliers (anomalies) in the data than standard scaling.

• •

  Non-Constant – Chebyshev and Legendre polynomials have in every second base element a constant addition, when represented in the ordinary polynomial basis. The constant value causes underflow because it is added to the low values normally outputted from the rest of the polynomial equation. Thus, choosing basis elements that do not have a constant element decreases underflow.

To help solve the Unrelated, Non-Sparse and Easy Detection obstacles, we propose improving the transformation creation process by selecting the best transformation out of randomly generated transformations. Improving the transformation selection enables the use of less transformations. It improves the classifier’s ability to distinguish between them and retain the transformations that best separate the anomalies behavior from the normal samples behavior. This boosts detection performance, in addition to reducing the computational resources needed.

To this end, in each step SORTAD randomly creates $k$ temporary transformations, assigns each transformation a score based on the resulting structure of the transformed samples, and then chooses the best transformation based on this score until $M$ transformations are chosen. The scoring function Eq. 3 is proposed. The main idea is to help the transformation classification algorithm by forcing the transformed samples to be farther away than the results of the previous transformations outputs center of masses (higher outer variance). In addition, the resulting samples should display lower inner variance (mean distance from the center) allowing further separation from previous transformations outputs. Since both the center of mass and the inner variance are guided by normal samples, anomalous sample’s, if present, affect are limited. More importantly, these metrics best captures the normal samples behavior and not the anomalous. Since we assume that anomalous samples will display irregular behavior under transformations they will probably be farther away from the center of mass and therefore close to previous transformations centers. Causing the classifier to give poor predictions to them, allowing the anomaly score to be high for anomaly samples. Moreover, since it strives to find transformations with low inner variance it facilitate our assumptions that normal samples will behave the same under transformations, since we choose the transformations that they do. Thus, choosing low inner variance helps strengthen our main assumption and facilitates anomaly detection. Our proposed score is a modification of the triplet loss objective [17]. The modified score is described in Eq. 3.

$$tscore_{m}=\beta\sum_{i}|x_{i}-c_{m}|-(1-\beta)\sum_{i}min_{m^{\prime}\neq m}|x_{i}-c_{m^{\prime}}|$$

(3)

Where $c_{m}$ is the center of transformation $m$ output, $m^{\prime}$ is an index running on all previous transformations, $\beta$ is a hyper-parameter that is a weighting factor for the inner and outer variance’s importance. We use L1 norm, which is more robust to outliers. Fig. 1 shows a qualitative example of two transformation and their corresponding scores. It shows that generating bad transformation can hurt the transformation process by generating transformations who’s outputs are too similar to former transformations. Thus, not allowing the classifier to be able to classify normal samples correctly. Avoiding this, is one of SORTAD’s biggest advantages.

After $M$ transformations are selected the outputs of the transformations and their corresponding labels are inputs for the transformation classifier. The transformation classifier receives as input a transformed sample and predicts the applied transformation. An overview of the model is given in Fig. 2.

After the training process the prediction of the classifier is used for scoring. The Summation Score, in which each sample’s score is the sum of the correct predictions of the sample, was found to be prone to Easy Detection and gives anomalies high scores. This is the main reason for the fast drop in performance RecTrans demonstrated in Fig. 4. The Dirichlet probability scoring method proposed in GEOM, described in Eq.4, helps address this problem. It considers when a probability score of the transformation is different from what was seen for the training samples, and uses the predictions for other transformations. This normality score computes the probability of a sample’s transformation prediction scores in the Dirichlet distribution space approximated from the classifier’s prediction on the training samples. The final score for each sample is the sum of probabilities of the classifier’s prediction for each transformation on the approximated Dirichlet distribution. This also helps solve the problem where predictions of the applied transformation for anomalous samples and normal samples receive the same score, but the scores on the wrong transformations are significantly different. Thus, for the Summation Score, both anomalies and normal samples will receive the same score, but in the Dirichlet probability score, anomalies will receive a lower score. Additionally, by considering all the transformations predictions for given transformations, more information is saved and used to determine which sample is anomalous.

$$n(x)=\sum_{m=0}n_{m}(x)=\sum_{m=0}\sum_{j=0}(\alpha_{m_{j}}-1)log(y(T_{m}(x))_{j})$$

(4)

Where $\alpha$ is the concentration parameter for the Dirichlet probability approximated from the training samples, $T$ is transformation function, $m$ is an index for all transformations, $j$ is an index for all transformation prediction scores. This means that $m_{j}$ is the index for transformation $j$ when transformation $m$ is applied. Normal samples scored a higher value than anomalies, as described below.

For the Dirichlet probability score, Easy Detection persists. This is in contrast to the intuition that the scores for anomalies are abnormally high, therefore they should get low probabilities. The reason for this occurrence lies in the Dirichlet probability score, which is not sensitive enough for the change. The $\alpha$ parameter of the Dirichlet distribution, in Easy Detection transformations will have a very high value for the transformation and very low values for the rest. This causes the distribution to be centered at the edge, where the correct transformation gets a prediction score close to 1. Fig. 3 shows the Dirichlet distribution for different $\alpha$ values showing how the distribution gets skewed to the vertex. As a result, every sample that gets a near perfect prediction score, gets a very high normality score regardless of how high. In addition, very low prediction values have a very large log value in absolute terms, causing the normality scores to be governed by the predictions for the wrong transformations, because they occur with $\alpha_{m_{j}}<1$. Since in these cases anomalies get a higher correct prediction score, their wrong prediction score is lower, therefore, their normality score is higher. They are so high that the modification needed is to use the distance from the mean score on the training samples. The final modified normality score is in Eq. 5.

$$n_{m}(x)=\begin{cases}n(x),&\min\limits_{j}(\alpha_{j})\geq 1\\ |n(x)-\overline{n}(x)|,&\min\limits_{j}(\alpha_{j})<1\land n(x)\geq 0\\ |n(x)-\overline{n}(x)|*R,&\min\limits_{j}(\alpha_{j})<1\land n(x)<0\end{cases}$$

(5)

Where $\overline{n}(x)$ is the mean of Eq. 4 on the training data. The last option in Eq. 5 is to account for the information loss while using the distance from the mean score. Since with Easy Detection all samples are expected to have positive results, samples that still have negative results are more likely to be anomalies. Therefore, a multiplication factor is added to incorporate this information into the score. For all the test results, R=3 was used. The higher the score the more "normal" the sample is.

Current metrics for anomaly detection use the well known F1-score with the anomaly percentage in the training data. Thus, they rely on having the similar anomalies percentage in both training, validation and test sets. In addition, for this evaluation metric to be accurate for production purposes, it is necessary for the algorithm to have a high enough number of samples. Otherwise, it is unclear where the split between normal and anomalous occurs. That is, if the anomaly percentage is 1.45% and only 100 samples are in the predicted set, it is unclear whether 2 or 1 samples should be considered anomalies. These assumptions don’t apply in many real anomaly detection applications, such as fault or fraud detection, which use a pre-computed threshold in the anomalies scorer to label samples as normal or abnormal. This threshold is computed offline on the train or validation set to achieve the desired balance between false negative and false positive errors.

To evaluate the models performance in these systems, we propose to use the the recall ratio vs a random classifier in addition to ROCAUC, and Recall metrics at predefined thresholds. The “VS Random” metric, is the ratio between the recall in the actual fraction of samples labeled as anomalous in the evaluated set using a threshold value determined by the training set and the recall a random classifier would have achieved in that same actual fraction. Unstable thresholds between train, validation and test set, pose a problem in production. One algorithm’s recall can be significantly higher than the other only because it’s learned thresholds are very different than those that should have been used, if learned on the different set. For example, if the 5% threshold score is used, it may classify 20% of the newly tested samples as anomalies. In this case, the "actual alert percentage" is 20%. This usually results in a higher recall but a higher percentage alerted, which is not favorable. The recall score without adjusting to the actual alert percentage will be called the "Non Adjusted Recall". In these cases the VS Random score will be much lower. In addition, algorithms that have a close actual alert percentage to the desired percentage, show that the underlying distribution of the training data was learned, which gives a higher confidence of its stability and generalization. We emphasize that the VS Random metric is the most important metric for anomaly detection, since it best describes the amount of target samples eliminated. For example a VS Random score of 2 means that the algorithm eliminated 2 times more anomalies than a random classifier would. For imbalance data sets it is common to use two thresholds the real desired threshold and a larger threshold to evaluate the stability of the model. In our evaluation 3% and 10% were used as the main and stability percentages respectively.

Datasets: All data sets where randomly split to train and test sets, with data sets containing more than 30 anomalies and enough samples split into train, validation and test sets. The split was done while preserving the anomaly rate between sets. When all three sets exist, the algorithms are trained on the training set, then hyper-parameters are tuned on the validation set and the training set and then evaluated on the test set. Hyper parameters tuned on the training and validation set show higher stability and are less likely to overfit. In cases were no validation set exists, the hyper-parameter tuning was done only on the training set. Evaluation was done on 10 predetermined data sets: Mammography, SMTP, Forest Cover, Thyroid, Shuttle, Pendigits, Arrihythmia, Annthyroid, Vowels [19], and Credit card [20]. The data sets are described in the Appendix. These data sets are commonly used to evaluate many anomaly detection algorithm and are from various domains and have a diverse number of samples and features. We focus on tabular data with lower dimension as opposed to image data sets since, we believe they better represent the challenges in anomaly detection in tabular data. All the data sets were scaled with robust scaling.

Baseline Methods: Classical baseline methods consist of: Isolation Forest [21], One-Class SVM [5], Elliptic Envelope [22] and LOF [4]. More recently developed algorithms, including state-of-the-art methods are: GOAD [11], Anomaly Detection for Tabular Data with Internal Contrastive Learning, (ICL) [23], COPOD [24], and RecTrans [14]. For the classical baseline methods the default hyper-parameters in the python package scikit-learn [25] were used. For ICL the non hyper-parameterized function was used with 5 different random seeds. For GOAD the hyperparameter search included 18 different hyper-parameter sets, consisting of a grid search of all hyper-parameters used in the paper with 1 epoch, including 3 random seeds, amounting to 54 different executions in total. For RecTrans 16 hyper-parameter sets were tested, 8 of which were later used for SORTAD, and 4 were from the paper.Additionally, 3 random seeds were used, a total of 36 execution sets. The full hyper-parameters for each method are summarized in the Appendix.

We propose a novel framework for anomaly detection named SORTAD: Self-supervised Optimized Random Transformation for Anomaly Detection. SORTAD’s main innovation is managing to select the best transformations out of randomly generated transformations, thus increasing detection performance and stability while decreasing computational time. In addition, SORTAD utilizes a modified scoring method, specifically designed for tabular data, which was shown to produce better and more stable results while facing the Easy Detection problems encountered in tabular data. To conclude, SORTAD achieved state-of-the-art results on multiple anomaly detection data sets and in overall results on 10 anomaly detection data sets. SORTAD was tested using a validation set for hyper-parameters tuning, in cases where this is not possible from Fig. 5 and 4 we can conclude that allowing SORTAD to use more transformations and to choose from more temporary transformation will usually yield better results.

We would like to express our sincere gratitude to Dr. Amitai Armon for his invaluable contributions to this research. His insightful comments and thorough review significantly enhanced the quality and clarity of this paper. Dr. Armon’s expertise and thoughtful suggestions were instrumental in shaping the final manuscript. We are truly thankful for his time, dedication, and constructive feedback.