Partial Resampling of Imbalanced Data

Imbalanced data refers to the skewed distribution of class labels in data. It occurs in a wide range of fields including medicine [30], cybersecurity [27], fraud detection [13], and others. One of the common approaches to dealing with imbalanced data is through sampling. In most cases, sampling is applied until a fully balanced dataset - where each class has an equal number of samples - is achieved. The balanced dataset is then passed on to the training phase [4]. However, it is not necessarily true that the training set must be fully balanced to achieve the optimal results.

The traditional argument for balancing the data is that classifiers tend to focus on the majority class samples at the expense of the minority class. For instance, given a training set with a 1/100 class imbalance ratio, most classifiers would end up classifying all the samples as negative (majority). Therefore, it is argued that class balancing is required to force classifiers to learn the minority samples. On the other hand, by artificially balancing the data we are distorting the reality. Ideally, the class ratios in the training and testing set should be the same to maintain the fidelity of the process. Sampling the training set while keeping the test set at the original class ratio goes against this philosophy of congruent train and test sets.

Excessive sampling - where the sampled class ratio is far greater than the original class ratio - may lead to artificial bias towards the minority class. In most cases, the increased accuracy on the minority class is accompanied with a drastic decrease in accuracy on the majority class. While in some instances the accuracy on minority class is more important than the majority class - as in the case of fraud detection or medical diagnostics - it is worthwhile to take into account the accuracy on the majority class. Therefore, the optimal sampling ratio of the minority to majority samples must be chosen with great care.

Increasing the number of minority points through ovesampling provides the classifier with more data to learn the representation of the minority class. On the other hand, since the new minority points are not generated from the true distribution, the increase in the number of sampled points may lead to model misspecification [8]. At some stage, the issues related to model misspecification outweight the benefits of learning the class representations. The optimal sampling ratio occurs at the stage when the model accuracy begins to deteriorate.

A compromise between full sampling and the original class ratio is partial sampling. Partial sampling implies sampling the data up to a specified class ratio (minority/majority) that is between the original class ratio and the 50/50 ratio. To obtain the optimal partial sampling ratio a grid search procedure can be utilized. Concretely, we can measure the performance of a classifier on the training set for different partial sampling ratios using cross-validation. After identifying and training with the optimal partial sampling ratio, the classifier is tested on the holdout set.

In this paper, we conduct a large-scale, systematic study of the effects of partial sampling of imbalanced data on the performance of classifiers. In particular, we evaluate the accuracy of classifiers that are trained on datasets sampled over a range of class ratios. Our study is distinguished from other similar studies by its breadth. While other studies employ only a few datasets and sampling methods [4, 25, 28], we consider 20 imbalanced datasets (Table 1) and 10 sampling methods (Table 2). For each dataset, we sample the original data with different ratios and evaluate the performance of the trained classifier. We use random forest (RF) and support vector machines (SVM) as the base classifiers in our numerical experiments. Given the scale of the present study, its results provide valuable insights into the effects of sampling ratio.

The results of the numerical experiments reveal two key insights: i) the optimal sampling ratio of the minority to majority samples is almost always less than 1 and often in the range of 0.7-0.8, and ii) the optimal ratio depends on the particular dataset. The results hold across an array of popular sampling techniques. Based on the outcomes of our empirical study, we recommend selecting the optimal sampling ratio using a grid search procedure. Furthermore, the default sampling ratio is recommended to be 0.75. We hope that our study helps researchers to better understand the effects of different sampling ratios and expedite the selection of the optimal ratio.

The paper is structured as follows. Section 2 discuses the current literature regarding imbalanced data and sampling methods. Section 3 presents the methodology employed in the study including the experimental setup, datasets, sampling methods, classifiers, and metrics. Section 4 contains the results and analysis of the numerical experiments. Section 5 concludes the paper with summary remarks and discussion of future research.

Data sampling is a widely used pre-processing step in many machine learning pipelines. Despite some skepticism about the usage of sampling [22] most researchers agree about its efficacy. There exists a variety of sampling techniques in the literature. A number of sampling techniques operate by generating new minority points between the existing neighboring points. For instance, in [5] the new points are generated using a uniform distribution, while in [16] the new points are generated via a gamma distribution. In a localized approach, more points are generated in the regions closer to the majority class points [6, 32]. New samples can also be generated by first estimating the underlying distribution of the minority points. Given a learned distribution, the new points are then obtained from the density distribution. To learn the distribution of the points both statistical and deep learning models are used . In the statistical approach kernel density estimation is applied [17], while in the deep learning approach generative adversarial networks are often employed [26, 31].

Partial sampling and its effects on classification accuracy have been studied by several authors. The authors in [29] found that the optimal sampling ratio depends on the classifier performance metric. While the full sampling ratio is preferred under the AUC, the original sampling ratio is preferred based on the classification accuracy. In [1], the authors concluded that the optimal class distribution is not necessarily achieved at the fully balanced distribution. Similarly, in [4] the authors studied the effects of different imbalance ratios in the context of image classification and convolutional neural networks. Their experiments showed that class imbalance is generally detrimental to classifier performance. They also found oversampling to be the most effective method for combating class imbalance. The effects of class imbalance were also studied in [28] who used undersampling to balance the data. The authors found that classification precision and recall are the lowest at class ratio 0.5.

The authors of [10] considered two factors affecting the performance of sampling methods: the employed classifier and the degree of imbalance. Their results indicate that the best sampling method depends of the class imbalance ratio. They concluded that for datasets having low or moderate class imbalance ratio, oversampling outperforms undersampling using local classifiers such as kNN. However, some undersampling methods outperform over-sampling when using global learning classifiers such as neural networks. The authors in [3] used random oversampling and undersampling to evaluate the efficacy of different sampling ratios. The results show no significant difference in classification accuracy for different sampling ratios with only a small decrease around $r=1$. In [25], the authors seek to find the optimal sampling ratio for intrusion detection data (KDD99). The authors use SMOTE to test the performance of various sampling ratios showing that the class ratio of $r=10$ provides the optimal results. A large scale study of 85 different oversampling methods was done in [19]. Although the primary goal of the study was to identify the best sampling method, the authors used a range of sampling ratios to train the classifier. Unfortunately, the study did not indicate the performance of the sampling methods at different ratios. A broader comparison of data-driven, algorithmic, and hybrid approaches was conducted in [9].

As discussed above, there are a number of studies that explore the issue of the optimal sampling ratio. However, the majority of the studies are either outdated or limited in scope. Our study provides an up-to-date evaluation of the popular sampling algorithms based on a large scale experimental database.

In this section, we discuss the experimental setup, datasets, sampling algorithms, and classifiers used in our study.

In this section, we present and discuss the results of our numerical experiments aimed at understanding the effects of different sampling ratios. The results are based on evaluation of 20 imbalanced datasets, 10 sampling methods, and 2 classifiers. As described in Section 3.4, the performance of the sampling strategies and ratios is measured based on the accuracy of the classifier that is trained on the resampled train set. We focus on the results obtained based on the RF classifier. The results based on the SVM classifier are in line with those of RF and are summarized in the corresponding tables and figures.

We begin by taking a close look at the effects of different sampling ratios based on the SMOTE sampling algorithm. In Table 3, we provide the accuracy results, as measured on the test set, for the RF classifier that is trained on a partially balanced set using the SMOTE algorithm. The first column in the table, labeled orig, shows the accuracy on the original imbalanced data. The last column in the table, labeled max, shows the maximum accuracy over all sampling ratios. Thus, for the ecoli dataset the maximum accuracy is 0.8473 which is achieved at the sampling ratio $r=0.8$. Similarly, for the abalone dataset, the maximum accuracy is 0.6698 which is achieved at the sampling ratio $r=0.9$. Note that almost all the values in the max column are greater than the corresponding values in the column $r=1$ which indicates that maximum accuracy is rarely achieved with full resampling. Indeed, it can be seen from Table 3 that the maximum accuracy is often achieved at lower sampling ratios between $r=0.2$ and $r=0.9$. It can also be seen from Table 3 that the accuracy generally increases as the sampling ratio increases from $r=0.2$ to $r=1$. This pattern holds across all the datasets in the table. Furthermore, partial resampling, at any class ratio, produces higher accuracy than the original imbalanced data.

In Figure 1, we present the average accuracy at each sampling ratio calculated over all the datasets based on Table 3. It can be seen from Figure 1 that the greatest average accuracy occurs at $r=0.8$. In particular, the mean accuracy at $r=0.8$ is greater than the accuracy at $r=1$. We also observe that the accuracy generally increases as the class ratio is increased via sampling.

We extend the above analysis that was done for the SMOTE algorithm to other sampling methods. Figure 2 shows the mean accuracy for each sampling method contained in Table 2. The mean is taken over all datasets in Table 1. As discussed earlier, the SMOTE algorithm achieves the best accuracy at ratio $r=0.8$ (Figure 2a). For the ADASYN method, the best result is obtained at $r=0.7$ and $r=1$ (Figure 2b). For the ROS, NearMiss, Border, PolySM, and Lee methods the highest accuracy is achieved at sampling ratios of $r=0.8,r=0.3,r=0.7,r=0.7$, and $r=0.8$ respectively. The results show that on average the best performance of these algorithms is achieved at $r<1$.

We also note that in most cases, sampling has a positive effect on accuracy. As shown in Figure 2, the accuracy at orig is lower than at all other ratios. This pattern holds for all the sampling methods. Based on Figure 2, we conclude, that while data sampling has a significant positive effect on the accuracy of the classifier, the optimal sampling ratio is generally below the full resampling. In fact, our experiments show that the best sampling ratio of minority to majority classes is between $r=0.7$ and $r=0.8$.

In Table 4, we provide yet another perspective on the performance of different sampling ratios. In particular, we present the average accuracy for each combination of dataset and sampling ratio, where for each pair (dataset, sampling ratio) the average is taken over all the sampling methods in Table 2. As shown in Table 4, the highest mean accuracy - taken over all the sampling methods - for the ecoli dataset is 0.8174 which is achieved at the sampling ratio $r=0.7$. Similarly, for the car_eval_34 dataset, the highest average accuracy is 0.9667 which is achieved at $r=0.5$. As can be seen from Table 4, in most cases, the highest average accuracy occurs at $r<1$. Only in 4 out of 20 datasets the best accuracy is achieved at $r=1$ . We also observe that the original sampling ratio yields the lowest average accuracy. Based on Table 4, we conclude that while resampling increases the average balanced accuracy for each dataset, the best results are often obtained at resampling ratios less than $r=1$. The practical implications of the accuracy results are that

1. i.

   data sampling always improves the accuracy of the classifier,

2. ii.

   the best sampling ratio is often between $r=0.7$ and $r=0.8$, and

3. iii.

   to obtain the best results it is necessary to perform a grid search evaluation of various sampling ratios.

The results in Table 4 show that there is little relation between the optimal sampling ratio and dataset characteristics. In particular, the original imbalance ratio of a dataset plays little role in determining its optimal sampling ratio. For instance, the ecoli and wine_quality datasets which have the lowest and highest imbalance ratios (Table 1) respectively, both achieve the optimal performance at the sampling ratio $r=0.7$ (Table 4). Similarly, the number of features in the dataset has little effect on the optimal sampling ratio. For instance, the datasets car_eval_34 and isolet both have the same optimal sampling ratio $r=0.5$, but the number of features is 21 and 617, respectively.

The only meaningful relationship between the optimal sampling ratio and the properties of the dataset that can be derived from the results in Table 4 is with respect to the number of samples. In particular, datasets with larger number of samples tend to have smaller optimal ratio. To derive this relationship, we consider the six datasets with the optimal ratio $r\leq 0.5$ : optical_digits, pen_digits, sick_euthroid, car_eval_34, isolet, and thyroid_sick. The average number of samples in the datasets with $r\leq 0.5$ is 5512, while the average number of samples in the datasets with $r\geq 0.6$ is 2705. Thus, the average number of samples in datasets with $r\leq 0.5$ is twice larger than in the datasets with $r\geq 0.6$. We postulate that given a large number of samples in the original dataset, there is less need for additional minority samples to learn the patterns within data. Conversely, a dataset with relatively few samples requires more minority samples for classifier to properly learn its patterns.

The mean F1-macro scores are presented in Table 5. As above, the average is taken over all the sampling methods. The results indicate that the optimal F1-macro values are achieved at lower sampling ratios. In particular, the highest F1-macro often occurs at $r=0.2$ and $r=0.3$. It indicates that to obtain the best performance in terms of precision and recall we need to apply a low sampling ratio. We also note that full sampling at $r=1$ produces suboptimal results which supports our findings based on the balanced accuracy metric.

The results in Tables 4 and 5 show that the choice of the performance metric affects the optimal sampling ratio for imbalanced data. If the goal is to achieve the maximum accuracy then sampling ratios $r=0.7$ and $r=0.8$ often produce the best performance. On the other hand, sampling ratios $r=0.2$ and $r=0.3$ produce the best F1-macro values. These values can be used as default parameters in sampling methods. In addition, regardless of the performance metric, we observe that sampling usually improves the performance and full sampling rarely produces the best results. In general, we strongly recommend to perform a grid search over a range of sampling ratios using cross-validation to identify the optimal sampling ratio.

The results using the SVM classifier are generally in line with above results obtained with the RF classifier. The SVM-based experiments show that sampling improves the accuracy of the classifier. The optimal sampling ratio often occurs at $r<1$. The main difference between the SVM and RF-based results is in the higher accuracy of the former. It appears that the SVM classifier is better suited for imbalanced data when used in conjunction with data sampling. The details of the SVM-based experiments are supplied in Table 6 and Figure 3.

As shown in Table 6, there is no discernible relation between the optimal sampling ratio and the original imbalance ratio. Similarly, the number of features in the dataset does not seem to have much impact on the value of the optimal $r$. The only dataset property that appears to affect the optimal ratio is the number of samples. Datasets with more samples, on average, achieve the optimal performance at lower values of $r$. We conjecture that large datasets require fewer additional minority samples to have enough samples points for effective pattern recognition.

As mentioned in the introduction, increasing the number of minority points through sampling has a dual effect on classifier performance. On one hand, sampling provides the classifier with more data to learn the representation of the minority class. On the other hand, since the new points are not generated from the true distribution, they may lead to model misspecification. As the the number of sampled points increases, the issues related to misspecification of the model begin to outweight the benefits of learning the minority class parameters. Our numerical experiments show that the point of inflection often occurs between $r=0.7$ and $r=0.8$. At this optimal ratio, we obtain enough information to produce statistically robust parameter estimates and avoid overly skewing the minority class distribution.

Another important factor in the analysis of different sampling ratios is the classifier training times. The number of minority points in the dataset increases as the sampling ratio increases. For instance, given an imbalanced dataset with 1/10 original class ratio, the number of minority points increases 10-fold if we choose $r=1$ sampling ratio. The larger dataset leads to longer training times. The classifier training times are approximately linearly related to the size of the dataset. The training times for the RF classifier with different sampling ratios is supplied in Figure 4.

In summary, our study reveals that while artificially increasing the class ratio in imbalanced data improves classification accuracy, full sampling ($r=1$) rarely produces the optimal results. Although the sampling ratio in the range $r=0.7$ to $r=0.8$ often produces the best accuracy and can be set as the default parameter value, it is best to conduct a grid search over a range of class ratios to identify the optimal ratio. Our study supports the previous findings in the literature about the general benefits of data sampling. However, the optimal sampling ratio identified in our experiments differs from the other studies.

The optimal sampling ratio depends on a number of factors including the data, sampling method, and performance metric. Given the same dataset, different sampling methods or performance measures can lead to different optimal ratios. We find little relation between the optimal ratio and dataset characteristics. In particular, dataset properties such as the original imbalance ratio and the number of features play a trivial role in determining the optimal ratio. The only factor that is found to affect the optimal ratio is the number of samples - datasets with large number of samples tend to have lower optimal ratio. We conclude that the optimal sampling ratio depends uniquely on the distribution of the data points in the feature space for a given dataset.

Despite the existence of a large number sampling techniques for imbalanced data very few studies are dedicated to the issue of the optimal sampling ratio. In this paper we fill the gap in the literature by conducting a large-scale study of the impact of sampling ratio on the classification accuracy. Our empirical study is based on 20 datasets and 10 sampling methods. The results show that i) sampling is generally a beneficial preprocessing step, ii) the optimal sampling ratio is between 0.7 and 0.8 albeit the exact value depends on the dataset, iii) full resampling ($r=1$) is rarely the best option, and iv) there is an inverse relation between the number of samples and the optimal ratio. The present study enhances our understanding of the effects of the sampling ratio and provides insights into selecting the optimal ratio.

Our experiments show that while factors such the original imbalance ratio and the number of features do not play a significant role in determining the optimal ratio, the number of samples in the dataset may have a tangible impact. It is possible that there exists a more complex interplay between the data characteristics and the optimal ratio. Therefore, a future in-depth investigation into the relationship between the intrinsic data properties and the optimal sampling ratio is warranted.