Managing dataset shift by adversarial validation for credit scoring

The term dataset shift was first introduced by J. Quionero-Candela et al [12]. In other studies, it has also been called the concept shift [21], changes of classification [22], fracture points [23] or contrast mining in classification learning [24]. Such inconsistent terminology confounds the discussion of this important problem. In this paper, the term dataset shift is represented for the situation where the data used to train the classifier and the environment where the classifier is deployed do not follow the same distribution, which means $P_{train}(y,x)\neq P_{test}(y,x)$.

A classification problem consists of three parts, namely a set of features or covariates $x$, a target variable $y$, and joint distribution $P(y,x)$. There are two kinds of classification problems according to [25]:

• i.

  $X\rightarrow Y$ problems, where the class label is causally determined by the values of the covariates. For example, user behavior in credit scoring, represented by the covariable space $X$, determines the class label $Y$: good or bad users.

• ii.

  $Y\rightarrow X$ problems, where the class label causally determines the values of the covariates. Medical diagnostics typically fall into this category, where the disease, which is modeled as the class label $Y$, determines the symptoms, represented as covariates $X$ in the machine learning task.

There are three different types of shift for the above two kinds of problems, depending on which probabilities change or not:

• i.

  Covariate shift represents the situation where training and testing data distribution may differ arbitrarily, but there is only one unknown target conditional class distribution. In other words, it appears only in $X\rightarrow Y$ problems, and is mathematically defined as the case where $P_{train}(y\mid x)=P_{test}(y\mid x)$ and $P_{train}(x)\neq P_{test}(x)$.

• ii.

  Prior probability shift is the reverse case of covariate shift. It appears only in $Y\rightarrow X$ problems, and is defined as the case where $P_{train}(x\mid y)=P_{test}(x\mid y)$ and $P_{train}(y)\neq P_{test}(y)$.

• iii.

  Concept shift happens when the relationship between the input and class variables changes, which is defined as

  • $\blacksquare$

    $P_{train}(y\mid x)\neq P_{test}(y\mid x)$ and $P_{train}(x)=P_{test}(x)$ in $X\rightarrow Y$ problems.

  • $\blacksquare$

    $P_{train}(x\mid y)\neq P_{test}(x\mid y)$ and $P_{train}(y)=P_{test}(y)$ in $Y\rightarrow X$ problems.

Both the covariate shift and the concept shift can occur in the credit scoring scenario. One example of the covariate shift is that as the economy grows and per capita wages rise, the old model of using a person’s income to determine his or her credit rating may slowly fail. For concept Shift, a common example is a sudden change in the macroeconomic environment or loan policy. The risk level may change after the loan interest rate change even for the same lender.

There are many possible reasons for dataset shift, the two most important of which are as follows:

Reason 1. Sample Selection Bias is a systematic defect in the data collection or labeling process, where the training set is obtained by a biased method and this non-uniform selection will cause the training set to fail to represent the real sample space. For example, in social science research, students at the researcher’s university or former research participants are more likely to be surveyed than other populations. These “easy” groups may be overrepresented in the training samples, while “difficult” groups (e.g., prisoners) may be underrepresented or completely excluded.

Joaquin et al. [26] give a mathematical definition of sample selection bias:

• i.

  $P_{train}=P(s=1\mid y,x)P(x)$ and $P_{test}=P(y\mid x)P(x)$ in $X\rightarrow Y$ problems.

• ii.

  $P_{train}=P(s=1\mid x,y)P(y)$ and $P_{test}=P(x\mid y)P(y)$ in $Y\rightarrow X$ problems.

where $s$ is a binary selection variable that decides whether an instance is included in the training samples ($s=1$) or rejected from it ($s=0$).

The problem of operating under sample selection bias has received substantially more attention in other domains than it has in the machine learning community [25]. In the credit scoring literature it goes by the name of reject inference, because potential credit applicants who are rejected under the previous model are not available to train future models [27, 28].

Reason 2. Non-stationary Environments is often caused by temporal or spatial changes, and is very common in real-world applications. Depending on the classification problem’s type, non-stationary environments can lead to different kinds of shift:

• i.

  In $X\rightarrow Y$ problems, a non-stationary environment could create changes in either $P(x)$ or $P(y\mid x)$, generating covariate shift or concept shift, respectively.

• ii.

  In $Y\rightarrow X$ problems, it could generate prior probability shift with a change in $P(y)$ or concept shift with a change in $P(x\mid y)$.

Non-stationary environments often appears in adversarial classification problems such as network intrusion detection [29], spam detection [30, 31] and fraud detection [32, 33]. The presence of an adversary trying to bypass the existing classifier introduces any possible dataset shift, and the bias can change dynamically. This kind of problem is getting more and more attention in the machine learning community [34, 35].

A common approach to cope with the dataset shift problem in production systems is monitoring and redeploying, i.e., continuous monitoring the model performance and retraining the model with new data when it degrades but does not completely fail [13]. Throughout the process, the model goes through a cycle of built, deployed, deprecated, and rebuilt, and this cyclical strategy is important to maintain an accurate and stable system. In the field of credit scoring, Kolmogorov-Smirnov (KS) and Population Stability Index (PSI) are often used as indicators to monitor model performance [36]. Although this approach can reduce the impact of dataset shift issues to some extent, it still encounters many challenges in practical use. For example, model predictions usually take some time to show their effects, and there is no immediate observable feedback. To solve this problem, some scholars have proposed intelligent machine learning methods to directly deal with dataset shift, which are divided into two main categories, algorithmic-level and data-level solutions.

Algorithmic-level solutions propose models that are robust in the presence of dataset shift. There are many types of algorithmic-level solutions, of which the Bayesian model is a common one, but they are mostly designed for regression rather than classification problems, such as the stock price [37] or temperature prediction [38]. In the scenario of online learning or data stream mining, many algorithmic-level solutions have also emerged. For example, Kolter et al. [39] proposed dynamic weighted majority (DWM) for concept drift that dynamically creates and removes weighted experts in response to changes in performance. In addition to that, Mello et al. [40] combine the twin support vector machine with a fuzzy membership function (FBTWSVM) to deal with large datasets and learn from data streams. In the field of credit scoring, Maldonado et al. [17] proposed a general version of hinge loss function applying aggregation operators to deal with dataset shift via fuzzy logic.

For data-level solutions, a common approach is the forgetting mechanism [13, 21], which removes outdated samples by sliding time windows. There is a tradeoff between reactivity and robustness to noise of the system, and the more abrupt forgetting is, the faster the reactivity, but also the higher the risk is of capturing noise. Another common data-level solution is the novel variant of cross-validation [41, 42, 43], which has the advantage that they do not require timestamps because they are designed to evaluate changes in the data distribution.

Adversarial validation is a method to detect dataset shift, which requires training a binary classifier and judging whether the sample is from the training set or the testing set. If the performance of the classifier is close to the result of random guess, it means that it is difficult to distinguish whether a sample is from the training set or the testing set, that is, the distribution of both is relatively consistent. On the contrary, if the classifier performance is much better than the result of random guess, it indicates that the sample distribution of the training set and testing set is inconsistent.

It should be noted that the adversarial validation mentioned in this article is not the same as the adversarial training introduced in Generative Adversarial Networks (GAN) [44]. GAN’s framework corresponds to a minimax two-player game, it simultaneously trains two models: a generative model $G$ that captures the data distribution, and a discriminative model $D$ that estimates the probability that a sample came from the training data rather than $G$. GAN is becoming more and more popular in the field of content generation [45, 46]. In the field of credit scoring, GAN has been used to solve the sample imbalance problem [47].

Specifically, the process of adversarial validation can be divided into three steps:

1. i.

   For the original dataset $\left\{train\_X,train\_y,val\_X,val\_y\right\}$, remove the old label column $\left\{train\_y,val\_y\right\}$, and add a new label column $\left\{train\_y_{s},val\_y_{s}\right\}$ that marks the source of the data, labeling the samples in the training set as 0 $(i.e.\ train\_y_{s}=0)$ and the samples in the testing set as 1 $(i.e.\ train\_y_{s}=1)$.

2. ii.

   Train the classifier on the dataset $\left\{train\_X,train\_y_{s},val\_X,val\_y_{s}\right\}$ with the newly labeled column. The output of the classifier is the probability that the sample belongs to the testing set. In this paper, 5-fold cross-validation is used.

3. iii.

   Observe the results of the classifier. If the model performance is poor (AUC score is close to 0.5), it indicates that the classifier cannot distinguish whether the samples come from the training set or the testing set. It can be judged that the distribution of the training set and testing set in the original data is consistent. On the contrary, if the model performance is high (AUC score is close to 1), it indicates that the classifier can easily distinguish sample sources. It can be determined that the training set and the testing set are very different in data distribution.

In addition to detecting inconsistent data distribution, the results of adversarial validation can further help balance training and testing sets, and improve the model performance on testing set. This appears in some data competition, but there is still relatively little published research. Researchers at Uber [20] proposed an adversarial validation based approach that addresses the issue of concept drift that commonly exists in large-scale user targeting automation systems, where the distribution of complex user features keeps evolving. They use the feature importance obtained from adversarial validation to filter the most inconsistently distributed features sequentially. However, there is a trade-off for this method between the improvement of generalization performance and losing information by dropping features from the model. The proposed method in this paper can improve the generalization performance of the model to the testing set without losing information.

The method proposed based on adversarial validation in this paper can not only judge whether the dataset distribution is consistent, but also further balance the training and testing sets. Specifically, gradient boosting decision tree (GBDT) [48] is used as the classifier for both the adversarial validation and the credit scoring phases, which is a boosting model that continuously reduces the residuals during the training process. It was chosen because the GBDT-based methods have been proven to be very effective in recent research in the field of credit scoring [49, 50]. Apart from that, GBDT has a very efficient modern library⁶⁶6https://github.com/microsoft/LightGBM, LightGBM[51], which has won many data competitions and will be used in this paper to build the model.

There are many ways to use the results of adversarial validation, and a total of three schemes are proposed in this paper.

Method 1. Use the adversarial validation results as sample weights added to the training process.

The adversarial validation probability result of the training set sample represents its similarity to the testing set, which can be used to determine how much to involve in the actual modeling process. If a sample is more closely distributed to the testing set data, it can be given a higher weight during training. Conversely, lower weights are assigned. Modern GBDT libraries can specify the weight of each sample during training, which is convenient to control the contribution of each sample. Specifically, set the “weight” parameter in LightGBM Dataset API to change the weight of each instance.

Method 2. Use only the data with the top-ranked adversarial validation results for 5-fold cross-validation.

Apart from reducing the weight of samples inconsistent with the distribution of the testing set, they can also be removed from the training process. In particular, the training data can be divided into two parts by screening the results of adversarial validation probability according to a certain threshold value. The samples that are more consistent with the testing set distribution are called $data\_X_{a}=\left\{train\_X_{a},val\_X_{a}\right\}$, and the remaining samples are called $data\_X_{b}$. $P_{data\_X_{a}}\approx P_{test\_X}\neq P_{data\_X_{b}}$, and only $data\_X_{a}$ is reserved for 5 fold cross-validation.

As a result, model evaluation metrics on the validation data should have similar results on the testing data, which means that if the model works well on the validation data, it should also work well on the testing data. Specifically, The LightGBM parameter “early_stopping_round” is set during training, and the model stops training when the AUC of the validation set $val\_X_{a}$ still doesn’t grow in a certain number of iterations. The model with optimal results is retained and used to predict the testing set.

Method 3. All data are used for training, and only the data with the top-ranked adversarial validation results are used for validation.

Although Method 2 alleviates the problem of inconsistent data distribution between training and testing sets, it has defects in data utilization. Only $data\_X_{a}$ data was used in the whole training process, and $data\_X_{b}$ data was wasted. To solve this problem, $data\_X_{b}$ is added to the training data of each fold in the process of 5-fold cross-validation to assist training, but it does not participate in the validation. This not only maintains the consistency of validation and testing results, but also makes full use of all data.

The dataset used in this paper comes from Lending Club, whose timestamp information helps divide training and testing sets in strict chronological order. The data includes samples from 2018M1⁷⁷7The representation of time in this paper consists of two parts, M is preceded by the year and followed by the month, for example, 2018M1 represents January 2018. to 2020M9 over a period of 33 months. The original dataset contains 1,160,066 samples with 151 features and “loan_status” as the target variable. As shown in Table 1, “loan_status” has 8 states, “Current”, “Fully Paid”, “Charged Off”, “Late (31-120 days)”, “In Grace Period”, “Issued”, “Late (16-30 days)”, and “Default”. Referring to the practice of previous papers [3, 52], only the samples with “Charged off” and “Fully Paid” status are taken as positive and negative samples respectively, all loans with other status have been filtered out as their final status are unknown. This results in an unbalanced dataset containing 276,685 samples with a positive sample ratio of 21.93%.

The number of samples in each month is shown in Fig 2. In chronological order, the data of 18 months from 2018M1 to 2019M6 are taken as the training set, which contains 247,276 samples in total. The data from 2019M7 to 2020M9 were taken as the testing set, including 29,409 samples.

Many features in the original data have a high proportion of missing values, and some of the remaining variables are unavailable to an investor before deciding to fund the loan. As a result, including the target variable “loan_status”, 25 variables are actually used, and each field description is shown in table 2.

The specific processing methods for some variables are as follows:

• i.

  The original “emp_length” variable contains $<$1 year, 1 year … 9 years, 10+ years for a total of 11 category variables, which are transformed into integer variables from 0 to 10 for better usage;

• ii.

  The original FICO score provides two values, “fico_score_low” and “fico_score_high”, and the difference between the two is always 4 points, which leads to data redundancy and is replaced by the average of the two;

• iii.

  Log transformation is performed on numerical variables with power-law distribution, including “annual_inc” and “revol_bal”;

• iv.

  For category variables, including “sub_grade”, “home_ownership”, “verification_status”, “initial_list_status”, “purpose”, “addr_state”, “application_type”, we employ the LightGBM built-in support.

After the above processing, the actual number of variables input to the model is 23. Table 3 shows the statistical characteristics of the numerical variables.

The same settings are used for LightGBM hyperparameters in both the adversarial validation and credit scoring phases. Specifically, “num_boost_round” is set to 50000, which is a relatively large value. However, by setting the “early_stopping_rounds” parameter, the model will stop training if the validation data’s AUC doesn’t improve in the last 200 rounds. It not only ensures sufficient training, but also prevents over-fitting. More detailed hyperparameter settings are shown in Table 4.

As shown in Fig 3, in order to demonstrate the effect of adversarial validation, a total of 5 sets of experiments are set up. The testing set data of each experiment are all from 2019M7 to 2020M9, while the training and validation sets are divided in different ways.

Experimental Set 1. Select data according to chronological order for 5-fold cross-validation.

A fixed time point is set and only data after that time point are used for the 5-fold cross-validation. Specifically, the starting month of the cross-validation data was selected from 2018M1 to 2019M6 for a total of 18 experiments. In these experiments, the starting point selection 2018M1, which uses all data for training can be used as a benchmark.

Experimental Set 2. Select data according to chronological order, training data before, validation data after.

Different from the 5 fold cross-validation used in Experimental Set 1, experiment set 2 only used data closer to the testing set for validation. Specifically, there are three choices of data time ranges, which are to use all data, 2018M6 and subsequent data, 2018M12 and subsequent data. These three groups of data will be divided into training and validation data according to the sequence of timeline, for a total of 17+11+5=33 experiments.

Experimental Set 3. Use the adversarial validation results as sample weights added to the training process.

The output probability of the adversarial validation classifier to the samples, i.e., the similarity with the testing set samples, is directly used as the weight in the training process. All data will be used in this one experiment, and no need to be divided by time or quantile.

Experimental Set 4. Use only the data with the top-ranked adversarial validation results for 5-fold cross-validation.

The ranking of the output probability of the sample by the adversarial validation classifier is regarded as the criterion of data partitioning. Data that is more inconsistent with the distribution of the testing set will be discarded, and the remaining data will be subjected to 5-fold cross-validation. Specifically, 0%, 5% … 90%, 95% of the data were discarded, respectively, for a total of 20 experiments.

Experimental Set 5. All data are used for training, and only the data with the top-ranked adversarial validation results are used for validation.

Although Experimental Set 4 constructs a dataset that is more consistent with the testing set distribution for cross-validation, it brings about the problem of wasting a lot of data. This can also harm the model performance, especially when the amount of discarded data is large. Experimental Set 5 is further optimized based on Experimental Set 4, by adding these discarded data that are inconsistent with the testing set distribution into the cross-validation training data, but does not participate in the validation. This not only addresses the problem of dataset shift, but also makes full use of all data. Similarly, the data are also divided according to the output probability ranking of the samples by the adversarial validation classifier, and the number of experiments is the same as in Experimental Set 4, with a total of 20 experiments.

There is a total of 18+33+1+20+20=92 experiments in 5 sets, comparing a variety of models’ performance on the testing set, which is trained on data divided by time or by adversarial validation results.

Analysis of Experimental Set 1.

Fig 4 shows the results of Experimental Set 1. With the increase of the starting month of the selected data, the AUC of the validation set shows a trend of gradual decline, and the decline speed increases with the decrease of the selected data. For the testing set, the AUC fluctuated steadily when the selected data started before 2019M2, and only after that did it start to show a significant decreasing trend. Although adding data with a long distance from the testing set in the training improves the offline validation effect of the model, it does not improve the performance of the test data. This confirms that the problem of dataset shift does exist, and the distribution of the data accumulated in the past is not consistent with the data that needs to be predicted in the recent stage, which leads to the problem of model generalization. The 5-fold cross-validation with all the data could be used as a benchmark, i.e., the starting month of the selected data was set to 2018M1. At this time, the AUC of the testing set was 0.7237. Among all the experiments of Experimental Set 1, the selection of 2018M2 and later data for cross-validation is the best, with the testing set AUC reaching 0.7256.

Analysis of Experimental Set 2.

Fig 5 shows the results of Experimental Set 2. From 5 (a) to (c), regardless of the selection range of training validation data starts from 2018M1, 2018M7, or 2019M1, with the gradual increase of data divided into the training set, the AUC of validation set and testing set both show an increasing trend, and the gap between them gradually decreases. This indicates that postponing the time point of splitting the training and validation sets can improve both the performance of the model and the consistency of the validation and testing set results.

Fig 5 (d) integrates the testing set AUC results of the three sub-experiments, and the optimal results that can be achieved by all three are relatively close. The best result occurs when using the data from 2018M7 to 2019M5 as the training set and the 2019M6 data as the validation set, the testing set AUC reaches 0.7220. This result is lower than using all the original data directly for 5-fold cross-validation, since the 2019M6 data, which is closest to the testing set distribution, is only involved in the validation and not in training.

Analysis of Experimental Set 3.

Experimental Set 3 uses all the data for the 5-fold cross-validation, and the prediction results of the adversarial validation are added as sample weights in the training process, so there is only 1 experiment. The AUC result of adversarial validation is 0.9681, much higher than 0.5, which indicates that the classifier can easily distinguish the training data from the test data, and the two are indeed inconsistent in distribution.

In Experimental Set 3, the final AUC obtained for the validation and testing set are 0.7149 and 0.7202, respectively, which is rather inferior to the benchmark of using the full data directly for the 5-fold cross-validation. This indicates that changing only the sample weights without changing the sample selection does not effectively solve the dataset bias problem.

Analysis of Experimental Set 4.

According to the results of adversarial validation, Experimental Set 4 selected the data with the top specific quantile for 5-fold cross-validation. Fig 6 (a) showed the results, with the increase of the probability quantile of adversarial validation, the training data also increased, and the testing set AUC showed a gradual rise at first, and then a relatively stable fluctuation. When the quantile selection is 75%, the maximum testing set AUC can reach is 0.7315, which is an improvement compared with the previous three experiments. When the quantile is small, the performance of the model is greatly reduced due to the lack of available training data, exposing the drawback that this method fails to make full use of all data.

Analysis of Experimental Set 5.

In Experimental Set 4, the quantile selection has a greater impact on the experimental results. Experimental Set 5 adds the data with the lower quantile ranking to each fold of the training data, but does not participate in the validation phase. This ensures the ability of the model to generalize to the testing set while making full use of all the data. Fig 6 (b) illustrates the results, the testing set AUC fluctuation of Experimental Set 5 is relatively more stable. When the quantile is chosen to be 40%, the maximum testing set AUC value can reach 0.7327, which is also the highest score among all experiments.

In Experimental Sets 4 and 5, the data partitioning quantiles that achieve the optimal accuracy are 75% and 40%, respectively. Fig 7 shows the distribution of these data in each month. It can be found that the closer to the testing set date, the higher the percentage of the selected data in the month’s original data. This is also in line with the phenomenon that the distribution of data at similar times is more consistent.

Comprehensive analysis of all experiments.

Fig 8 shows the comprehensive comparison of the results of all 5 experimental sets, which can be summarized as follows:

• i.

  The dataset shift problem does exist in credit scoring, and dividing the training and validation sets in different ways will indeed affect the model performance on the testing set.

• ii.

  Compared with other partitioning or data utilization methods, using adversarial validation to select data more consistent with testing set distribution for cross-validation can improve the optimal accuracy. However, attention should be paid to the amount of data selected, too little data may degrade the model performance.

• iii.

  To further improve the model performance, data that are not consistent with the testing set distribution can be added to the training process as well, but not involved in the validation. This will allow the optimal accuracy of the testing set to be further improved, and the results obtained by choosing different quantiles are more stable.