DNN2LR: Automatic Feature Crossing for Credit Scoring

It is a direct way to explicitly search for useful cross feature fields in a candidate set. However, the candidate set for searching is usually inevitably large, which leads to low efficiency of learning feature crossing, especially on credit scoring datasets with hundreds of input feature fields. Most of these works focus on generating second-order cross features (Lou et al., 2013; Chapelle et al., 2015; Katz et al., 2016; Nargesian et al., 2017; Kaul et al., 2017). GA²M (Lou et al., 2013) extends GAM (Lou et al., 2012) with second-order cross features, which is hard to optimize when the candidate set is large. In (Chapelle et al., 2015), the authors try to generate and select second-order cross feature fields according to Conditional Mutual Information (CMI). However, once the mutual information of an original feature field is high, the generated cross feature fields containing it will also have high conditional mutual information. AutoLearn (Kaul et al., 2017) selects cross feature fields by using regularized regression models, where it is hard to learn a regression model for all the cross feature fields on a wide dataset. Some works (Katz et al., 2016; Nargesian et al., 2017) takes meta-learning into consideration for feature generation. However, the effectiveness of meta-learning in these methods remains a question and requires extremely large amount of data. There are also some methods incorporating genetic algorithm (Tran et al., 2016) and reinforcement learning (Khurana et al., 2018; Chen et al., 2019) for finding feature combinations. However, with genetic algorithm or reinforcement learning, we still have a large space to explore. Moreover, as introduced in (Khurana et al., 2018), it also requires large amount of data for the training of reinforcement learning. To tackle with above problems, AutoCross (Luo et al., 2019) presents an approximate framework to search in the large candidate set of both second- and higher-order cross feature fields more efficiently. AutoCross uniformly divides the whole dataset into at least $\sum\nolimits_{i=0}^{\left\lceil{\mathop{\log}\nolimits_{2}t}\right\rceil-1}{\mathop{2}\nolimits^{i}}$ batches, where $t$ is the size of candidate set in AutoCross. Then AutoCross iteratively trains a field-wise LR model on part of the data to validate the contribution of a cross feature field. Though AutoCross achieves state-of-the-art performances, it still faces low searching efficiency on some wide datasets. Moreover, when the candidate set is large, data for training the field-wise LR model of each candidate cross feature field will be too little to produce reliable results. Therefore, AutoCross may face problems in both effectiveness and efficiency on credit scoring datasets with hundreds of feature fields.

Besides above methods explicitly searching for cross features, some works (He et al., 2014; Shi et al., 2020) utilize tree ensemble models (Chen and Guestrin, 2016; Ke et al., 2017) for generating cross features. In (He et al., 2014), each tree in GBDT (Gradient Boosting Decision Tree) (Ke et al., 2017) corresponds to a cross feature field, and each leaf node in a tree corresponds to a cross feature. As trees in GBDT may be deep, the generated cross features may be with too high orders, so that they are hard for human to understand. And cross features in the same cross feature field usually share different meanings. Besides, Factorization Machine (FM) (Rendle, 2010) has been a successful way to explicitly capture second-order feature interactions. To overcome the problem that conventional FM can only model second-order feature interactions Higher-Order FM (HOFM) (Blondel et al., 2016) and Interaction Machine (IM) (Yu et al., 2020) propose to model higher-order feature interactions in the FM architecture in linear time. The calculation of feature interactions in FM is somehow product-based similarity. However, it is hard for these factorization-based methods to capture all kinds of feature interactions in various real-world application scenarios. Another drawback of FM is that, it models all feature interactions of specific-order, most of which are not useful for making predictions. To deal with this problem, Automatic Feature Interaction Selection (AutoFIS) (Liu et al., 2020b) directly learns the weight for each cross feature field, for both second-order and higher-order, and thus an AutoFM model is obtained. It promotes the performances of FM, but faces problems in high computational cost and optimization difficulty, when it is performed on credit scoring datasets with hundreds of feature fields and the desired interaction order is high.

Nowadays, some works try to design various deep learning-based crossing modules for the task of CTR prediction. For better performances, these crossing modules are usually applied along with multi-layer fully-connected DNNs. The Wide & Deep model (Cheng et al., 2016) directly learns parameters of manually designed cross features in the wide component. The Product-based Neural Network (PNN) (Qu et al., 2016) applies inner-product or outer-product to capture second-order features. In Deep & Cross Network (DCN) (Wang et al., 2017), the authors design an incremental crossing module, named CrossNet, to capture second-order as well as higher-order features. Factorization-based methods have also been extended with deep architectures (Guo et al., 2017; Yu et al., 2020; Liu et al., 2020b). In xDeepFM (Lian et al., 2018), Compressed Interaction Network (CIN) is proposed as a crossing module based on outer-product calculation of features. AutoINT (Song et al., 2018) is a combination of residual connections (He et al., 2016) and a crossing module based on Self-Attention (Vaswani et al., 2017). With the residual connections, the fully-connected DNN is performed with the input of original features in AutoINT. Convolutional neural networks are also incorporated for modeling local interactions among nearby features (Liu et al., 2015, 2019). Moreover, feature interactions are modeled as graphs, and then graph neural networks are performed (Xie et al., 2020; Su et al., 2021). Neural architecture search approaches have also been incorporated for finding feature interaction architectures in deep learning-based CTR prediction models (Song et al., 2020; Khawar et al., 2020).

Inherit from deep learning technologies, above feature crossing modules are mostly associated with hidden projection, in each layer or between adjacent layers. This makes deep learning-based methods still black-box models and hard to be globally interpreted, which does not meet our requirements for credit scoring.

DNN has been a powerful black-box model (Guidotti et al., 2018). Recently, extensive works have been done to study local piece-wise interpretability of DNN, which means assigning a piece of local interpretation weights for each sample, a DNN model can be regarded as a combination of numbers of linear classifiers (Montufar et al., 2014; Ribeiro et al., 2016; Lundberg and Lee, 2017; Chu et al., 2018). Some works investigate the gradients from the final predictions to the input features in deep models, which can be applied in the visualization of deep vision models (Zhou et al., 2016; Selvaraju et al., 2017; Smilkov et al., 2017; Alvarez-Melis and Jaakkola, 2018), as well as the interpretation of language models (Li et al., 2015; Yuan et al., 2019). Perturbation on input features is also utilized to find local interpretations of both vision models (Fong and Vedaldi, 2017) and language models (Guan et al., 2019). In this work, we adopt gradient backpropagation for the calculation of local interpretations. Therefore, we can first define the local interpretations in DNN. As introduced, we usually conduct feature discretization for numerical features. Thus, we focus on analyzing DNN with embeddings of categorical features as input.

Local interpretation refers to the contribution of the corresponding feature to the final prediction in the corresponding sample. Sometimes, local interpretations of a specific feature are inconsistent in different samples. When local interpretations are consistent in different samples, it means the corresponding feature contributes to the final prediction on its own. When local interpretations are inconsistent, it means the contribution of the corresponding feature is affected by other features. The inconsistency of local interpretations in DNN is caused by nonlinear feature interactions in the hidden layers of DNN.

Suppose feature fields $\mathop{f}\nolimits_{1},\mathop{f}\nolimits_{2},......$ are nonlinearly interacted in DNN, i.e., ${{\hat{y}}_{k}}=P\left({{\rm{}}{e_{k,{\rm{}}{f_{1}}}},{\rm{}}{e_{k,{\rm{}}{f_{2}}}},......}\right)$, where $P\left(\cdot\right)$ is a nonlinear interaction function. For example, $\mathop{\hat{y}}\nolimits_{k}=\mathop{e}\nolimits_{k,\mathop{f}\nolimits_{1}}\mathop{e}\nolimits_{k,\mathop{f}\nolimits_{2}}^{\top}$. Then, we can have the following local interpretation

In the example of $\mathop{\hat{y}}\nolimits_{k}=\mathop{e}\nolimits_{k,\mathop{f}\nolimits_{1}}\mathop{e}\nolimits_{k,\mathop{f}\nolimits_{2}}^{\top}$, the local interpretations are $\mathop{I}\nolimits_{k,\mathop{f}\nolimits_{1}}^{l}=\mathop{e}\nolimits_{k,\mathop{f}\nolimits_{2}}\mathop{e}\nolimits_{k,\mathop{f}\nolimits_{1}}^{\top}$ and $\mathop{I}\nolimits_{k,\mathop{f}\nolimits_{2}}^{l}=\mathop{e}\nolimits_{k,\mathop{f}\nolimits_{1}}\mathop{e}\nolimits_{k,\mathop{f}\nolimits_{2}}^{\top}$. Obviously, the local interpretation of ${\mathop{x}\nolimits_{k,\mathop{f}\nolimits_{i}}}$ is affected by the features associated with other feature fields. As the values of ${x_{k,{\rm{}}{f_{i^{\prime}}}}}(i^{\prime}\neq i)$ change among different samples, the local interpretations of ${\mathop{x}\nolimits_{k,\mathop{f}\nolimits_{i}}}$ are inconsistent. In contrast, if there is no nonlinear interaction among $\mathop{f}\nolimits_{1},\mathop{f}\nolimits_{2},......$, we have an addition form, i.e., ${{\hat{y}}_{k}}={\rm{}}{P_{1}}\left({{\rm{}}{e_{k,{\rm{}}{f_{1}}}}}\right)+{\rm{}}{P_{2}}\left({{\rm{}}{e_{k,{\rm{}}{f_{2}}}}}\right)+......$, where $\mathop{P}\nolimits_{i}\left(\cdot\right)$ is an arbitrary function. Then, we can have the following local interpretation

Obviously, with the same feature ${{x_{k,{\rm{}}{f_{i}}}}}$, there will be consistent local interpretations among different samples.

To measure the degree of inconsistency among local interpretations of a specific feature in different samples, we first need to calculate its global interpretation, and then calculate the interpretation inconsistency between the local interpretation and the global interpretation.

According to Lemma 3.2, the interpretation inconsistency in DNN is able to lead us to generate a compact and accurate candidate set of cross feature fields. And larger the values of interpretation inconsistency of a specific feature, more the corresponding feature field can work for feature crossing. To verify the validity of interpretation inconsistency for feature crossing, we conduct empirical experiments in Sec. 5.6.

For generating the compact and accurate candidate set of cross features, we need to obtain the local piece-wise interpretations in DNN. Thus, we need first to train a DNN model. Specifically, in this work, we train a multi-layer fully-connected DNN. As the original features are sparse categorical features, we use an embedding layer (Zhang et al., 2016) to transform the input features into low dimensional dense representations. Then, the dense representations are passed through some linear transformation and nonlinear activation to obtain the predictions of samples, where we use ReLu as the activation for hidden layers, and sigmoid for the output layer to support binary classification tasks. To be noted, though we focus on learning feature crossing from multi-layer fully-connected DNN in this work, our proposed approach can also works with other types of DNN, such as AutoINT (Song et al., 2018) and xDeepFM (Lian et al., 2018).

Based on the trained DNN model, in each validation sample $k$, we compute the interpretation inconsistency $\mathop{d}\nolimits_{k,f}$ of the feature $\mathop{x}\nolimits_{\mathop{k},f}$, as defined in Def. 3.4. Therefore, we obtain an interpretation inconsistency matrix $D$, where $\mathop{D}\left[k,f\right]$ is the interpretation inconsistency value of the $f$-th feature field in the $k$-th sample. Then, we conduct an element-wise filtering on matrix $D$ with a threshold $\eta$, which can be formulated as

where $\mathop{D}^{*}$ is a binary feasible feature matrix, and $\mathop{D}\nolimits^{\rm{*}}\left[k,f\right]$ indicates whether the $f$-th feature in the $k$-th sample has interacted with other features in the hidden layers. $Quantile\left(D,1-\eta\right)$ denotes the $1-\eta$ quantile of the matrix $D$, which means keeping the top $\eta$ elements ($0\%<\eta<100\%$) in $D$ with largest values of interpretation inconsistency. Then, for each feasible feature $\mathop{D}\nolimits^{\rm{*}}\left[k,f\right]=1$, the corresponding feature field $f$ can be used to generate candidate cross feature fields.

Finally, we greedily generate the candidate set of cross features fields. Considering that extremely high-order cross features are rarely useful, as in (Luo et al., 2019), we construct our candidate set with only second-order, third-order and forth-order cross feature fields. To construct a compact and accurate candidate set, we need to find cross feature fields which are most frequently interacted in DNN. According to the feasible feature matrix $\mathop{D}^{*}$, we count the occurrences of possible cross feature fields. We then rank the cross feature fields in a descending order according to the corresponding occurrence frequency, and select the top $\varepsilon$ cross feature fields as our candidate set. According to our experiments in Sec. 5.5, we can use $\varepsilon=3n$ as the optimal hyper-parameter, where $n$ is the size of the original feature fields. Accordingly, a compact candidate set of cross feature fields is generated, and searching for useful cross feature fields can be efficiently conducted.

After obtaining the candidate set of cross feature fields $\mathop{C}=\{\mathop{c}_{1},\mathop{c}_{2},...,$ $\mathop{c}_{\varepsilon}\}$, we can search for final useful cross feature fields. The searching strategy in AutoCross (Luo et al., 2019) has been proven effective. Facing a large candidate set, AutoCross trains a LR model for each candidate cross feature field, and select final cross feature fields based on their contribution measured on the validation set. Obviously, there are too many LR models to train, and the efficiency is low. In contrast, we have a compact and accurate candidate set, and are able to feed all candidate cross feature fields into a single LR model. Thus, we do not need to involve the complex searching structure in (Luo et al., 2019), and can simply select useful cross feature fields during validation.

To search for useful cross feature fields, we need to train a sparse LR model consisting of both original feature fields $\mathop{F}=\{\mathop{f}_{1},\mathop{f}_{2},...,$ $\mathop{f}_{n}\}$ and candidate cross feature fields $\mathop{C}=\{\mathop{c}_{1},\mathop{c}_{2},...,$ $\mathop{c}_{\varepsilon}\}$. For a specific sample $k$, the prediction can be made by sparse LR as

where ${\mathop{x}\nolimits_{k,\mathop{c}\nolimits_{i}}}$ is the cross feature associated with cross feature field $c_{i}$ in the sample $k$, ${\rm{LOOKUP}}\left(\cdot\right)$ is a lookup function for the weight in sparse LR of the corresponding feature, $\gamma(\cdot)$ is the sigmoid function, and $b$ is a bias term. Specifically, we first train the sparse LR model with only original features on the training set $\mathop{\Omega}\nolimits_{{\rm{train}}}$. Then, we fix the weights of original features, and train the model with candidate cross features on the training set $\mathop{\Omega}\nolimits_{{\rm{train}}}$. According to our experiments in Sec. 5.5, we can use $\varepsilon=3n$ as the optimal hyper-parameter. Therefore, the time cost of training the above model is in the same order of magnitude with training with original features.

Based on the trained LR model, we conduct searching procedure for useful cross feature fields according to their contribution measured on the validation set $\mathop{\Omega}\nolimits_{{\rm{valid}}}$ according to the AUC (Area Under the ROC Curve) metric. Pseudocode of the searching procedure is presented in Alg. 1. Moreover, the steps 7-12 in Alg. 1 are paralleled with multi-threading implementation, where the measuring of each candidate cross feature field is conducted on one thread. The main idea in Alg. 1 is that, based on parameters in the trained LR model, we measure the contribution in AUC of each candidate cross feature field on the validation set, and iteratively select those have positive contribution as the final set $\mathop{S}\nolimits^{*}$ of cross feature fields. Moreover, we can further incorporate beam search (Cohen and Beck, 2019) in Alg. 1. And this will be further investigated in our experiments in Sec. 5.3.

After obtaining the final set of cross feature fields and a white-box model, we need to deploy the final model to online production. As the final model is a LR model, and LR is widely-used for credit scoring or default prediction in banks or online lending companies, the final white-box model is easily deployed to the online system. We only need to link the final LR model to the existing online LR serving system, and generate cross features based on data flows.

We conduct automatic feature crossing experiments on $4$ datasets: BNP, PPD, Bussiness1 and Bussiness2. BNP²²2https://www.kaggle.com/c/bnp-paribas-cardif-claims-management/data is a public credit scoring dataset from a bank, and PPD³³3https://www.kesci.com/home/competition/56cd5f02b89b5bd026cb39c9/content/1 is a public default prediction dataset from an online lending company. Bussiness1 and Bussiness2 are business datasets from our real-world applications in credit scoring of banks, after anonymization and sanitization. For BNP and PPD, we randomly use $75\%$ and $25\%$ samples for training and testing respectively. For all the four datasets, we randomly use $20\%$ samples in the training set for validation. Details of these datasets can be found in Tab. 1. There are about $100$ to $300$ feature fields in these datasets, characterizing users in profiles, loan history, repayment history and other behavior records.

In our experiments, we are going to verify whether we can obtain a white-box model with high accuracy and global interpretability simultaneously. Specifically, we are going to verify whether DNN2LR can empower the simple LR model to achieve better performances comparing with conventional classifiers, as well as other competitive feature crossing methods. First, we compared DNN2LR with several conventional common-used classifiers: LR, DNN and GBDT. Then, some competitive feature crossing methods are included: FM (Rendle, 2010), HOFM (Blondel et al., 2016), AutoFM (Liu et al., 2020b), GA²M (Lou et al., 2013), GBDT+LR (He et al., 2014) and AutoCross+LR (Luo et al., 2019). Meanwhile, we further include several deep-learning based feature crossing modules: CrossNet (Wang et al., 2017), CIN (Lian et al., 2018) and Self-Attention (Song et al., 2018; Vaswani et al., 2017). They are the feature crossing modules used in CTR prediction models DCN (Wang et al., 2017), xDeepFM (Lian et al., 2018) and AutoINT (Song et al., 2018) respectively. To be noted, as discussed in Sec. 2.2, these deep learning-based methods are still black-box models, and hard to be globally interpreted. Moreover, we run our proposed DNN2LR method $10$ times with different parameter initialization, where average results and significance test are reported. The detailed settings of above compared methods are introduced as following.

For LR, GA²M (Lou et al., 2013) and LR models used in GBDT+LR (He et al., 2014), AutoCross+LR (Luo et al., 2019) and DNN2LR, we tune the learning rate in the range of $\{0.0001,0.001,$ $0.01,0.1,1.0\}$, and the l2 regularization in the range of $\{0.0001,0.001,$ $0.01,0.1,1.0\}$.

For DNN and the DNN model used in DNN2LR, we set the dimensionality of feature embeddings as $10$, the learning rate as $0.001$, and the l2 regularization as $0.0001$. We use Adam for optimization, use the commonly-applied ReLu as activation function, and tune batch size in the range of $\{256,512,1024,4096\}$. The DNN model is a multi-layer fully-connected DNN, whose hidden components in deep layers are set to $[400,100]$.

For FM-based methods, i.e., FM (Rendle, 2010), HOFM (Blondel et al., 2016) and AutoFM (Liu et al., 2020b), we set the dimensionality of feature embeddings as $10$, the learning rate as $0.001$, the l2 regularization as $0.0001$, and tune batch size in the range of $\{256,512,1024,4096\}$. We generate up to forth-order cross features with HOFM. The efficiency of AutoFM is low when there are too many input feature fields and desired interaction order is high. Thus, for efficiency, with AutoFM, we generate up to second-order cross features on default prediction datasets with hundreds of feature fields used in our experiments.

For deep learning-based crossing methods, i.e., CrossNet (Wang et al., 2017), CIN (Lian et al., 2018) and Self-Attention (Song et al., 2018; Vaswani et al., 2017), we set the dimensionality of feature embeddings as $10$, the learning rate as $0.001$, and the l2 regularization as $0.0001$. We use Adam for optimization, use the commonly-applied ReLu as activation function, and tune batch size in the range of $\{256,512,1024,4096\}$. For these three methods, we have $3$ layers of feature crossing, which means conducting up to forth-order crossing. For Self-Attention, the number of hidden units and the number of attention heads are set to $32$ and $2$ respectively.

For GBDT and GBDT+LR (He et al., 2014), we set the number of trees to $1000$, with $10$-round early-stopping. The max depth in the tree is tuned in the range of $\{3,4,5,6,7,8,9,10\}$.

For AutoCross (Luo et al., 2019), we divide the dataset into $\sum\nolimits_{i=0}^{\left\lceil{\mathop{\log}\nolimits_{2}t}\right\rceil-1}{\mathop{2}\nolimits^{i}}$ batches of data for evaluating the contribution of each candidate cross feature field, where $t$ is the size of candidate set in AutoCross.

For our proposed DNN2LR approach, the quantile threshold $\eta$ for filtering feasible features in the interpretation inconsistency matrix is tuned in the range of $\{10\%,5\%,1\%,0.5\%,0.1\%\}$, and the size $\varepsilon$ of candidate set is tuned in the range of $\{n,2n,3n,4n,5n\}$, where $n$ is the size of the original feature fields.

Moreover, our experiments are conducted on a machine with Intel(R) Xeon(R) CPU (E5-26p30 v4 @ 2.20GHz, 24 cores) and 128G memory. For feature preprocessing, we apply multi-granularity discretization (Luo et al., 2019) on numerical features. The granularities in this method are set as $\{10,100,1000\}$, and we only keep the best granularity for each numerical feature field. We evaluate the performances in terms of two commonly-used metrics for credit scoring: the AUC (Area Under the ROC Curve) metric⁴⁴4https://en.wikipedia.org/wiki/Area_under_the_curve_(pharmacokinetics) and the KS (Kolmogorov-Smirnov) metric⁵⁵5https://en.wikipedia.org/wiki/Kolmogorov%E2%80%93Smirnov_test. AUC measures the overall ranking performance. KS measures the largest difference between true positive rate and false positive rate on the ROC curve, which can be a reference of the threshold for loan approval.

Tab. 2 shows the performances comparison among different methods. We can observe that, in most cases, different feature crossing methods can achieve somehow improvements on the performances of LR. Among deep learning-based cross methods, Self-Attention has good performances, and can slightly outperform DNN and GBDT. FM-based methods, i.e., FM, HOFM and AutoFM, does not achieve good performances. This may indicate that, product-based similarity calculation in FM-based methods is not suitable for interactions in credit evaluation tasks. GA²M performs poor, for it is hard to optimize models with too many candidate cross feature fields on credit scoring datasets with hundreds of input feature fields. GBDT+LR has relatively good performances. Obviously, AutoCross+LR is the best one among baseline methods. However, DNN2LR performs even beeter than AutoCross+LR. This is because, on credit scoring datasets with hundreds of feature fields, data for training the field-wise LR model of each candidate cross feature field will be too little to produce reliable results in AutoCross. These experimental results show DNN2LR outperforms conventional classifiers, as well as competitive feature crossing methods on the task of credit scoring. Moreover, the improvements achieved by DNN2LR are similar between public datasets and business datasets.

As discussed in Sec. 4.2, we can incorporate beam search (Cohen and Beck, 2019) in Alg. 1. To investigate this, we show performance comparison between DNN2LR with and without beam search in Tab. 3. To be noted, this experiment is done when the DNN is trained and fixed. With beam search, we keep the best $3$ sets of cross feature fields during each step in Alg. 1, and use the set with best overall performance as the final set. It is clear that, with or without beam search, the performances are close. Accordingly, it is not necessary to incorporate beam search in the searching procedure of DNN2LR.

Furthermore, as shown in Fig. 3, we illustrate the count of both second- and higher-order cross feature fields generated by DNN2LR on each dataset. Roughly speaking, more original feature fields in the dataset, more cross feature fields we need to generate.

For the efficiency comparison, we involve the best baseline method AutoCross, according to Tab. 2. Fig. 4 shows the running time comparison between AutoCross and DNN2LR. DNN2LR accelerates AutoCross by about $10\times$, $31\times$, $23\times$ and $42\times$ on BNP, PPD, Business1 and Business2 respectively. According to Tab. 1, there are $100$ to $300$ feature fields in these datasets. It is clear that, the speedup ratios are significant, and get larger when we have more input feature fields in the dataset. These results strongly illustrate the high efficiency of DNN2LR for feature crossing on credit scoring datasets with hundreds of feature fields.

As shown in Fig. 5, we illustrate the sensitivity of DNN2LR to hyper-parameters. First, we investigate the performances of DNN2LR with varying quantile thresholds $\eta$ for filtering feasible features in the interpretation inconsistency matrix. According to the curves in Fig. 5, the performances of DNN2LR are relatively stable, especially in the range of $\{5\%,1\%,0.5\%\}$. Then, we investigate the performances of DNN2LR with varying sizes $\varepsilon$ of candidate set. As discussed in Sec. 5.3, we need to generate more cross feature fields when we have more original feature fields. Thus, we make the size of candidate set related to the number $n$ of original feature fields. We can observe that, the curves are stable when $\varepsilon\geq 2n$. And when we have larger $\varepsilon$, the performances of DNN2LR slightly increase. According to our observations, we set $\eta=5\%$ and $\varepsilon=3n$ as the optimal hyper-parameters in our experiments. This makes the candidate size extremely small, comparing to the set of possible cross feature fields on credit scoring datasets with hundreds of feature fields.

To investigate the validity of interpretation inconsistency for feature crossing, we conduct some empirical experiments. We conduct experiments on some mathematic formulations, which are shown in Tab. 4. In each formulation, there are three feature fields: $\alpha$, $\beta$ and $\lambda$, where $\alpha$ and $\beta$ are interacted, while $\lambda$ has no interaction with the other two fields. The interactions between $\alpha$ and $\beta$ cover different forms, and the calculation of $\lambda$ also varies from simple to complex. For each formulation, we randomly sample features in $[0.0,1.0]$, and keep two decimal places. We perform DNN on each formulation, and calculate the average interpretation inconsistency values of the three feature fields according to Eq. (7), as shown in Tab. 4. To be noted, we do not perform feature discretization in this experiment. It is clear that, in each formulation, the average interpretation inconsistency values of $\alpha$ and $\beta$ are much larger than that of $\lambda$. This observation clearly confirms to the formulations.

Furthermore, we conduct experiments on sentence classification. We train BiLSTM on the MR dataset⁶⁶6http://www.cs.cornell.edu/people/pabo/movie-review-data/. There are $5331$ and $5331$ positive and negative movie reviews respectively. Values of interpretation inconsistency in several example sentences are shown in Fig. 6. We achieve two observations about words with large values of interpretation inconsistency. First, words that form phrases have large values of interpretation inconsistency, e.g., “in other words,” “can’t … get anywhere near,” “meets those standards,” “an hour and a half to blow” and “it goes nowhere.” Second, words with specific intensities or orientations have large values of interpretation inconsistency, e.g., “a lot more fun than the film,” “a lot better” and “a little more than this.” These observations confirm to the feature interactions among words in sentences. These experiments suggest that it is valid to learn feature crossing from interpretation inconsistency in DNN.

In Tab. 5, we illustrate top-$20$ cross feature fields generated by DNN2LR on public datasets, i.e., BNP and PPD. Once the meanings of original feature fields are known, we can obtain the meanings of cross feature fields. The final model generated by DNN2LR is a LR model empowered with cross features, which is a white-box model with high accuracy and global interpretability simultaneously.