A decentralized aggregation mechanism for training deep learning models using smart contract system for bank loan prediction

Artificial Intelligence (AI) and blockchain are proving to be one of the most powerful and dynamic pair of technologies used together, improving the performance of the other one in every discipline where they are implemented. They have been used together to cause major upgrades in various applications including supply chain logistics, finance-related data, and medical health records pujahnp (2020). Blockchains are decentralized ledgers which maintain an immutable record of transactional data in digital format Zheng et al. (2018). Deep learning is a sub-domain in AI that involves multi-layered architectures that extract features from the input data and map them to the output labels. Recent years have seen a rise in the widespread application of AI models and algorithms in various areas Wang et al. (2018); Ratadiya and Mishra (2019). Deep learning-based AI systems strongly rely on data, something which blockchains can store with an exceptionally high degree of authenticity. The Blockchain-AI convergence is imminent since both of them provide a distinct, significant advantage to entities which store data.

With the advancement of AI, many of the data privacy challenges have also been cited, especially when working on real-world tasks with the existing data Jagannathan and Wright (2005); Mohassel and Zhang (2017). Being able to re-identify personal information using large datasets, lack of transparency in the use of consumer data Dorschel (2020), regulations by countries, and unions have been some major factors in intensifying this problem. With more features and personal data being collected and stored, there is an inherent risk and higher probability of sensitive information getting exposed and shared.

Federated learning has been an approach that has been used in the past to tackle data-sharing issues Konečnỳ et al. (2016). However, federated learning has mostly stressed on combining updates from models trained on different samples of data, but those which have similar types of features i.e. horizontally partitioned data Smith et al. (2017). However, many organizations also follow a vertical partitioning of data. Further, there might be hosts who wish to train a smart system based on data that is currently present at different organizations. The involved parties may not always be permitted to share the data. Consider the task of bank loan prediction. The best prediction model can be trained from the existing data by combining both the personal details of the users and also their bank or finance-related information. However, if these details are not present in a single place, quality feature extraction from such decentralized data is a challenging task.

In this paper, we propose a smart contract-based system for aggregation of features for better training from decentralized data. We prove the effectiveness of our architecture when working with artificial neural networks to build a model for the task of bank loan prediction. Setting up the architecture over a blockchain also makes the exchange of intermediate information more secure and free from external threats. The proposed mechanism is found to be superior in handling the data privacy issue and also provides optimum results on a publicly available dataset. Our main contributions in this paper could be listed as:

1. 1.

   We have been able to achieve quality results despite training the host model from two different sets of features extracted from local models which are located on separate nodes, as opposed to the conventional centralized model setup in deep learning.

2. 2.

   We have made effective use of the Ethereum Blockchain and IPFS to store, retrieve, and exchange the intermediate feature vectors required for the training, and test data required for the inferences from the local models.

3. 3.

   The performance of the host model trained on aggregated features from decentralized data is at par with a centrally trained model, thus providing a better alternative that is more private and secure as implemented on a blockchain.

The rest of the paper is structured as follows: Section 2 talks about the related work in this area while the dataset description is presented in section 3. The proposed methodology is explained in section 4. Our obtained results and analysis of the work are discussed in section 5 while the paper is concluded in section 6.

In the domain of data privacy, several vulnerabilities have been detected. These have been consistently explored and worked upon by researchers to reduce the possibility of violation of any privacy norms, either unethically or legally.

Some of the earlier works are based upon the principle of secure multi-party computation(SMC) which has been used for k-means clustering Bunn and Ostrovsky (2007), Jagannathan and Wright (2005), SVM classification Lindell and Pinkas (2000), linear regression functions Du et al. (2004), stochastic gradient descent method Mohassel and Zhang (2017), association rule mining in vertically partitioned data Vaidya and Clifton (2002), and Naive Bayes classification Vaidya et al. (2008). These approaches eliminated the trade-off between data usability and data privacy and safeguarded the system against scooping on the data. Privacy-preserving machine learning using a neural network has also been worked upon quite actively. Xie et al. (2014), Bos et al. (2014) proposed a method in encrypted domain which applies homomorphic encryption to activation functions with neural networks to ensure the privacy of the data. Barni et al. (2011) used a privacy-preserving automatic diagnosis system based on linear branching programs and neural networks. Pathak et al. (2011) presented a model that contains an asymmetric framework for the encryption of probabilities that uses public-key additively homomorphic cryptosystem. They went on to propose a privacy-preserving speaker identification based on the Gaussian Mixture Model-based protocol Pathak and Raj (2012). Recent years have seen an active rise in the work being done in the area of federated learning, which proposes a way for combining weights or learning of the client nodes and returning them with a global update after aggregation of those weights Konečnỳ et al. (2016). Federated learning is also being tried out for multitask and semistructured learning Smith et al. (2017). Hardy et al. Hardy et al. (2017) proposed a novel federated learning approach for vertically partitioned data using both entity resolution and homomorphic encryption. However, the scope of their data modeling algorithms was restricted to logistic regression and Taylor approximation, which are not always sufficient, especially when modeling complex real-world data.

The majority of these previous approaches have applied secure multi-party computation. However, SMC models are subjected to computational overhead and high communication cost between the participants. Homomorphic encryption-based models using neural networks are computationally expensive. Although differential privacy safeguards privacy against a wide range of privacy attacks, it generates too much noise which ultimately reduces the data utility when dealing with diverse data. Thus, a more balanced solution, capable of working with separate data in an efficient yet effective way is the need of the hour.

We make use of a publicly available dataset¹¹1https://www.kaggle.com/itsmesunil/bank-loan-modelling for personal bank loan classification. Given their details, the task is to predict whether a person will opt for the bank loan or not.

The dataset consists of thirteen structured input columns and one binary output label, personal loan (0:loan not taken, 1:loan taken). The irrelevant features are eliminated using correlation analysis of individual columns with the output variable. We introduce a combination_feature based on income and average card spending. After these steps, we are left with 10 input features. The dataset consists of 5000 samples, 480 of which are having output label 1 (loan is taken). The dataset is split into training and testing sets in the ratio of 7:3.

For our demonstration of decentralized setup, we split the training dataset into two parts: personal details (6 features) and bank-specific details (4 features). The two parts consist of the following columns:

• •

  Personal details: Education level, no. of family members, annual income, average credit card spending, years of work experience, the value of house mortgage.

• •

  Bank-specific details: Does the customer have a security account with the bank, do they have a CD account with the bank, do they have a credit card issued by the Universal Bank, combination_feature.

Each of these splits are now considered as separate data nodes as could be the case in real life. Our objective is to train a model that can capitalize on both these splits during training while still not needing to explicitly share these values, hence preserving the data privacy regulations.

We work on the task of building a deep learning model for bank loan prediction. The existing training data is partitioned and available on two different nodes: one containing the personal details, and others containing the bank-specific details of the users. This is achieved by loading the respective data on two distinct nodes of the Ethereum blockchain. The nodes perform training on their local data and send the intermediate representations to the host. The host combines these representations and trains a third model based on which the final predictions will be derived. We explain our mechanism by describing our approach for each phase in the data modeling pipeline- training local models on respective data nodes, passing feature representations of training data to the host node, model training on the host node, and data flow for a new test input at the host.

The results of the models are evaluated on their respective datasets as well as the combined datasets. We first compare the performance based on the accuracy obtained by the models on the respective nodes in Table 2.

It can be seen that our proposed aggregated mechanism has helped to come up with a host model better than the individual models trained on their respective data in terms of accuracy. To further solidify our claims, we also evaluate the performance in terms of their precision, recall, and F1 score and show our results in Table 3.

Thus it can be seen that our aggregated model has been able to combine the learned representations of both the models which have helped in an enhanced performance across all performance metrics. We have been able to train well from decentralized data setups and achieve quality results on new test cases.

It may seem obvious to few that combination and more data columns will give better results than involved sub-models. However, we focus on the aggregation within a decentralized environment and thus also compare our decentralized model performance with the third kind of model: the traditional centralized setup. We train and test a model directly on the 10 input values gathered together in one place in a conventional format. The classification reports for our decentralized model and the centralized model are tabulated in Table 4 and 5 respectively.

It can be seen that there is not much difference in the performance of our decentralized model as compared to that of the centralized model across all the metrics. While the centralized model does have better recall and F1 score value for the minority class, it comes at the expense of having to share and combine two different data sources, thus introducing security and privacy concerns. There is only a 0.2% difference between their accuracies, thus making a strong case for our decentralized setup. However, such a vertically partitioned dataset must not destroy the learning experience and performance of the model, and hence data partitioning needs to be appropriate.

Further, we also plot the training accuracies and loss variations to the epochs in the case of both the centralized and decentralized models. Their respective plots are visualized in Figure 3 and 4.

It can be seen that the decentralized model converges faster as compared to the centralized data model. This could be attributed to the fact that the decentralized data model receives richer data features and thus gets a better initialization, hence leading to faster convergence. We have thus been able to propose a system which gives at par results even with distributed, decentralized data and yet has a faster convergence period as compared to previous approaches.

Our proposed aggregation mechanism is more secure as it operates over a blockchain, makes use of an efficient file hashing mechanism, gives almost the same results on decentralized data as compared to centralized datasets, and hence makes for a strong case for organizations who wish to train models on their vertically partitioned data.

Despite having a decentralized setup, we have been able to accomplish standard results for the task of bank loan prediction using an effective aggregation mechanism. The performance of the aggregated model is better than the local nodes both in terms of performance metrics as well as convergence time. It is also almost equivalent to the performance of a centralized data setup. Blockchain and Artificial Intelligence can prove a powerful integrated architecture in preserving data privacy and maintaining data security. While the majority of the previous approaches have demonstrated their work on a centralized data format, our results show that using the aggregation of feature representations, we can still obtain maximum performance. Further improvements in our work include the use of more secure encryption techniques for encrypting the files before uploading them to IPFS. The mechanism can also be extended further for image and time series based data where more advanced deep learning models would be required. In the future, techniques that can extract information from multiple data sources without needing to share the actual data would be desired and will lay the foundation for a more secure, ethical, and private data analysis environment.