Temporal Knowledge Distillation for Time Sensitive Financial Services Applications

Machine learning and artificial intelligence solutions have been widely used in the financial services industry. The applications of AI/ML range from trading to consumer and business credit decisions, financial crime detection, mobile banking, and authentication systems (Forum, 2020). These use cases also span almost all critical business functions from business operations to risk management (Heaton et al., 2016; Oprea et al., 2019). Overall, AI and machine learning solutions have a significant impact on the performance and operations of today’s financial services firms.

Financial firms frequently use machine learning to detect anomalies in cybersecurity applications, fraud detection, compliance, and financial crime detection (Anandakrishnan et al., 2018). Among these, there is a sizable list of mission-critical applications, each of which requires effective and timely detection as well as response to anomalous events promptly (Anandakrishnan et al., 2017).

The number of models used in high-volume and time-sensitive applications is growing due to the recent trends in mobile payments and digital banking (Council, 2019). In the past few years, mobile payment systems have experienced unprecedented growth (Zelle, 2021). Zelle transaction volumes increased by 61% year-to-year as of 2021 (Heun, 2021),(Systems, 2019). As the high-speed, high-volume nature of digital transactions attracts financial crime organizations, digital payments fraud and cybercrimes have rapidly become among the top challenges in financial firms. In digital payments, mobile P2P fraud experienced a 733% increase from 2016 to 2019. Similarly, account takeover fraud (ATO) grew by 72% just in one year, from 2018 to 2019 (Reports, 2020) and rose over 200% in 2021 according to industry reports (Shein, 2021).

One of the grand challenges machine learning models face in these use cases is the dynamic and adversarial nature of the detection process. Unlike naturally stable datasets used in other application areas of AI/ML, fraud and anomaly detection systems experience frequent and rapid pattern changes (Marfaing and Garcia, 2018), (Buehler, 2019). The pattern changes occur in both (i) normal events, as in changes in the non-fraud transactions and customer behavior, as well as in (ii) anomalous events, where perpetrators implement new fraud tactics. As an example, in account takeover fraud (which is characterized by perpetrators gaining access to the customers’ account and draining the funds across multiple channels (Council, 2019),(Harrison, 2020)), fraud tactics are known to show rapid changes. In some instances, the patterns change from one popular tactic to the next in a matter of hours. This raises serious concerns for industry-standard machine learning solutions that rely on supervised learning. In payment fraud detection systems, which process and score millions of transactions every day with millisecond response time SLAs, the underlying modeling challenges become more pronounced (Hasham et al., 2019),(Shepard et al., 2019), (Chatain et al., 2011).

In high-volume and dynamic application environments, such as payment fraud detection, models face serious dilemmas:

• •

  Data Size & Balance: Retraining the models with the new patterns is a common approach to improve the model performance for recent data patterns. However, it often degrades the performance of historical patterns that repeat. Excluding the historical patterns causes retention challenges. Yet, continuously extending the training data set with additional data causes size and training time issues (A.Shah, 2020). This highlights the need to balance the historical and recent data while preventing uncontrolled growth in the training data. Similarly, in anomaly detection cases model retraining aims to (i) capture all anomalous patterns and (ii) balance the anomalous/normal event percentages in the training data to achieve the highest model performance. The combination of both goals often causes the machine learning models to use larger data sizes to deal with highly dynamic environments.

• •

  Training Times & Model Agility: Similarly, the amount of time required to retrain the models is a serious challenge in the deployment of time-sensitive production models. Models need to be rapidly updated and moved back to production systems to prevent further financial and cybersecurity damage. In response, numerous techniques have been explored to reduce the training times. Hardware acceleration using graphical processing units (GPU) (Zhang et al., 2017), (Shepovalov and Akella, 2020), field programmable gate arrays (FPGA) (Zhang et al., 2015)(T Wang et al., 2018), memory optimizations (Eleftheriou et al., 2019), vectorization and other techniques have been proposed (Daghaghi et al., 2021), (N et al., 2016), (Chen et al., 2018). While the hardware and system optimizations provide benefits, optimizing the models themselves for training time is equally important in improving the overall solution.

This paper explores a novel supervised learning approach to tackle these key challenges by providing a way to update models faster, while balancing the current and historical datasets in high-volume and time-sensitive financial services applications. The proposed technique, Temporal Knowledge Distillation (TKD), transfers knowledge from the historical data to boost the model performance through data label augmentation. It aims to balance the data to enhance the model’s robustness. Further, it improves the training time for agile response in adversarial use cases, such as fraud detection and account takeover. The paper explores a fraud detection use case to analyze the effectiveness of the proposed approach. However, the approach can be broadly applied to a wide range of financial services applications due to the prominence of high-volume and time-sensitive applications in the financial services systems.

The paper is organized as follows: Section 2 discusses the related work in knowledge distillation; Section 3 outlines the TKD temporal knowledge distillation approach; Section 4 describes the experimental analysis setup; Section 5 overviews the experimental results; finally Section 6 discusses the conclusions.

The concept of Knowledge Distillation (KD) was explored by a number of researchers (Buciluǎ et al., 2006; Ba and Caruana, 2014; Hinton et al., 2015; Urban et al., 2016; Furlanello et al., 2018). Initially, the goal of KD was to produce a compact student model that retains the performance of a more complex teacher model that takes up more space and/or requires more computation to make predictions. Dark Knowledge (Hinton et al., 2015), which includes a softmax distribution of the teacher model, was first proposed to guide the student model.

Recently, the focus of this line of research has shifted from model compression to label augmentation which can be considered a form of regularization using Dark Knowledge. In (Furlanello et al., 2018), a chain of retraining models, parameterized identically to their teachers, Born Again Network (BAN), was proposed. The final ensemble of all trained models can outperform their teacher network on computer vision and NLP tasks. Additionally, (Furlanello et al., 2018) investigated the importance of each term to quantify the contribution of dark knowledge to the success of KD.

Following this direction of research, self distillation has emerged as a new technique to improve the classification performance of the teacher model rather than merely mitigating computational or deployment burden. Label refinery (Bagherinezhad et al., 2018) iteratively updates the ground truth labels after cropping the entire image dataset and generates a set of informative, collective, and dynamic labels, from which one can learn a more robust model. In another related study, (Romero et al., 2014) aimed to compress models by approximating the mapping between hidden layers of the teacher and the student models, using linear projection layers to train relatively narrower students.

KD has already gained success in many applications in a wide range of domains. In (Yang et al., 2019), authors have proposed to distill the knowledge of essence in an ensemble of models to a single model that needs much less computation to deploy for building a speech recognition system. Similarly, KD was utilized in computer vision domain: object detection (Chen et al., 2017), deepfake detection (Kim et al., 2021), image super-resolution (Zhang et al., 2021). (Sanh et al., 2019) proposed a method to pre-train a smaller general-purpose language representation model, called DistilBERT based on Knowledge Distillation, which can reduce the size of a large BERT NLP model by 40%. In the emerging field of federated learning, KD has started to be used to tackle some challenges such as user heterogeneity issue (Zhu et al., 2021) and expensive model payload during communication (Seo et al., 2020). Besides the traditional response-based distillation, feature-based distillation (Chen et al., 2020) has recently gained attention from researchers with an attempt to leverage knowledge from the intermediate layers in addition to the output layer (Gou et al., 2020). As KD shifts its focus from model compression to domain adaption, it has shown its potential in handling heterogeneous data sources. Our work is motivated by this trend and shares some conceptual similarities with (Farhadi and Yang, 2019) for both attempt to distill knowledge temporally. However, (Farhadi and Yang, 2019) utilizes temporal correlations across video frames while the proposed work in this paper mainly tackles the dynamic nature of anomaly detection use case.

The proposed TKD label augmentation incorporates Dark Knowledge from previously trained models, which have been trained with different time ranges to augment the labels of the latest dataset. This new knowledge enables the transfer of learning from historical patterns extracted by experienced experts. With the assistance of their expertise, the new model sees performance improvement without having the historical datasets in its training. This enables more effective detection of anomalous events, and streamlines model retraining and deployment in a time-sensitive scenario.

Consider the classical classification setting with a sequence of training datasets corresponding to $N$ different time frames consisting feature vectors: $X_{t}$ and labels $Y_{t}$ where $t=0,1,...N$. For traditional supervised learning algorithms, a model is trained on $\{X_{<t},Y_{<t}\}$ for each time frame. Naturally, the size of $\{X_{<t},Y_{<t}\}$ increases as time passes. TKD leverages the outputs generated by previously trained models $M_{<t}$ prior to each time frame $t$ instead of including historical data in the training directly. These outputs are used to augment labels of the latest dataset and construct a regularizer to the conventional loss function. For time frame $t$, the training dataset will be $\{X_{t},Y_{t}\}$ only and the general form loss function to optimize in the training becomes:

where $O_{i,t}$ and $y_{t}$ represents model $M_{i}$ output on data $X_{t}$ and model output at the current time frame, respectively. $CE$ and $KL$ are Cross-Entropy and Kullback–Leibler divergence. With this second term in the loss function Eq. 1, existing ground truth labels are augmented by the experienced experts. Typically, KL divergence is used to minimize the discrepancy between the logits outputs from the teacher model and the student model, respectively.

$AGG[\cdot]$ serves as an aggregating function which collects ‘vote‘ from each experienced expert. There are many options for this aggregating function e.g. $max()$, $sum()$, $mean()$ and etc. and in this work $mean()$ is selected after some empirical experiments. The coefficient $\alpha$ is used for balancing the label augmentation term and the cross-entropy on new data. As $\alpha$ gets closer to one, the training for the target model depends more on the ground-truth labels of the new data and this approach gradually converges to a classic classification without any regularizer. For simplicity, $\alpha$ is set to 0.5 in this paper.

As the number of models increases over time, the historical models, whose underlying training data patterns have changed provide increasingly less meaningful information on the recent anomaly patterns. Thus, including them in the training may not provide further performance gain for retraining and possibly deteriorate the performance. To reduce the negative impact of this distribution shift, we use parameter $K$ to determine which model to start with and truncate all the previous models prior to the current one. In this study we used an empirical approach to determine $K$. With all the specific setups, the loss function used can be simplified as follows:

Fig. 1 illustrates the architecture of TKD. For the first time frame $t=0$, a model $M_{0}$ is trained on dataset $\{X_{0},Y_{0}\}$. Then, for each of the following time frames (depending on the specific retraining schedule), a new identical model $M_{t}$ is trained from, $O_{i,t}$ the supervision of previous models $M_{<t}$ by using Eq. 2. Auxiliary soft labels (outputs) from the previous models are highlighted in orange in Fig. 1. Note that this figure depicts the $K=0$ case. When the number of historical models becomes large, certain truncation can be applied based on prior knowledge of each historical model.

This section presents the performance comparisons between baseline machine learning models and their TKD counterparts. Both one month and consecutive months results as described in the Section are presented 4.2). Next, we analyze the model training time over the entire 6 months experiment period to show the training time characteristics of TKD compared to the baseline.

This study proposes a label augmentation algorithm, Temporal Knowledge Distillation (TKD), for time-sensitive financial anomaly detection applications. This technique aims to provide a new way to boost the model performance by incorporating a wider range of patterns including older and newer patterns without causing unmanageable increases in the data size. Furthermore, it minimizes the model retraining times compared to the baseline models. In adversarial and time-critical use cases, such as cybersecurity and payment fraud detection applications, this yields significantly higher agility and more effective response capabilities to attacks. Despite the recent advancements in acceleration techniques, model retraining times remain as a challenge in deploying AI/ML models in production systems. TKD delivers an alternative approach to model retraining in time-sensitive and high-volume applications, which provides key benefits to the overall success of the deployment systems.