Enhancing Precision of Automated Teller Machines Network Quality Assessment: Machine Learning and Multi Classifier Fusion Approaches

The primary challenge addressed in this study is the frequent occurrence of false alarms in ATM status detection. ATM availability is usually tracked using ATM Status Files, but these files often generate false alerts, leading to unnecessary maintenance activities and misallocation of resources. Conversely, ATM Journal Files offer precise information on ATM status, detailing every operational event and error. However, their substantial size and complexity make real-time processing infeasible.

To address this challenge, the problem was approached as a binary classification task, aiming to categorize each ATM as either in-service or out-of-service. The ground truth labels for this classification were derived from the ATM Journal Files, which underwent extensive preprocessing to serve as the reference standard. Given the significant size constraints of these files, they were used solely for labeling purposes rather than being directly fed into the classification models.

Out-of-service statuses were extracted by coding and systematically parsing ATM Journal Files, focusing on component-specific errors such as card reader failures, keypad malfunctions, and network disconnections. Specifically, errors related to the card reader, keypad, and network disconnections were analyzed to identify when an ATM was genuinely out of service. Figure 3 illustrates this extraction methodology, highlighting how component-level errors were mapped to the ATM’s operational status.

To develop effective classification models, relevant features were extracted from various data sources, primarily focusing on the ATM Status Files and financial transaction records. The following key features were identified and utilized:

• 1.

  ATM Status Files: The operational state of an ATM (either in-service or out-of-service) was derived by analyzing the status of three crucial components: the keypad, card reader, and network connection. If any of these components exhibited errors or if the network connection was interrupted, the ATM was classified as out-of-service. Conversely, an ATM was considered in-service if all components were functioning properly. Figure 3 demonstrates the extraction process, showing how component-specific errors are used to determine the overall status of the ATM. Although this data source is susceptible to inaccuracies, including false and missed alarms, it remains a valuable input for modeling. Accounting for these potential inaccuracies is essential in developing a more robust classification strategy for ATM status.

• 2.

  Day of the Month: This feature captures the specific day of the month (ranging from 1 to the final day), providing insights into patterns of ATM usage that may correspond to specific periods, such as salary disbursements or holidays.

• 3.

  Type of Working Day: Days were categorized as regular working days, part-time working days, or holidays. ATMs experience varying demand levels based on the day type, which can influence the likelihood of service disruptions.

• 4.

  Time of Day: The time when an ATM event occurs is crucial for distinguishing between periods of high usage (e.g., midday or evening) and low-traffic periods (e.g., early morning or late at night).

• 5.

  Number of Monthly Transactions: This feature represents the number of financial transactions processed by an ATM within a month, stratified into quantiles. ATMs with higher transaction volumes tend to experience increased wear and tear, making them more prone to service issues.

• 6.

  Transaction Status Feature: Under the assumption that ATM transactions follow a Poisson process, the time intervals between consecutive transactions were modeled using an exponential distribution. The Kolmogorov-Smirnov (KS) test validated this hypothesis, confirming the suitability of the exponential model. This feature provided a supplementary indicator for detecting out-of-service conditions, achieving an accuracy rate of 85.43%.

The Kolmogorov-Smirnov (KS) test was conducted to validate the hypothesis that transaction intervals follow an exponential distribution, a critical assumption for estimating ATM service reliability. Figure 4 presents the histogram of transaction intervals overlaid with the probability density functions (PDFs) of various distributions, including Exponential, Gamma, Logistic, and Normal. The analysis revealed that the exponential distribution provided the best fit, validating its use for modeling transaction intervals and estimating the likelihood of the next transaction.

The Kolmogorov-Smirnov (KS) test results, summarized in Table 1, indicate that the exponential distribution provides the best fit for the transaction interval data. Specifically, the KS statistic for the exponential distribution is 0.1493, which is lower than the values for the Gamma (0.1654), Logistic (0.1906), and Normal (0.2557) distributions. These findings support the hypothesis that the intervals between transactions follow an exponential distribution, thereby justifying the use of a Poisson process model to estimate transaction occurrences. This assumption is pivotal because it enables the calculation of the probability of the next transaction occurring within a certain time frame, which is essential for detecting potential out-of-service conditions.

The dataset exhibited a significant class imbalance, with only 0.85% of the instances representing the out-of-service class. To mitigate this issue, the Synthetic Minority Over-sampling Technique (SMOTE) was utilized, with k = 3 nearest neighbors selected to generate new synthetic instances for the minority class. This approach successfully balanced the dataset to a 50:50 ratio between the in-service and out-of-service classes.

For example, SMOTE generates synthetic samples for out-of-service events, enabling models to better recognize rare instances without overfitting. However, the choice of the k parameter is critical to maintain the integrity of the newly generated samples. A smaller k value, such as k=3, was chosen to ensure that synthetic samples are generated based on the closest neighbors of each minority instance. This method preserves the local structure of the minority class and minimizes the risk of adding noise or outliers to the dataset.

Selecting k=3 achieves a balance between producing a sufficient number of synthetic samples and maintaining the quality of the data. This parameterization minimizes the potential for overfitting that can occur when synthetic samples are generated using distant neighbors, which may not accurately represent the minority class distribution.

The following machine learning models were applied to the binary classification problem, each trained using the extracted features:

• 1.

  Support Vector Machines (SVM): A robust classification model, known for its effectiveness in high-dimensional spaces and suitability for binary classification tasks. SVM works by identifying the optimal hyperplane that maximizes the separation margin between different classes.

• 2.

  Decision Tree: A straightforward yet powerful model that partitions the feature space into regions based on decision rules, enabling it to handle both categorical and numerical data with ease.

• 3.

  Ensemble Learning:

  • (a)

    Bootstrap Aggregation (Bagging): An ensemble technique that combines multiple classifiers to reduce variance and improve stability by training each model on randomly sampled subsets of the data.

  • (b)

    Random Forest: A sophisticated extension of bagging, where decision trees serve as the base learners. It aggregates the predictions from multiple decision trees to reduce overfitting and enhance generalization. Random Forest achieved the highest accuracy in this study, with a score of 99.26%.

  • (c)

    LightGBM (LGBM): A highly efficient Gradient Boosting Decision Tree (GBDT) framework optimized for speed and performance, especially on large datasets. Its leaf-wise tree growth approach makes it a preferred choice for imbalanced datasets.

  • (d)

    CatBoost: Another GBDT-based algorithm, designed to handle categorical features natively without extensive preprocessing. CatBoost addresses issues like target leakage and reduces overfitting, making it ideal for complex datasets.

  • (e)

    Stacking Classifier: Ensemble learning methodology combines several base classifiers for an improved predictive performance by leveraging their complementary strengths. In this study, base learners are Random Forest, LightGBM, and CatBoost, and the meta-learner is Logistic Regression. The stacking classifier is one that combines insights from various algorithms for a more robust model.

• 4.

  Dynamic Classifier Selection (DCS) and Dynamic Ensemble Selection (DES): Advanced ensemble techniques that enhance decision-making by selecting the most suitable classifiers based on the local accuracy for each test instance. These methods ensure that predictions are tailored to the specific characteristics of the data, improving overall classification performance.

Certain key metrics were employed for the performance testing of the classification models for assessing the status of ATM services. The focus was on average scores across the in-service and out-of-service classes for a balanced view of the performance of every model. This includes:

• 1.

  Precision: Precision is the ratio of positively identified instances by the total number of instances predicted as positive. Averaging precision over both classes provides an overall view of the accuracy of each model without weighting any particular class too much. It is given by:

  $$\text{Average Precision}=\frac{\text{Precision}_{\text{down}}+\text{Precision}_{\text{up}}}{2}$$

  (1)

  This means higher precision reflects the model’s ability to minimize false alarms; hence, minimal instances where there was an ATM that had been flagged inappropriately. This improves resource allocation.

• 2.

  Recall (Sensitivity): Recall is the ratio of correctly predicted positive instances to all actual positive instances. Averaged recall from both classes points to a model’s ability to identify in-service and out-of-service status, hence showing balanced performance. It is computed as:

  $$\text{Average Recall}=\frac{\text{Recall}_{\text{down}}+\text{Recall}_{\text{up}}}{2}$$

  (2)

  High recall signifies that the model detects true instances of each class accordingly, hence giving fewer missed alarms and building reliability.

• 3.

  F1-Score: The F1-score is the harmonic mean of precision and recall and gives a measure that balances the two. Averaging the F1-scores of both classes gives an overall measure of balanced performance. It can be computed as:

  $$\text{Average F1-Score}=\frac{\text{F1-Score}_{\text{down}}+\text{F1-Score}_{\text{up}}}{2}$$

  (3)

  The average F1-Score shows, quite nicely, the ability of the model to handle both classes well so that decisions are not disproportionately skewed by either false positives or false negatives.

Paying attention to average metrics is important in both classes for a number of reasons:

• 1.

  Minimizing Operational Disruptions: This balance between precision in both classes reduces false alarms, missed alarms, optimizes resource utilization, and minimizes superfluous intervention.

• 2.

  Enhancing Response to Service Status Changes: High recall across classes ensures timely attention for ATMs that need service, improving customer satisfaction and reducing revenue losses by unresolved service issues.

• 3.

  Maximizing Service Reliability: Any rise in the average F1-score shows that the proposed model is reliable in terms of detecting both in-service and out-of-service status more precisely, hence making for a far more operationally efficient environment and also allowing customers to trust services more.

Overall accuracy is generally considered; however, due to issues of class imbalance, it is less informative here. Instead, average metrics—precision, recall, and F1-score—provide a more appropriate, informative estimate of model performance for the reliable management of ATM service status across classes.

The features used in this study include various ATM-related data points such as the day of the month, type of working day, time of day, and the transaction rate of the ATM, which were classified into high or low transaction categories. The transaction rate was normalized into a value ranging from 0 to 1, where ATMs with lower transaction volumes received values closer to 0, and ATMs with higher transaction volumes were assigned values closer to 1. This quantile-based approach helps capture the relationship between transaction load and potential error occurrences.

In addition, the study employs two primary status features: Extracted Status from ATM Status Files and Extracted Status from ATM Transactions. These features provide binary information on the ATM’s operational state, with ‘1‘ indicating an in-service status and ‘0‘ indicating an out-of-service status.

The type of working day was categorized into five distinct types: first day of the week, last working day, holidays, part-time days, and regular working days. A value between 0 and 1 was assigned to each category based on their frequency of occurrence.

Figure 6 illustrates the correlation matrix of these features, showing the relationships and interdependencies between them.

The ATM Status Files, which indicate the ATM’s in-service and out-of-service states, were evaluated using accuracy, precision, recall, and F1-score metrics. Similarly, the statuses derived from the transaction tables, which operate under the assumption that ATM transactions follow a Poisson distribution with exponential inter-arrival times, were also assessed using these metrics.

Table 2 compares the performance of ATM Status Files and transaction log-derived statuses, highlighting the accuracy trade-offs between these sources. The results show that the ATM Status Files achieved an overall accuracy of 96.14%, while the status extracted from transaction logs resulted in a lower overall accuracy of 85.43%.

To address the class imbalance present in the dataset, the Synthetic Minority Over-sampling Technique (SMOTE) was implemented to upsample the minority class (out-of-service). Initially, the ratio of down to up labels was significantly skewed at 0.0085, indicating a severe imbalance towards in-service (up) labels. After applying SMOTE, the distribution was equalized, resulting in an even representation of in-service and out-of-service instances.

Table 3 illustrates the impact of SMOTE on the classification performance of three models: Support Vector Machine (SVM), Random Forest, and LightGBM. The precision and recall values for the down (out-of-service) class improved significantly, demonstrating the effectiveness of SMOTE in overcoming the class imbalance issue.

The Support Vector Machine (SVM) classifier was employed to tackle the binary classification problem of ATM service status prediction. The SVM classifier optimizes the following objective function:

where $w$ represents the weight vector, $b$ denotes the bias term, $C$ is the regularization parameter, and $\zeta_{i}$ are slack variables that allow for classification errors. The SVM model works by maximizing the margin between the two classes (in-service and out-of-service) while penalizing any misclassifications.

The following parameters were configured for the SVM model:

• 1.

  Penalty Term (C): The regularization parameter was set to 1, striking a balance between maximizing the margin and minimizing classification errors.

• 2.

  Loss Function: The squared hinge loss was utilized, defined as:

  $$\sum_{i=1}^{n}\max(0,1-y_{i}(w^{\top}x_{i}+b))^{2}$$

  (5)

  This loss function penalizes misclassified samples that lie beyond the decision boundary, ensuring that the model places emphasis on difficult examples.

• 3.

  Kernel: A linear kernel was chosen for this study, which is effective for separating the two classes in high-dimensional feature space.

The performance of the SVM classifier was evaluated both before and after applying SMOTE. SMOTE significantly enhanced the precision, recall, and F1-score for the down (out-of-service) class.

This improvement underscores the importance of SMOTE in addressing class imbalance, allowing the SVM model to more effectively identify the minority class instances.

Table 4 presents the performance metrics for the SVM classifier trained with SMOTE.

The Decision Tree classifier was applied to solve the classification problem in this study. The model’s parameters are outlined as follows:

• 1.

  Criterion: The Gini impurity was used as the splitting criterion, which measures the quality of a split by evaluating the distribution of classes across the branches.

• 2.

  Minimum Samples for Split: A minimum of 2 samples was required to perform a split at each node, ensuring that each decision node has sufficient data to make a reliable split.

• 3.

  Minimum Samples for Leaf: The minimum number of samples in a leaf node was set to 1, which allows the model to grow deeper trees to capture finer details in the data.

The Decision Tree model creates branches by recursively splitting the data based on the feature that maximizes the Gini impurity reduction at each step. This process continues until the stopping criteria (minimum samples per split or leaf) are met, resulting in a tree structure that assigns a label to each leaf node, representing the predicted class (either in-service or out-of-service).

Table 5 presents the performance metrics of the Decision Tree classifier on the dataset.

The Bagging (Bootstrap Aggregation) classifier was implemented as an ensemble learning approach to enhance prediction accuracy by combining multiple base models. In this study, Decision Trees were used as the base learners for Bagging. This strategy stabilizes individual model predictions, reducing variance and improving overall reliability.

The key parameters used in the Bagging model were as follows:

• 1.

  Base Model: Decision Trees were chosen as the base models due to their high interpretability and ability to capture complex patterns.

• 2.

  Number of Base Learners: A total of three Decision Trees were trained as individual base learners, providing diverse perspectives on the data.

• 3.

  Max Samples: Each base learner was trained using 100% of the original dataset, ensuring complete representation and consistency.

• 4.

  Max Features: All features (100%) were used by each base learner, ensuring that each tree had access to the same set of information.

Bagging is particularly useful when base models are prone to overfitting, as in the case of Decision Trees, by aggregating predictions across multiple versions to generate a consensus output.

The performance metrics in Table 6 show that the Bagging classifier achieved high precision and recall for both the out-of-service and in-service classes, making it a reliable approach for minimizing false alarms and missed detections.

The LightGBM classifier, known for its efficiency and scalability, was utilized in this study to classify ATMs as either in-service or out-of-service. LightGBM is particularly effective for large datasets due to its fast training speed and low memory footprint. Additionally, it constructs trees leaf-wise rather than level-wise, enabling faster convergence and better loss reduction. It also handles categorical data effectively, making it a strong candidate for real-world applications.

The model parameters used for the LightGBM classifier were:

• 1.

  Number of Leaves: 31 (sets the maximum number of leaves in a single tree).

• 2.

  Max Depth: -1 (no limit on tree depth to ensure full growth).

• 3.

  Number of Estimators: 100 (indicates the total number of boosting iterations).

• 4.

  Random State: Not specified (controls randomness for reproducibility if set).

LightGBM’s leaf-wise growth strategy provides it with the ability to reduce more loss at each step, making it computationally more efficient than traditional gradient boosting methods. This characteristic is particularly beneficial when dealing with extensive datasets.

The performance metrics presented in Table 7 show that the LightGBM model maintained a high level of precision and recall for both in-service and out-of-service classes, achieving an overall accuracy of 93.98%. This highlights its robustness and capability in handling the ATM classification task with minimal errors.

CatBoost, another gradient boosting algorithm, was used to classify ATMs as either in-service or out-of-service. This model is particularly effective at handling categorical data without requiring extensive preprocessing, making it a robust choice for this study. CatBoost, developed by Yandex, is designed to reduce overfitting and perform well on datasets of varying sizes. Its key strength lies in its use of ordered boosting and symmetrical trees, which help minimize prediction bias and maintain model integrity even with complex data.

The CatBoost model was trained using the default parameters provided by the open-source CatBoost library, ensuring a reliable and standard approach for benchmarking its performance against other models.

Table 8 illustrates the performance of CatBoost, demonstrating high precision and recall for both the in-service and out-of-service categories, with an overall accuracy of 98.48%. This strong performance underscores CatBoost’s suitability for handling complex classification tasks, ensuring reliable and consistent results even with limited tuning.

Dynamic Classifier Selection with Local Accuracy (DCS LA) is a technique that dynamically selects the most suitable classifier for each test instance based on its local accuracy within a defined neighborhood of training samples. In this study, a pool of classifiers, comprising Decision Trees and Support Vector Machines (SVM), was used. The DCS LA method evaluates the performance of each classifier in the pool and chooses the one that demonstrates the highest local accuracy for the specific test instance.

The main parameters for configuring the DCS LA model are as follows:

• 1.

  Classifier Pool: The pool includes Decision Trees and Support Vector Machines (SVM).

• 2.

  Number of Neighbors: The local accuracy is determined using the 7 nearest neighbors for each test instance.

The performance of the DCS LA method is summarized in Table 9, highlighting its effectiveness in classifying ATMs into in-service or out-of-service categories.

The results in Table 9 show that DCS LA achieves an overall accuracy of 99.20%, making it a reliable choice for ATM status classification by dynamically selecting the best-performing classifier based on local accuracy.

Dynamic Ensemble Selection (DES) with K Nearest Oracles Eliminate (KNORAE) is a technique designed to identify the best ensemble of classifiers for a given test instance based on their local performance within a neighborhood of test samples. This method leverages the K nearest neighbors to assess the classifiers’ effectiveness, dynamically choosing the optimal ensemble for each prediction scenario.

The primary parameters for configuring the DES KNORAE model are outlined below:

• 1.

  Classifier Pool: Bagging classifiers were utilized as the base learners to construct the ensembles.

• 2.

  Number of Neighbors: The method considered the 7 nearest neighbors to determine classifier performance for each instance.

• 3.

  Voting Mechanism: Majority voting was employed to aggregate predictions from the chosen ensemble.

Table 10 outlines the performance metrics of the DES KNORAE method, demonstrating its efficiency in ATM status classification.

The results in Table 10 indicate that DES KNORAE achieves a high precision and recall, making it a robust method for ensuring accurate ATM status prediction through dynamic ensemble selection.

Random Forest is an ensemble learning technique that builds multiple decision trees (weak learners) and combines their outputs to generate a more robust and accurate prediction. Each tree is constructed using a random subset of the data, and the final decision is based on the majority vote of all the trees in the ensemble, which helps to mitigate overfitting and variance issues commonly found in single decision tree models.

The configuration parameters used for the Random Forest model in this study are as follows:

• 1.

  Number of Trees: A total of 100 decision trees were employed in the forest to ensure sufficient learning capacity.

• 2.

  Criterion: The Gini impurity index was chosen as the splitting criterion to measure the quality of the splits.

• 3.

  Minimum Samples per Leaf: Each leaf node was required to have at least one sample to prevent the model from creating overly complex trees.

Table 11 provides a summary of the performance metrics for the Random Forest classifier.

As seen in Table 11, the Random Forest model demonstrated high precision and recall for both classes, making it a reliable choice for ATM status classification.

The stacking classifier is an ensemble learning method that combines multiple classification models. In this paper, the stacking classifier technique was applied to improve predictive performance by leveraging the complementary strengths of several models [Wolpert, 1992]. In our study, a stacking classifier was implemented to enhance the accuracy of ATM status prediction.

• 1.

  Base Learners: The base classifiers in the study were Random Forest, LightGBM, and CatBoost. They were selected based on their excellent individual performances and diversity in learning algorithms. Random Forest is an ensemble of decision trees that reduces variance through bagging. LightGBM is an efficient gradient boosting framework optimized for speed and performance. CatBoost is another gradient boosting algorithm that handles categorical features effectively.

• 2.

  Meta-Learner: We used a Logistic Regression model as the meta-learner. Logistic Regression is applicable for binary classification tasks and effectively combines the predictions from various base learners.

• 3.

  Training Process:

  1. (a)

     First Level: The base learners were trained on the training dataset after the application of SMOTE to handle class imbalance. Each model was trained using their optimized parameters independently.

  2. (b)

     Second Level: The base learners’ predictions were used as input features to the meta-learner. The meta-learner minimized the cross-entropy loss to optimize the final prediction.

• 4.

  Parameter Settings:

  • (a)

    Random Forest: 100 trees, Gini impurity criterion, random state set to 1.

  • (b)

    LightGBM: 100 estimators, learning rate of 0.1, random state set to 1.

  • (c)

    CatBoost: 100 iterations, learning rate of 0.1, depth of 6, silent mode enabled.

  • (d)

    Logistic Regression: Using default setting with L2 regularization.

Performance Metrics: The performance of the stacking classifier is summarised in the following Table 12.

The overall accuracy of the stacking classifier was 99.29% among all models tested. For both classes, precision and recall were high, which is indicative of its performance in reducing both false alarms and missed alarms.

The application of SMOTE greatly improves classification with the use of this highly imbalanced dataset, especially for those models that were quite poor in detecting out-of-service instances. That is quite clearly visible in the case of the SVM model, for which the average F1-score was around 0.50 before the application of SMOTE—hence showing its inability to find instances from a minority class effectively. After applying SMOTE, the average F1-score of the SVM has risen to 0.86, indicating a great improvement in its capability of classification without big bias over either class.

Similarly, Random Forest and LightGBM showed remarkable improvements in average F1-scores after the application of SMOTE. The F1-score for Random Forest rose from 0.78 to 0.99, showcasing its strong adaptability to the resampled dataset. In its turn, LightGBM showed a great boost: its average F1-score improved from 0.73 to 0.94, reiterating how important this step was in enhancing LightGBM sensitivity toward out-of-service instances.

Figure 8 illustrates the effect of SMOTE on the average F1-scores of SVM, Random Forest, and LightGBM, indicating where it significantly improved after the application of SMOTE in either class.

The results reveal that ensemble methods such as Random Forest, Bagging, and DCS LA consistently outperformed simpler models like SVM and Decision Tree. These ensemble models leveraged their ability to combine predictions from multiple weak learners, resulting in higher accuracy and robustness for both in-service and out-of-service classifications. Additionally, the application of SMOTE was pivotal in enhancing recall for the minority class (Down), ensuring that the models could reliably detect out-of-service ATMs.

The integration of SMOTE with ensemble learning techniques delivered the best results, with Random Forest standing out as the top performer, achieving the highest F1-scores overall.