Towards an Ensemble Regressor Model for Anomalous ISP Traffic Prediction

Prediction of network traffic behavior in advance is immensely important for the effective management of modern telecommunication networks. It is equally vital for Internet Service Providers (ISP) for better resource management, efficient routing scheduling, short-term and long-term business capacity planning, advanced network design, and anomalous traffic detection and mitigation. An accurate traffic predictor framework can help ISP to manage their network proactively, which can ensure network Quality of Service (QoS) and Quality of Experience (QoE) [1]. The modern network predictor tools predict and forecast traffic based on their historical data collected over a long time. This traffic information is captured using different network flow collection protocols, e.g., NetFlow [2] and then converted the raw data into time-series format for further processing. The tools and techniques for processing temporal traffic information are extensively studied in the Time Series Forecasting domain [3].

There are mainly two different types of mathematical models for analyzing network traffic, such as linear and non-linear model [4]. Linear models can learn patterns from those time series data where the next data point can be viewed as a linear combination of the past values. We can model the linear time series data using different statistical models AutoRegressive Integrated Moving Average (ARIMA), Seasonal ARIMA (SARIMA), Holt-Winter, etc. There are also linear machine learning models, e.g., linear regression, for analyzing the linear components of the time series data. But the real internet traffic is much more complex, having non-linear components challenging to capture using a linear model. In that case, non-linear models are deployed to forecast the real IP traffic. These models can be a statistical model such as Threshold AutoRegressive (TAR) [5], Exponential AutoRegressive (ExpAR) [4] etc. or machine learning model such as Lasso regression, Ridge regression, SVM (Support Vector Machine) with a non-linear kernel, Quantile regression, Bayesian Inference, etc. Also, we can use neural network-based machine learning models such as multi-resolution Finite-Impulse-Response (FIR) model [6], Genetic Algorithm and Radial Based Function Network (GA-RBF) [7] etc., deep learning model, e.g., DNN (Deep Neural Network), CNN(Convolutional Neural Network) or sequence model such RNN (Recurrent Neural Network), LSTM (Long Short Term Memory), etc.

Real-world IP network traffic is susceptible to various external and internal factors which may abruptly change the normal traffic flow. Internal factors are related to ISP companies, such as introducing new services, traffic migration, speed up-gradation, etc. In contrast, external factors are related to external ISP events and conditions such as new internet applications, regional economic factors, seasonal effects, etc. However, the unexpected suspicious changes, differing significantly from most data points, are prevalent in real IP traffic known as outliers or anomalies. The errors or problems in the traffic data collection sensor might be a source of those anomalous data points in the traffic. Those abnormal data points in traffic flow might impact learning the general trend in data. As a result, the prediction model might produce wrong inferences in the future, considering them as normal behavior. Therefore, it is essential to identify and handle the anomalies/outlier in the internet traffic before applying any prediction model. In addition, it might help the prediction model increase the generalization capability. Machine Learning models have been used to predict the complex real IP traffic, and they compared their performance with the statistical model or deep learning model. However, the comparative performance analysis among various regression models and stacking ensemble model to predict anomalous internet traffic has not been studied considerably. This work extensively analyzes the real-world IP traffic to identify the abnormal data points and mitigate them before feeding them into the regression model. We focus on the performance evaluation of the regression model and their ensemble models in IP traffic prediction with and without the outlier. Also, we experimented with a different feature set for training our machine learning model to find the optimal number of input features. Our experimental results show better accuracy when the machine learning models are trained using the anomaly-adjusted data. The main contributions of this work are:

• •

  Outlier or anomaly detection using the standard statistical procedure and unsupervised learning method before using them to design our predictive model.

• •

  Finding the optimum number of features for the machine learning model.

• •

  Comparative performance analysis of different regression modeling techniques such as Extra Gradient Boosting, Light Gradient Boosting Machine (LightGBM), Cat Boosting (CatBoost), Stochastic Gradient Boosting (SGD), Gradient Boosting Regressor (GBR), and their stacking ensemble model with an outlier in the data.

• •

  Comparative performance analysis of the regression modeling techniques after adjusting the outliers in the data.

This paper is organized as follows. Section II describes the literature review of current traffic prediction using machine learning models. Section III presents the methodology, including dataset description, machine learning models explanation, anomaly identification process, and experiment details. Section IV summarizes the different machine learning methods’ performance and draws a comparative picture among prediction models with and without outliers in the dataset. Finally, section V concludes our paper and sheds light on future research directions.

S. Fischer et al. [8] proposed a traffic prediction system called DEEPFLOW that processes the ingress traffic data and produces prediction for all traffic flows using different machine learning techniques. The prediction model includes two different categories of models such as statistical model and neural network based deep learning model. They specifically focused on handling traffic volatility using non-neural model called VAR (Vector Autoregression). The average model prediction error was around 10% according to their experimental results which could have been improved. Their DEEPFLOW framework did not consider any regression machine learning model for the traffic prediction.

D. Szosta et al. [9] extensively investigated both machine learning classification and regression models for optical network traffic prediction. The proposed model considered four different supervised machine learning model tested on real traffic pattern collected from Seattle Internet Exchange Point. Total of five different datasets they used in their experiment to evaluate the prediction performance based on their proposed evaluation metric called Traffic Level Prediction Quality (TLPQ). The prediction models were trained on different feature set arranging from various window size and other feature such minute, day, and traffic values. Their experimental claims the outperformance of regression model over classification model in traffic prediction. Y. Xu et. al [10] proposed a Gaussian Process (GP) based machine learning model for real world traffic prediction. They compared the model performance against the state-of-the-art seasonal ARIMA (SARIMA) and sinusoid superposition. The computational complexity of extracting optimal hypermeters for the prediction model has also been reduced from $O(n^{3})$ to $O(n^{2})$. The GP based machine learning model shows the average prediction of 3% when predicting one-hour-ahead traffic and it’s increased to 5% when the prediction length is extended to ten-hours-ahead.

T. P. Oliveira et. al [11] experimented with two different machine learning model such as multi-layer perceptron and stacked autoencoder for traffic prediction. They used a dataset collected from a private ISP in European cities.They used two hidden layers of MLP (Multi Layer Perceptron) with 60 and 40 neurons, respectively, while they found the best result for four hidden layers SAE with 80, 60, 60, and 40 neurons. They trained their model for 1000 epochs in both MLP and SAE (Stacked Auto Encoder), although the SAE training was divided into two-stage as the unsupervised pre-training for 900 epochs and 100 epochs supervised training. Different prediction length has been investigated using their prediction model and the error was increasing with longer prediction length. Their experimental results show the better performance of the simpler MLP than the SAE deep neural network. Also, the MLP has taken lesser computational resource than SAE. D. Szostak et. al [12] formulated traffic prediction problem into a classification problem and their proposed classification model predicts traffic bitrates level instead of exact traffic volume. They used Linear Discriminant Analysis (LDA) classifier to forecast the future traffic and it shows 93% accuracy according to their experimental results. The experiment was conducted on the real data collected by Internet Exchange Point in Seattle. Y. Zang et. al [13] author proposed a traffic prediction model by combining K-means clustering, Elman-NN, and wavelet decomposition. They cluster the adjacent and correlated base stations using K-means so it can improve the overall prediction accuracy. They reconstructed traffic into high-frequency and low-frequency components using wavelet decomposition for feeding to their traffic prediction model. Finally, a three-layer feed-forward neural network ENN is used to train the decomposed traffic data for making the prediction.

From the above discussion, it is apparent that the machine learning is extensively used for traffic prediction. Many works compared the performance of the machine learing model with neural network model and statistical model. However, we found a lack of investigation in the performance comparision among differnt regressor model and their ensemble model. Also, the real world internet traffic is very erratic in nature due to different external and internal factors which inject abnormal data points in the traffic flow. In this work, we identify and mitigate the outlier before using them to train our model and compare the results with data having outlier. To the best of our knowledge, this is the first work in the internet traffic prediction domain that offers a comparative view from different regressor and their ensemble model performance with anomaly detection, mitigation, and input optimization.

First of all, we explained our data preprocessing module in subsection III-A to handle the missing values and analyze traffic characteristics. Then, we described the data windowing phase for making the compatible for supervised learning in section III-B. The anomaly detection and mitigation module is explained in subsection III-C. After that, we briefly described different regression model background in subsection III-D. The model performance evaluation metrics are described in subsection III-E. Finally, we summarize the configuration of our experimental environment in subsection III-F.

Our experiment considered 21 days of IP traffic to train our machine learning models, while the last eight days of traffic were used for testing. Our experiment has mainly two phases: I) anomaly detection and mitigation, and II) machine learning model implementation and performance evaluation.

Two different methods were applied to determine the abnormal data points in our traffic in phase I. The Three-Sigma rule identifies those data points as outliers which are three standards that deviated from the mean of the traffic data. Fig. 2 illustrates the inlier points wrapped by the upper and lower horizontal red line. The data points lie outside the top, and bottom red line boundary in 2 are identified as outliers according to the Three-Sigma rule.

Furthermore, an unsupervised learning algorithm called Isolation Forest is also used to identify the outliers, and the result is presented in Fig. 3. From 2 and 3, a similar patterns of outliers are identified in our traffic data. We adjusted the outliers identified by the Three-Sigma rule before using them to train our prediction models. The technique used for adjusting the outliers is called backward filling, in which the following valid data point replaces the outlier.

Finally, in phase II, we applied several machine learning models such as XGBoost, LightGBM, CatBoost, SGD, GBR, and an ensemble model based on stacking technique. We used five-fold cross-validation to train our models. The performance evaluation metrics of our experimental model are summarized in Table I. Two different versions of each model were investigated based on the data with and without outlier. We used a total of five different feature sets of equivalent traffic length of 30 minutes, 45 minutes, 60 minutes, 75 minutes, and 90 minutes for training our model to identify the optimum number of inputs for our prediction model. The best results for the individual model are marked bold and underlined in the table.

For the XGBoost model, we achieved the best prediction using 12 and 15 inputs respectively for model with and without outlier data. The best average deviation between actual and predicted traffic is 7.47% and 5.15% with and without outlier, respectively, which indicated more than 30% improvement in traffic prediction after adjusting outliers. In the LightGBM model, the minimum prediction error between actual and predicted traffic is 5.09% for 9 outlier-adjusted input variables.The prediction improvement is around 40% when compared with the best forecasting error of 8.51% for the prediction model having outliers in the training data. For other input variables 9, 12, 15, and 18, we have seen a similar amount, around 40% (8.51% to 5.11%, 8.47% to 5.13%, 8.47% to 5.14%, and 8.53% to 5.10% respectively) of prediction enhancements after adjusting the anomalies in the data. For SGD, we found the best prediction result with MAPE of 6.10% using input length of 6, and the result improved by more than 41% after removing the outliers compared with 10.44% error in prediction with outlier data. . The other input variables show a lower gap between actual and predicted traffic for outlier-adjusted data. The GBR model showed the minimum deviation of 5.20% when trained using thirty minutes of traffic data without the outlier. After mitigating abnormal data points, our experimental results depict more than 30% of better prediction for all input variables. In CatBoost model, the minimum prediction error of 5.08% is achieved using 9 input variables. The traffic prediction improvement for the CatBoost model is more than 30% compared with the minimum error of 7.56% in the case of outlier data. Finally, we ensemble the individual prediction model based on the stacking technique to further reduce the gap between actual and predicted traffic. According to our experimental result, the ensemble model is the best prediction model with a minimum prediction error of 7.23% (12 input variables) with outliers in the data and 5.04% (9 input variables) without outliers. Our ensemble model provided the lowest error amongst all the prediction models in both training strategies. A graphical representation of actual traffic and predicted traffic using the best performing ensemble model with and without outliers are shown in Fig. 4(a) and Fig. 4(b). Our experimental results show overall performance improvement for all considering machine learning models after adjusting the abnormal traffic. From our experimental result we can conclude, the ourlier mitigation in the dataset before trining the prediction model can improve the prediction by average 30% more accuracy. Our experimental result shows that outlier mitigation in the dataset before training the prediction model can improve the prediction by an average of 30% more accuracy.

The comparative analysis of the model accuracy (100-MAPE) based on different input lengths are depicted in Fig. 5. Our experimental results showed that the performance of SGD is more sensitive to the input lengths. The SGD model accuracy rises from input 6 to 9 then plummets from 15 to 18 when trained with outlier data. But in the other case, the accuracy decreases with the increased input length. There is a variation in the model accuracy with the input length for other models, but the magnitude is much lower than the SGD.

However, many internal and external factors can affect the regular traffic pattern in the real world. Since machine learning-based traffic prediction algorithms learn the general pattern in the dataset and predict accordingly, it is essential to handle the outliers before providing them to learn. Otherwise, there is a chance of learning from abnormal traffic patterns, affecting the prediction result. Our experimental results also showed that outliers in the data make the model performance poor than the clean data. Our results also showed the out-performance of the ensemble machine learning model than the individual model. Moreover, we noticed a varying model performance for different input lengths. Therefore, it is also crucial to optimize input size for the prediction model.

In this work, we experimented with several regression models in internet traffic prediction. A total of five individual regressor model such as XGBoost, LightGBM, SGD, GBR, CatBoost and one ensemble model were implemented to analyze our real-world traffic. We applied these models with and without outliers in traffic to demonstrate the impact of anomalies in traffic prediction. Also we demonstrated the outperformance of ensemble model than individual regressor model. Our experimental results show a significant performance improvement for all models after adjusting the abnormal data points in the original dataset. The ensemble model shows the minimum deviation between actual and predicted traffic with and without outlier in the data. In the real world, the internet traffic is very susceptible to the different external and internal factors which make the traffic non-linear and difficult to predict. The sudden changes in the data should be treated first before providing them into a machine learning model to predict. Otherwise, the model will be unable to generalize the actual traffic pattern, and predictions will not be effective. In the future, we would like to extend this work by multi-step traffic prediction. Also, we would explore different unsupervised methods in real-time anomaly detection and mitigation.