Online Deep Learning based on Auto-Encoder

In online learning scenario, at each time instant $t$ the agent acquires the example ${{x_{t}}={({x_{t,1}},{x_{t,2}},\ldots,{x_{t,n}})^{T}}}\in\mathbb{R}^{D_{X}}$.An agent was required to predict its corresponding output ${y_{t}}\in\mathbb{R}^{D_{Y}}$ through learnnig a mapping $f(x)$ according to information of the previous input-output sequence $({x_{1}},{y_{1}}),({x_{2}},{y_{2}}),\ldots,({x_{t-1}},{y_{t-1}})$.For online deep learning setup, we use the deep neural network to replace the input-output mapping $f(x)$.After obtaining the prediction value ${\hat{y}_{t}}\in\mathbb{R}^{D_{Y}}$ from the deep neural network, the agent would receive the real output value ${y_{t}}\in\mathbb{R}^{D_{Y}}$ from the environments. By calculating the loss between the prediction and the real value, the agent suffers the prediction loss. Then the loss information would be fed back to online learning algorithm to guide the update process of the model parameters. If the model predicts incorrectly, the parameters of the model are dynamically adjusted according to prediction loss. Therefore, online learning can reflect real-time changes more timely.

There are redundancy and noise information in the input instances, these redundant and noisy information not only make the prediction inaccurate, but also increase the calculation cost, so it is necessary to remove redundancy and noise from input instances, and find compact and non-redundant representation. To our knowledge, at present, in the field of online learning, there is little research on how to use deep learning to learn input hidden representation and realize feature selection, and then use the hidden representation to predict class labels of instances.

Hence, we utilize autoencoder to extract abstract hierarchical features from example, yield input example’s hidden representation.

The encoder of autoencoder receives the input sequentially and builds a fixed-length latent vector representation (denoted as ${h_{L}}\in\mathbb{R}^{{D^{\prime}}_{X}}$ in Fig.1). Conditioned on the encoded latent representation, the decoder of autoencoder generates the reconstructed input (denoted as $\hat{x}\in\mathbb{R}^{D_{X}}$ in Fig.1). The deviation between the input and reconstructed one was described by mean square error ${L_{re}}(x,\hat{x})$, which is defined as:

We then update the parameters of the auto-encoder according to the reconstruction loss function.

In this subsection, we propose two different data fusion approaches, one is output-level fusion, and the other one is feature-level fusion. In the first fusion scheme, the $L+1$ classifiers is constructed separately from outputs of encoder’s $L+1$ hidden layers, and then they are combined and merged into an ensemble classifier through the weighted fusion strategies. In the second fusion scheme, each hidden layer is given different alignment weights by self-attention mechanism, and then they are integrated into a context vector, which is used as input to directly generate the final classifier.

To simplify the representation, in the following section, we use ODLAE-1 and ODLAE-2 to denote the output-level and feature-level fusion strategies respectively.

Output-Level Fusion As seen in Fig.1, encoder of autoencoder is composed with hidden layers in the Fig.1. The transformation can be formulated as

where ${s_{f}}$ is the activation functions of the encoder and decoder, the parameters for such an autoencoder are generally defined as: ${W_{0}}\in\mathbb{R}^{{{D^{\prime}}_{X}}\times{D_{X}}}$, ${W_{l}}\in\mathbb{R}^{{{D^{\prime}}_{X}}\times{{D^{\prime}}_{X}}}$ and ${\hat{W}_{0}}\in\mathbb{R}^{{{D^{\prime}}_{X}}\times{D_{X}}}$, ${\hat{W}_{l}}\in\mathbb{R}^{{{D^{\prime}}_{X}}\times{{D^{\prime}}_{X}}}$, $l=0,1,\ldots,L$ are the weight matrices of encoder and decoder, respectively.${b_{l}}\in\mathbb{R}^{{{D^{\prime}}_{X}}}$, $l=0,1,\ldots,L$ and ${\hat{b}_{0}}\in\mathbb{R}^{D_{X}}$, ${\hat{b}_{l}}\in\mathbb{R}^{{{D^{\prime}}_{X}}}$, $l=0,1,\ldots,L$ is the bias vectors of encoder and decoder, respectively.

Each hidden layer’s output $l=0,1,\ldots,L$ is used directly as input to construct $L+1$ base classifiers

where ${c_{l}}\in\mathbb{R}^{{D_{Y}}\times{{D^{\prime}}_{x}}}$ is the weight matrix, and ${b_{cl}}\in\mathbb{R}^{D_{Y}}$ is the correspoding bias vector.

In the scenario of online learning, we could regard each base classifier as an expert. We then assign weights ${\beta_{l}}$, $l=0,1,\ldots,L$, to each base classifier according to the prediction performance of whole ensemble classifier, which is our predicted output value:

Note that ${\beta_{l}}$ denotes the weight vector of base classifier ${f_{l}}$, the weight value represents the contribution rate of each classifier to the final classification result.

In this paper, the combination of online gradient descent and multiplicative updating rules are applied to update each classifier’s weight ${\beta_{l}}$, $l=0,1,\ldots,L$, the concrete expression for iterative formula of classifier’s weight ${\beta_{l}}$ is as follows:

where ${\theta_{0}}\in(0,1)$ is a discount factor. When calculating the parameter update ${\beta_{l}}^{t+1}$, $l=0,1,\ldots,L$ at the next time $t+1$, ${L_{pre}}(f_{l}^{t},{y_{t}})$ represents the cross entropy loss between the predictive output of each classifier and the real output value at the current time $t$. Thus each classifier’s weight is updated by a factor of $theta_{0}^{{L_{pre}}(f_{l}^{t},{y_{t}})}$. The initial weight vector ${\beta_{l}}$, $l=0,1,\ldots,L$, may be arbitrary, and can be regarded as a “prior” over the set of expert opinions. Generally, we can set all of the initial weights equally to $1/(L+1)$, which would ensure the value of the next instant is must be in the open interval (0, 1). The new classifier’s weight values would be updated sequentially according to the cross entropy loss.

After calculating the loss ${L_{pre}}(f_{l}^{t},{y_{t}})$ between the real output value ${y_{t}}$ and the predictive value of each classifier $f_{l}^{t}$ , we also need to calculate the loss ${L_{pre}}({y_{t}},{\hat{y}_{t}})$ between the real output value ${y_{t}}$ and the weighted sum of each base classifier ${\hat{y}_{t}}$ and regard it as the whole prediction loss.

As for the other parameters in the network, online gradient descent method is adopted to update these parameters.

where $\Theta=\{{c_{l}},{b_{cl}},{w_{l}},{b_{l}},{w^{\prime}_{l}},{b^{\prime}_{l}}\}$ represents the set of parameters and ${L_{total}}$ is the total loss, the details of $\Theta=\{{c_{l}},{b_{cl}},{w_{l}},{b_{l}},{w^{\prime}_{l}},{b^{\prime}_{l}}\}$ can be seen in subsection 3.4.

The overall specific implementation process is presented in Algorithm 1. To simplify the representation, we use ODLAE-1 to represent the output-level fusion method.

Feature-Level Fusion Fig.2(a) shows the feature fusion process of ODLAE-2, instead of directly constructing the $L+1$ classifiers for the $L+1$ outputs of the hidden layers, we first fusion the $L+1$ outputs of the hidden layers to form context latent representation. At each time, we first concatenate them to form a latent representation matrix $H=({h_{0}},{h_{1}},\ldots,{h_{L}})$, where $H$ is $(L+1)\times{D^{\prime}_{X}}$ matrix, as illustrated by Fig.2(b). The self-attention mechanism [31, 32] use $H$ as input, and obtain the alignment weight vector $A=[{a_{0}},{a_{1}},\ldots,{a_{L}}]$ through a softmax layer:

Where ${W_{s1}}\in\mathbb{R}^{{d_{a}}\times{{D^{\prime}}_{X}}}$ is the weight matrix, and ${w_{s2}}\in\mathbb{R}^{{d_{a}}}$, the dimension ${d_{a}}$ is predefined hyperparameter.$A$ is a $\left({L+1}\right)\times 1$ alignment weight vector, which represent the contributions of the columns ${h_{i}}$ of $H$, and because the role of softmax, it ensures that the sum of components in equals to 1.

The feature fusion context vector $C$ can be obtained by multiplying the alignment weight vector and the latent representation matrix $H$:

Finally, the results of feature fusion are fed into the classifier output layer:

In this way, we get our prediction output ${\hat{y}_{k}}=f$.Moreover, the cross-entropy loss is applied to get the prediction loss ${L_{pre}}({y_{t}},{\hat{y}_{t}})$.

As for the other parameters in the network, online gradient descent method is adopted to update parameters.

where $\Omega=\left\{{{w_{l}},{b_{l}},{{w^{\prime}}_{l}},{{b^{\prime}}_{l}},{W_{s1}},{w_{s2}},{W_{f}},{b_{f}}}\right\}$ represents the set of parameters of the network and ${L_{total}}$ is the total loss, the details of ${L_{total}}$ can be seen in subsection 3.4.

The overall process is presented in Algorithm 2. To simplify the representation, we use ODLAE-2 to denote the algorithm of the feature-level fusion.

In this section, we would introduce our objective function, which is divided into two components: the prediction loss ${L_{pre}}({y_{t}},{\hat{y}_{t}})$ and reconstructing loss ${L_{re}}({x_{t}},{\hat{x}_{t}})$. The prediction loss${L_{pre}}({y_{t}},{\hat{y}_{t}})$ between the output’s prediction value (label) and the true value (label) is represented by the cross-entropy function:

Meanwhile,considering reconstruction loss ${L_{re}}({x_{t}},{\hat{x}_{t}})$, which is introduced in Section 3.2, we could project data to a hidden space that is of lower dimensionality, and therefore make the learning representation more compact and more powerful, fully exploit existing underlying information of input, in the nutshell, our goal is to predict output labels more accurately, at the same time, learn a good implicit representation, so we define the following loss function as the whole objective function:

where ${a_{re}}$ and ${a_{pre}}$ represent the trade-off parameters between the reconstructing loss and prediction loss, respectively, where ${a_{re}}\leftarrow\frac{{{a_{re}}\cdot{\beta_{re}}^{Lre}}}{{{a_{re}}\cdot{\beta_{re}}^{Lre}+{a_{pre}}\cdot{\beta_{pre}}^{Lpre}}}$ and ${a_{pre}}\leftarrow\frac{{{a_{pre}}\cdot{\beta_{pre}}^{Lpre}}}{{{a_{re}}\cdot{\beta_{re}}^{Lre}+{a_{pre}}\cdot{\beta_{pre}}^{Lpre}}}$ respectively. ${\beta_{re}}\in(0,1)$ and ${\beta_{pre}}\in(0,1)$ is the discount rate parameter,${L_{re}}\in(0,1)$ and ${L_{pre}}\in(0,1)$. Hence, trade-off parameters of the reconstruction loss and the prediction loss is discounted by a factor of ${\beta_{re}}^{Lre}$ and ${\beta_{pre}}^{Lpre}$ respectively in every iteration. Then, the trade-off parameters are normalized by dividing the sum of these two trade-off parameters.

AOILAE starts the learning process by establishing an autoencoder as the underlying structure. We use fully-connected layers as the encoder and decoder, Relu is used as the activation function for each hidden layer. The entire network parameters are updated using the Adam optimizer with a learning rate of 0.01[37]. In the feature-level fusion part, the parameter ${d_{a}}$ for self-attention mechanism is a user-defined parameter, in our experiments, we set it to 30 to get good results.

To verify the effectiveness and evaluate the performance of the proposed algorithm, we conduct various experiments on stationary datasets and non-stationary datasets. In this section, seven different datasets are chosen to verify our algorithm, which varies in the scales, feature dimensions, and the number of classes. The attributes of the datasets are shown in Table 1, and details about these data sets are summarized as follows:

(1) forestcovtype dataset[38]: The classification task on this data is to predict forest cover types only from cartographic variables without remote sensing data, the actual forest cover type was provided by Resource Information System (RIS) of US Forest Service (USFS) Region 2. The data is in its original form (non-scale) and contains binary (0 or 1) columns of data with qualitative independent variables, such as wilderness area and soil type. Because the input distribution would change with time, the data contains covariate drift and belongs to non-stationary data.

(2) gesture dataset[39]: This dataset is composed of seven features extracted from gesture videos to study the phase segmentation of gestures. Each video is represented by two files: an original file that contains the location of the user’s hands, wrists, head, and spine in each frame; and a processed file that contains the velocity and acceleration of the hand and wrist.

(3) mnist dataset[40]: This dataset contains 70K samples. These numbers are dimensionally standardized and located in the center of the image, which is of a fixed size (28x28 pixels) with a value between 0 and 1. For simplicity, each image is flattened and converted into a one-dimensional numpy array of 784 (28 * 28) features.

(4) rotatedmnist dataset[41]: By rotating the original sample, the extension of the traditional mnist problem[40] is formed, which leads to the abrupt drift of the concept. Specifically, handwritten digits are rotated to any angle in the range of -$\pi$ to $\pi$, resulting in covariate drift.

(5) permutedmnist dataset[15]: It is a modification of the MNIST datasets [18] which performs the permutations of pixels in mnist after quantization, in fact, it uses a group of random indexes to scramble the position of each element in the vector, and different random indexes produce different tasks. Hence, the real drift [42] is present in this dataset.

(6) rfid dataset[43]: Indoor RFID location data is a multi-class problem, which aims to identify the location of objects in the manufacturing workshop by using three input attributes. RFID readers are placed in different locations and four areas are created in the manufacturing workshop, resulting in four categories problems.

(7)N-Balo dataset[44]: This dataset solves the lack of public botnet datasets, especially the Internet of things. It shows the real traffic data collected from 9 Commercial Internet of things devices. Because the malicious data can be divided into 10 kinds of attacks carried by two botnets and one kind of benign attack. Therefore, it belongs to multi-category data set.

ODLAE algorithm is a progressive learning approach. Because the total loss is composed of two components, namely prediction and reconstruction loss, how to reasonably allocate the proportion of these two losses in the total loss has a great impact on the accuracy rate, which can effectively ensure the communication between the old and new data, resulting positive back propagation. There are two options for us to choose from: one is fixed allocation weight value selection strategy and the other is change the weight value according to the loss of the previous moment selection strategy. As discussed in the previous section, we use the method of constantly adjusting the trade-off parameters, and the changing total loss function constantly updates the network parameters through back-propagation.

The weight value selection strategy strategy balances the trade off between prediction loss and reconstruction loss. On the one hand, we want the model to explore and learn new knowledge as much as possible to reduce the prediction loss. On the other hand, we also hope that the model will not forget its own input characteristics and reduce the loss of reconstruction. In addition, because the probability distribution of data in the data stream may change, we need to update the total loss function dynamically according to the different characteristics of the data.

To further illustrate that the adaptive change scheme is better than the method of fixing the trade-off parameters, we designed several groups of comparative experiments. We set these two parameters to $({a_{re}},{a_{pre}})$ = (0.1,0.9), (0.2,0.8), (0.3,0.7), (0.4,0.6), ….(0.9,0.1) and the dynamic change strategy is set as $(\frac{{{a_{re}}\cdot{\beta_{re}}^{Lre}}}{{{a_{re}}\cdot{\beta_{re}}^{Lre}+{a_{pre}}\cdot{\beta_{pre}}^{Lpre}}},\frac{{{a_{pre}}\cdot{\beta_{pre}}^{Lpre}}}{{{a_{re}}\cdot{\beta_{re}}^{Lre}+{a_{pre}}\cdot{\beta_{pre}}^{Lpre}}})$ respectively.

As we can see in Fig.3, this illustrates that our proposed tuning strategy that dynamically controls the equilibrium between prediction and reconstruction loss can help significantly reduce of the error rate of the algorithm. Specifically, the best performance is achieved by changing these two trade-off parameters adaptively. This enables the model to dynamically change the overall loss according to its continuous learning performance on input data, and then constantly update the model, improving the learning effect of the model.

Similarly, we have done the same experiments on the algorithm ODLAE-2, as we can see in Fig.4, for smaller ${a_{re}}$, that is, the reconstruction loss accounts for a small proportion of the total loss, the network tends to forget its original input characteristics while learning the data, however, when we set ${a_{pre}}$ to a smaller value, it means that we rely too much on the original characteristics of data and neglect to explore new knowledge. However, we should take different measures for different data, hence, Our strategy with changing trade-off parameters is better than the general strategy with fixed trade-off parameters.

Variations in $L$ and ${D_{X^{\prime}}}$: $L$ represents the hidden layers and ${D_{X^{\prime}}}$ represents the hidden units of the autoencoder. These two parameters controls the model control the complexity of the network. The hidden layers and hidden units of the network is very important to the performance of the model. When these two parameters of network layers is increased, the network can extract more complex feature patterns. Therefore, when the model is more complex, the theoretical results can be better. However, the degradation of the deep network may occur, the accuracy is saturated or even decreased.

Therefore, we use the method of super parameter retrieval to find the ideal super parameters on each dataset. First, we fix the number of hidden units of the autoencoder ${D_{X^{\prime}}}$ to 32, and then increase the number of hidden layers from 2 layers to 5 layers to find out the optimal number of layers. Then we fix the optimal number of layers $L$ of the autoencoder and increase the number of hidden layer units ${D_{X^{\prime}}}$ from 32, 64, 128, to 256 to find the best combination $\left({L,{D_{X^{\prime}}}}\right)$. As we can see in Fig.5, the horizontal axis represents different combinations of experiments. On forestcovtype dataset, we find the best couple $\left({L,{D_{X^{\prime}}}}\right)=(5,128)$ in ODLAE-1 algorithm, and $\left({L,{D_{X^{\prime}}}}\right)=(5,64)$ in ODLAE-2 algorithm, And these optimal parameter combinations are used as the fixed parameter settings of the follow-up experiments.

We extensively compare the proposed technique with existing state-of-the-art methods for online learning. These include Relaxed Online Maximum Margin Algorithm[ROMMA] and its variation aggressive ROMMA algorithms(AROMMAS)[45], Adaptive Regularization of Weights(AROW)[46], confidence weighted algorithms (CW)[47], Soft Confidence Weight Learning algorithm (SCW1 and SCW2)[48], Online Gradient Descent algorithms(OGD)[49], Passive-Aggressive learning algorithms for Multiclass(PAM) and its variations PAM1 and PAM2 [50], Perceptron Algorithms with Max-score update(PerceptronM), Perceptron Algorithms with similarity-score update(PerceptronS), Perceptron Algorithms with uniform update(PerceptronU)[51], Online Deep Learning: Learning Deep Neural Networks on the Fly(ONN)[52].

Fig. 6 compares different methods on different datasets, the results indicate the superior performance of the proposed method. Compared with the second best method on the rotatedmnist dataset, our approach achieves a relative gain of 3.5% and 4.77% respectively. For the case of N-Balo dataset, our performance is only 2.35% and 2.25% below the SCW1 approach. In addition, the predictive results of different algorithms on these datasets are also listed in Table 2, and in Table 2, we provide more comparisons on the performance on different datasets, such as precision, F-1, and haming loss. For each evaluation index, our algorithm has achieved good results. The best performances are indicated in the bold.

Generally speaking, a good encoding representation should be devoted to capturing useful and stable structures from maybe partially collapsed inputs with unknown distribution. In this section, motivated by this principle, in order to further improve the robustness of the algorithm to learning latent representation of partially corrupted input data. We introduce and motivate a new training principle for ODLAE based on the idea of Denoising Auto-Encoder(DAE), a more recent variant of the basic autoencoder.

This is done by first corrupting the initial input $x$ into $\tilde{x}$ in the form of a stochastic mapping $\tilde{x}\sim{q_{D}}(\tilde{x}\left|x\right.)$.Then the collapsed input $\tilde{x}$ is then mapped, as the same process as auto-encoder does before, to a hidden representation

where ${s_{g}}$ is a non-liner activation function, ${W^{(0)}}\in\mathbb{R}^{{{D^{\prime}}_{X}}\times{D_{X}}}$, and ${W^{(l)}}\in\mathbb{R}^{{{D^{\prime}}_{X}}\times{{D^{\prime}}_{X}}}$, $l=1,2,\ldots,L-1$ are the weight matrix, and ${b^{(0)}}\in\mathbb{R}^{{{D^{\prime}}_{X}}}$, $l=1,\ldots,L-1$ are the bias vector, from which we could then obtain the reconstructed input $z$:

where ${W^{\prime(0)}}\in\mathbb{R}^{{D_{Y}}\times{{D^{\prime}}_{X}}}$ and ${W^{(l)}}\in\mathbb{R}^{{{D^{\prime}}_{X}}\times{{D^{\prime}}_{X}}}$, $l=1,2,\ldots,L$ are the weight matrix, and ${b^{\prime(0)}}\in\mathbb{R}^{{D_{Y}}}$ and ${b^{(l)}}\in\mathbb{R}^{{{D^{\prime}}_{X}}}$, $l=1,2,\ldots,L$ are the bias vector. The DAE training process consists of finding parameters $\Xi=\{{W^{(l)}},{W^{\prime(l)}},{b^{(l)}},{b^{\prime(l)}}\}$, $l=0,1,\ldots,L$ which minimize the reconstruction error. This corresponds to minimizing the following the squared error loss ${L_{re}}(x,z)=\left\|{x-z}\right\|_{2}^{2}$. Other different from the ODLAE-1 and ODLAE-2 is that, we only replace the auto-encoder in ODLAE-1 or ODLAE-2 with the denoising auto-encoder, the remaining part is consistent with the ODLAE-1 and ODLAE-2, so for convenience, we dub it as ODLDAE-1and ODLDAE-2 respectively.

As we can see in Fig.6, we compare the accuracies of different algorithms on different datasets with and without adding noise. The results indicate superior performance of the proposed method in all datasets, including N-Balo dataset, From the Fig.7, we can see that the accuracies of ODLDAE-1and ODLDAE-2 algorithms are slightly inferior to SCW1 algorithm when noise data is not added on denoising autoencoder, but the accuracies of our algorithm are higher than that of SCW1 algorithm after adding noise to data, and the accuracies of ODLDAE-1and ODLDAE-2 algorithms are higher than other baseline algorithms on other datasets. In addition, the accuracies of different algorithm on different dataset with adding noise are listed in the Table3. The best performances are indicated in the bold.