Contextual-Bandit Anomaly Detection for IoT Data in Distributed Hierarchical Edge Computing

We propose an adaptive model selection scheme to select the most suitable AD model based on the contextual information of input data, so that each data sample will be directly fed to its best-suited model. This is in contrast to traditional approaches where input data either (i) always go to one same model regardless of the hardness of detection [5], or (ii) are successively offloaded to higher layers until meeting a desired accuracy or confidence [7].

Our proposed adaptive scheme is a reinforcement learning algorithm that adapts its model selection strategy to maximize the expected reward of a model to be selected. We frame the learning problem as a contextual bandit problem [10, 11] and use a policy gradient method to solve it. See Fig. 2. Formally, given the contextual information $\mathbf{z_{x}}$ of an input data $\mathbf{x}$, and $K$ trained AD models deployed at the $K$ layers of an HEC system, we build a policy network that takes $\mathbf{z_{x}}$ as the input state and outputs a policy of selecting which model (or equivalently which layer of HEC) to do anomaly detection, in the form of a categorical distribution $\pi_{\theta}(\mathbf{a}|\mathbf{z_{x}})=\prod_{k=1}^{K}s_{k}^{a_{k}},$ where $\mathbf{a}$ is the actions encoded as a one-hot vector that defines which model (or HEC layer) to perform the task, $\mathbf{s}=(s_{1},s_{2},\cdots,s_{K})=f_{\theta}(\mathbf{z_{x}})$, is a vector representing the likelihood of selecting each model $k$. We denote the selected action as $|\mathbf{a}|=k$ if $k=\arg\max_{k}(s_{k})$.

The policy network $f_{\theta}(.)$ is designed as a neural networks with parameters $\theta$. To make the policy network small enough to run fast on IoT devices, we use the extracted features $\mathbf{z_{x}}$ instead of the raw input data $\mathbf{x}$, to represent the contextual information of input data. We train the policy network using policy gradient method [11, 10] to find an optimal policy $\pi$ that maps an input state $\mathbf{z_{x}}$ to an action (which model or layer) to minimize the negative expected reward:

$$\min L(\theta)=-\mathop{\mathbb{E}}_{\mathbf{a}\sim\pi_{\theta}}[R(\mathbf{a},\mathbf{z_{x}})],$$

where $R(\mathbf{a},\mathbf{z_{x}})$ is a reward function of action $\mathbf{a}$ given state $\mathbf{z_{x}}$. To reduce the variance of reward value and increase the convergence rate, we utilize reinforcement comparison [11] with a baseline $R(\tilde{\mathbf{a}},\mathbf{z_{x}})$. In order to encourage selecting an appropriate model that jointly increases accuracy and reduces the cost of offloading tasks further away from the edge, we propose a reward function as follows: $R(\mathbf{a},\mathbf{z_{x}})=\text{accuracy}(\mathbf{x})-C(\mathbf{a},\mathbf{x}).$ We define the cost function $C(\mathbf{a},\mathbf{x})$ as a function maps the end-to-end detection delay $t_{\text{e2e}}(\mathbf{x},\mathbf{a})$ to an equivalent accuracy in scale $[0,1]$ with intuition that a higher delay will result in a greater reduction of accuracy:

$\displaystyle C(\mathbf{a},\mathbf{x})=\frac{\alpha\cdot t_{\text{e2e}}(\mathbf{x},\mathbf{a})}{1+\alpha\cdot t_{\text{e2e}}(\mathbf{x},\mathbf{a})},$

(1)

where $\alpha$ is a tunable parameter. Further details are available in [3].

We evaluate our proposed approach with two public datasets. The data is standardized to zero mean and unit variance for all of the above training tasks and datasets.

Univariate dataset. We use a dataset on power consumption,²²2http://www.cs.ucr.edu/~eamonn/discords/which is used in [2, 9, 3]. For training details, please refer to our previous work [3].

Multivariate dataset. We use MHEALTH³³3http://archive.ics.uci.edu/ml/datasets/mhealth+dataset which consists of 12 human activities of 10 different people, and each person wore two motion sensors: one on left-ankle and the other on right-wrist. Each motion sensor contains a 3-axis accelerator, a 3-axis gyroscope, and a 3-axis magnetometer; hence the input data has 18 dimensions. The sampling rate of these sensors is 50 Hz. We use a window sequence of 128 time-steps ($\sim$2.56 second) with a step-size of 64. Adopting the common practice, we choose the dominant human activity (e.g., walking) as normal and treat the other activities as anomalous. For the AD task, we select 70% of normal samples of all the subjects (people) as the training set; and the rest 30% of normal samples plus 5% of each of the other activities as the test set. To train the policy network, we select 30% of normal samples and 5% of each of the other activities as the training set, and the whole dataset as the test set.

We use Tensorflow and Keras to implement the AD models (i.e., three AE models and three LSTM-seq2seq models as shown in Fig. 1a) and the policy network model. Note that, since the edge server and cloud server are empowered with GPU, we implement the LSTM-seq2seq-Edge and BiLSTM-seq2seq-Cloud models based on CuDNNLSTM units to accelerate training and inference time. Before deploying the LSTM-seq2seq-IoT and LSTM-seq2seq-Edge models on Raspberry Pi3 and Jetson-TX2, we compress them by (i) removing the trainable nodes from the graph, and convert variables into constants; (ii) quantizing the model parameters from floating-point 32-bit (FP32) to FP16. We observe no performance decrease of these compressed AD models running on Raspberry Pi3 and Jetson-TX2.

The policy network requires low complexity and needs to run fast on IoT devices, so the state input to the policy network needs to be small but still well represent the whole sequence of input data. For the univariate data, we define the contextual state as an extracted feature vector which includes min, max, mean, and standard deviation of each day’s sensor data. For the multivariate data, we use the encoded states of the LSTM-encoder to represent the input state for the policy network. Subsequently, we build the policy network as a single hidden neural network with 100 hidden units and an output layer with 3 units. We empirically select $\alpha=0.0005$ for the univariate dataset, and $\alpha=0.00035$ for the multivariate dataset to calculate the cost of executing detection as given by (1).

The software architecture of our demo is shown in Fig. 1b. It consists of a GUI, the adaptive model selection scheme based on the policy network, and the three AD models at the three layers of HEC. The GUI allows a user to select which dataset and model selection scheme to use, as well as to tune parameters, as shown in Fig. 3a, and displays the sensory raw signals and performance results, as shown in Fig. 3b. All the communication services use keep-alive TCP sockets to reduce the overhead of connection establishment. Network latency as shown in Fig. 1a is configured by using Linux traffic control tool, tc, to emulate the WAN connections in HEC. The hardware setup for the HEC testbed is shown in Fig. 4.

User Actions: As shown in Fig. 3a, we allow users (e.g., ICDCS demo session participants) to evaluate the HEC testbed performance with (i) either univariate or multivariate datasets, (ii) different fractions of normal and abnormal data in the datasets to use, and (iii) different schemes under evaluation: (1) always detects anomaly at IoT Device, (2) always offloads detection tasks to Edge server, (3) always offloads to Cloud, (4) Successive, i.e., executes at IoT devices first and then offloads to higher layers successively until reaching a confident output or the cloud, and (5) Adaptive which is our proposed adaptive model selection scheme. After the user clicks “Start”, our result panel as in Fig. 3b will show the continuously updated raw sensory data (accelerometer, gyroscope, magnetometer), anomaly detection outcome (0 or 1) vs. ground truth, detection delay vs. the actions determined by our policy network, and the accumulative accuracy and F1-score.