SenseFi: A Library and Benchmark on Deep-Learning-Empowered WiFi Human Sensing

In WiFi communication, channel state information reflects how wireless signals propagate in a physical environment after diffraction, reflections, and scattering, which describes the channel properties of a communication link. For modern wireless communication networks following the IEEE 802.11 standard, Multiple-Input Multiple-Output (MIMO) and Orthogonal Frequency Division Multiplexing (OFDM) at the physical layer contribute to increasing data capacity and better orthogonality in transmission channels affected by multi-path propagation. As a result, current WiFi APs usually have multiple antennas with many subcarriers for OFDM. For a pair of transmitter and receiver antennas, CSI describes the phase shift of multi-path and amplitude attenuation on each subcarrier. Compared to received signal strength, CSI data has better resolutions for sensing and can be regarded as “WiFi images” for the environment where WiFi signals propagate. Specifically, the Channel Impulse Response (CIR) $h(\tau)$ of the WiFi signals is defined in the frequency domain:

where $\alpha_{l}$ and $\phi_{l}$ denote the amplitude and phase of the $l$-th multi-path component, respectively, $\tau_{l}$ is the time delay, $L$ denotes the number of multi-paths and $\delta(\tau)$ is the Dirac delta function. To estimate the CIR, the OFDM receiver samples the signal spectrum at subcarrier level in the realistic implementation, which represents amplitude attenuation and phase shift via complex number. In WiFi sensing, the CSI recording functions are realized by specific tools [1, 2]. The estimation can be represented by:

where $||H_{i}||$ and $\angle H_{i}$ are the amplitude and phase of $i$-th subcarrier, respectively.

The number of subcarriers is decided by the bandwidth and the tool. The more subcarriers one has, the better resolution the CSI data is. Existing CSI tools include Intel 5300 NIC [1], Atheros CSI Tool [2] and Nexmon CSI Tool [35], and many realistic sensing platforms are built on them. The Intel 5300 NIC is the most commonly used tool, which is the first released CSI tool. It can record 30 subcarriers for each pair of antennas running with 20MHz bandwidth. Atheros CSI Tool increases the CSI data resolution by improving the recording CSI to 56 subcarriers for 20MHz and 114 subcarriers for 40MHz, which has been widely used for many applications [5, 22, 6, 9, 36]. The Nexmon CSI Tool firstly enables CSI recording on smartphones and Raspberry Pi, and can capture 256 subcarriers for 80MHz. However, past works [37, 38] show that their CSI data is quite noisy, and there do not exist common datasets based on Nexmon. In this paper, we only investigate the effectiveness of the deep learning models trained on representative CSI data from the widely-used Intel 5300 NIC and Atheros CSI Tool.

In general, the CSI data consists of a vector of complex number including the amplitude and phase. The question is how we process these data for the deep models of WiFi sensing? We summarize the answers derived from existing works:

1. 1.

   Only use the amplitude data as input. As the raw phases from a single antenna are randomly distributed due to the random phase offsets [39], the amplitude of CSI is more stable and suitable for WiFi sensing. A simple denoising scheme is enough to filter the high-frequency noise of CSI amplitudes, such as the wavelet denoising [22]. This is the most common practice for most WiFi sensing applications.

2. 2.

   Use the CSI difference between antennas for model-based methods. Though the raw phases are noisy, the phase difference between two antennas is quite stable [9], which can better reflect subtle gestures than amplitudes. Then the CSI ratio [40] is proposed to mitigate the noise by the division operation and thus increases the sensing range. These techniques are mostly designed for model-based solutions as they require clean data for selecting thresholds.

3. 3.

   Use the processed doppler representation of CSI. To eliminate the environmental dependency of CSI data, the body-coordinate velocity profile (BVP) is proposed to simulate the doppler feature [34] that only reflects human motions.

In our benchmark, as we focus on the learning-based methods, we choose the most common data modality (i.e., amplitude only) and the novel BVP modality that is domain-invariant.

As shown in Figure 2, the CSI data for human sensing is composed of two dimensions: the subcarrier and the packet number (i.e., time duration). For each packet or timestamp $t$, we have $X_{t}=N_{T}\times N_{R}\times N_{sub}$ where $N_{T}$, $N_{R}$ and $N_{sub}$ denote the number of transmitter antennas, receiver antennas and subcarriers per antenna, respectively. This can be regarded as a “CSI image” for the surrounding environment at time $t$. Then along with subsequent timestamps, the CSI images form a “CSI video” that can describe human activity patterns. To connect CSI data with deep learning models, we summarize the data properties that serve for a better understanding of deep model design:

1. 1.

   Subcarrier dimension $\to$ spatial features. The values of many subcarriers can represent how the signal propagates after diffraction, reflections, and scattering, and thus describe the spatial environment. These subcarriers are seen as an analogy for image pixels, from which convolutional layers can extract spatial features [41].

2. 2.

   Time dimension $\to$ temporal features. For each subcarrier, its temporal dynamics indicate an environmental change. In deep learning, the temporal dynamics are usually modeled by recurrent neural networks [42].

3. 3.

   Antenna dimension $\to$ resolution and channel features. As each antenna captures a different propagation path of signals, it can be regarded as a channel in deep learning that is similar to RGB channels of an image. If only one pair of antennas exists, then the CSI data is similar to a gray image with only one channel. Hence, the more antennas we have, the higher resolution the CSI has. The antenna features should be processed separately in convolutional layers or recurrent neurons.

Multilayer perceptron (MLP) [68] is one of the most classic architectures and has played the classifier role in most deep classification networks. It normally consists of multiple fully-connected layers followed by activation functions. The first layer is termed the input layer that transforms the input data into the hidden latent space, and after several hidden layers, the last layer maps the latent feature into the categorical space. Each layer is calculated as

where $W_{i}$ is the parameters of $\Phi_{i}$, and $\sigma(\cdot)$ is the activation function that aims to increase the non-linearity for MLP. The input CSI has to be flattened to a vector and then fed into the MLP, such that $x\in\mathbb{R}^{N_{s}T}$. Such a process mixes the spatial and temporal dimensions and damages the intrinsic structure of CSI data. Despite this, the MLP can still work with massive labeled data, because the MLP has a fully-connected structure with a large number of parameters, yet leading to slow convergence and huge computational costs. Therefore, though the MLP shows satisfactory performance, stacking many layers in MLP is not common for feature learning, which makes MLP usually serve as a classifier. In WiFi sensing, MLP is commonly utilized as a classifier [5, 46, 7, 32, 44, 34].

Convolutional neural network (CNN) was firstly proposed for image recognition tasks by LeCun [69]. It addresses the drawbacks of MLP by weight sharing and spatial pooling. CNN models have achieved remarkable performances in classification problems of 2D data in computer vision [70, 71] and sequential data in speech recognition [72] and natural language processing [73]. CNN learns features by stacking convolutional kernels and spatial pooling operations. The convolution operation refers to the dot product between a filter $\mathbf{k}\in\mathbb{R}^{d}$ and an input vector $\mathbf{v}\in\mathbb{R}^{d}$, defined as follows:

The pooling operation is a down-sampling strategy that calculates the maximum (max pooling) or mean (average pooling) inside a kernel. The CNNs normally consist of several convolutional layers, max-pooling layers, and the MLP classifier. Generally speaking, increasing the depth of CNNs can lead to better model capacity. Nevertheless, when the depth of CNN is too large (e.g., greater than 20 layers), the gradient vanishing problem leads to degrading performance. Such degradation is addressed by ResNet [74], which uses the residual connections to reduce the difficulty of optimization.

In WiFi sensing, the convolution kernel can operate on a 2D patch of CSI data (i.e., Conv1D) that includes a spatial-temporal feature, or on a 1D patch of each subcarrier of CSI data (i.e., Conv2D). For Conv2D, a 2D convolution kernel $\mathbf{k}_{2D}\in\mathbb{R}^{h\times w}$ operates on all patches of the CSI data via the sliding window strategy to obtain the output of the feature map, while the Conv1D only extracts the spatial feature along the subcarrier dimension. The Conv2D can be applied independently as it considers both spatial and temporal features, while the Conv1D is usually used with other temporal feature learning methods. To enhance the capacity of CNN, multiple convolution kernels with a random initialization process are used. The advantages of CNNs for WiFi sensing consist of fewer training parameters and the preservation of the subcarrier and time dimension in CSI data. However, the disadvantage is that CNN has an insufficient receptive field due to the limited kernel size and thus fails to capture the dependencies that exceed the kernel size. Another drawback is that CNN stack all the feature maps of kernels equally, which has been revamped by an attention mechanism that assigns different weights in the kernel or spatial level while stacking features. For CSI data, due to the varying locations of human motions, the patterns of different subcarriers should have different importance, which can be depicted by spatial attention in CTS-AM [59]. More attention techniques have been successfully developed to extract temporal-level, antenna-level and subcarrier-level features for WiFi sensing [48, 75, 76].

Recurrent neural network (RNN) is one of the deepest network architectures that can memorize arbitrary-length sequences of input patterns. The unique advantage of RNN is that it enables multiple inputs and multiple outputs, which makes it very effective for time sequence data, such as video [77] and CSI [5, 78, 79]. Its principle is to create internal memory to store historical patterns, which are trained via back-propagation through time [80].

For a CSI sample $x$, we denote a CSI frame at the $t$ as $x_{t}\in\mathbb{R}^{N_{s}}$. The vanilla RNN uses two sharing matrices $W_{x},W_{h}$ to generate the hidden state $h_{t}$:

where the activation function $\sigma(\cdot)$ is usually Tanh or Sigmoid functions. RNN is designed to capture temporal dynamics, but it suffers from the vanishing gradient problem during back-propagation and thus cannot capture long-term dependencies of CSI data.

To tackle the problem of long-term dependencies of RNN, Long-short term memory (LSTM) [81] is proposed by designing several gates with varying purposes and mitigating the gradient instability during training. The standard LSTM sequentially updates a hidden sequence by a memory cell that contains four states: a memory state $c_{t}$, an output gate $o_{t}$ that controls the effect of output, an input gate $i_{t}$ and a forget gate $f_{t}$ that decides what to preserve and forget in the memory. The LSTM is parameterized by weight matrices $W_{i},W_{f},W_{c},W_{o},U_{i},U_{f},U_{c},U_{o}$ and biases $b^{i},b^{f},b^{c},b^{o}$, and the whole update is performed at each $t\in\{1,...,T\}$:

where $\sigma$ is a Sigmoid function.

Apart from the LSTM cell [82, 83, 84], the multi-layer and bi-directional structure further boost the model capacity. The bidirectional LSTM (BiLSTM) model processes the sequence in two directions and concatenates the features of the forward input $\grave{x}$ and backward input $\acute{x}$. It has been proven that BiLSTM shows better results than LSTM in [45, 85].

Though LSTM addresses the long-term dependency, it leads to a large computation overhead. To overcome this issue, Gated Recurrent Unit (GRU) is proposed. GRU combines the forget gate and input gate into one gate, and does not employ the memory state in LSTM, which simplifies the model but can still capture long-term dependency. GRU is regarded as a simple yet effective version of LSTM. Leveraging the simple recurrent network, we can integrate the Conv1D and GRU to extract spatial and temporal features, respectively. [86, 87] show that CNN-GRU is effective for human activity recognition. In WiFi sensing, DeepSense [5] firstly proposes Conv2D with LSTM for human activity recognition. CNN-GRU is also revamped for CSI-based human gesture recognition in Widar [34]. SiaNet [9] further proposes Conv1D with BiLSTM for few-shot gesture recognition. As they perform quite similarly, we use CNN-GRU with fewer parameters in this paper for the benchmark.

Transformer [88] was firstly proposed for NLP applications to extract sequence embeddings by exploiting attention of words, and then it was extended to the computer vision field where each patch is regarded as a word and one image consists of many patches [89]. The vanilla consists of an encoder and a decoder to perform machine translation, and only the encoder is what we need. The transformer block is composed of a multi-head attention layer, a feed-forward neural network (MLP), and layer normalization. Since MLP has been explained in previous section, we mainly introduce the attention mechanism in this section. For a CSI sample $x$, we first divide it into $P$ patches $x_{p}\in\mathbb{R}^{h\times w}$, of which each patch has contained spatial-temporal features. Then these patches are concatenated and added by positional embeddings that infer the spatial position of patches, which makes the input matrix $v\in\mathbb{R}^{d_{k}}$ where $d_{k}=P\times hw$. This matrix is transformed into three different matrices via linear embedding: the query $Q$, the key $K$, and the value $V$. The self-attention process is calculated by

Intuitively, such a process calculates the attention of any two patches via dot product, i.e., cosine similarity, and then the weighting is performed with normalization to enhance gradient stability for improved training. Multi-head attention just repeats the self-attention several times and enhances the diversity of attentions. The transformer architecture can interconnect with every patch of CSI, which makes it strong if given sufficient training data, such as THAT [56]. However, transformer has a great number of parameters that makes the training cost expensive, and enormous labeled CSI data is hard to collect, which makes transformers not really attractive for the supervised learning.

Different from the aforementioned discriminative models that mainly conducts classification, generative models aim to capture the data distribution of CSI. Generative Adversarial Network (GAN) [90] is a classic generative model that learns to generate real-like data via an adversarial game between a generative network and a discriminator network. In WiFi sensing, GAN helps deal with the environmental dependency by generating labeled samples in the new environment from the well-trained environment [47, 58]. GAN also inspires domain-adversarial training that enables deep models to learn domain-invariant representations for the training and real-world testing environments [25, 91, 92, 93]. Variational network [94] is another common generative model that maps the input variable to a multivariate latent distribution. Variational autoencoder learns the data distribution by a stochastic variational inference and learning algorithm [94], which has been used in CSI-based localization [95, 96] and CSI compression [36]. For instance, EfficientFi [36] leverages the quantized variational model to compress the CSI transmission data for large-scale WiFi sensing in the future.

We choose two public CSI datasets (UT-HAR [33] and Widar [34]) collected using Intel 5300 NIC. To validate the effectiveness of deep learning models on CSI data of different platforms, we collect two new datasets using Atheros CSI Tool [2] and our embedded IoT system [22], namely NTU-Fi HAR and NTU-Fi Human-ID. The details and statistics of these datasets are summarized in Table II.

UT-HAR [33] is the first public CSI dataset for human activity recognition. It consists of seven categories and is collected via Intel 5300 NIC with 3 pairs of antennas that record 30 subcarriers per pair. All the data is collected in the same environment. However, its data is collected continuously and has no golden labels for activity segmentation. Following existing works [56], the data is segmented using a sliding window, inevitably causing many repeated data among samples. Hence, though the total number of samples reaches around 5000, it is a small dataset with intrinsic drawbacks.

Widar [34] is the largest WiFi sensing dataset for gesture recognition, which is composed of 22 categories and 43K samples. It is collected via Intel 5300 NIC with $3\times 3$ pairs of antennas in many distinct environments. To eliminate the environmental dependencies, the data is processed to the body-coordinate velocity profile (BVP).

NTU-Fi is our proposed dataset for this benchmark that includes both human activity recognition (HAR) and human identification (Human ID) tasks. Different from UT-HAR and Widar, our dataset is collected using Atheros CSI Tool and has a higher resolution of subcarriers (114 per pair of antennas). Each CSI sample is perfectly segmented. For the HAR dataset, we collect the data in three different layouts. For the Human ID dataset, we collect the human walking gaits in three situations: wearing a T-shirt, a coat, or a backpack, which brings many difficulties. The NTU-Fi data is simultaneously collected in these works [36, 12] that describe the detailed layouts for data collection.

We normalize the data for each dataset and implement all the aforementioned methods using the PyTorch framework [103]. To ensure the convergence, we train the UT-HAR, Widar, and NTU-Fi for 200, 100, and 30 epochs, respectively, for all the models except RNN. As the vanilla RNN is hard to converge due to the gradient vanishing, we train them for two times of the specified epochs. We use the Adam optimizer with a learning rate of 0.001, and the beta of 0.9 and 0.999. We follow the original Adam paper [104] to set these hyper-parameters. The ratio of training and testing splits is 8:2 for all datasets using stratified sampling.

We design the baseline networks of MLP, CNN, RNN, GRU, LSTM, BiLSTM, CNN+GRU, and Transformer following the experiences learned from existing works in Table I. The CNN-5 is modified from LeNet-5 [69]. We further introduce the series of ResNet [74] that have deeper layers. The transformer network is based on the vision transformer (ViT) [89] so that each patch can contain spatial and temporal dimensions. It is found that given sufficient parameters and reasonable depth of layers, they can converge to more than 98% in the training split. Since the data sizes of UT-HAR, Widar and NTU-Fi are different, we use a convolutional layer to map them into a unified size, which enables us to use the same network architecture. The specific network architectures for all models are illustrated in the Appendix. The hyper-parameters of the networks have been tuned to ensure the satisfactory convergence. To compare the baseline models, we select three classic criteria: accuracy (Acc) that evaluates the prediction ability, floating-point operations (Flops) that evaluates the computational complexity, and the number of parameters (Params) that measures the requirement of GPU memory. As WiFi sensing is usually performed on the edge, the Flops and Params also matter with limited resources.

Overall Comparison. We summarize the performance of all baseline models in Table III. On UT-HAR, the ResNet-18 achieves the best accuracy of 98.11% and the CNN-5 achieves the second best. The shallow CNN-5 can attain good results on all datasets but the deep ResNet-18 fails to generalize on Widar, which will be explained in Section V-F. The BiLSTM yields the best performance on two NTU-Fi benchmarks. To compare these results, we visualize them in Figure 5, from which we can conclude the observations:

• •

  The MLP, CNN, GRU, LSTM, and Transformer can achieve satisfactory results on all benchmarks.

• •

  The MLP, GRU, and CNN show stable and superior performances when they are compared to others.

• •

  The very deep networks (i.e., the series of ResNet) perform well on UT-HAR and Widar, but do not perform better than simple CNN on NTU-Fi. The performance does not increase as the number of network layers increases, which is different from visual recognition results [74]. Compared to simple CNN-5, the improvement margin is quite limited.

• •

  The RNN is worse than LSTM and GRU.

• •

  The transformer cannot work well when only limited training data is available in NTU-Fi Human-ID.

• •

  The models show inconsistent performances on different datasets, as the Widar dataset is much more difficult.

Computational Complexity. The Flops value shows the computational complexity of models in Table III. The vanilla RNN has low complexity but cannot perform well. The GRU and CNN-5 are the second-best models and simultaneously generate good results. It is also noteworthy that the ViT (transformer) has a very large computational complexity as it is composed of many MLPs for feature embedding. Since its performance is similar to that of CNN, MLP, and GRU, the transformer is not suitable for supervised learning tasks in WiFi sensing.

Model Parameters. The number of model parameters determines how many GPU memories are occupied during inference. As shown in Table III, the vanilla RNN has the smallest parameter size and then is followed by the CNN-5 and CNN-GRU. The parameter sizes of CNN-5, RNN, GRU, LSTM, BiLSTM, and CNN-GRU are all small and acceptable for model inference in the edge. Considering both the Params and Acc, CNN-5, GRU, BiLSTM, and CNN-GRU are good choices for WiFi sensing. Though the model parameters can be reduced by model pruning [105], quantization [106] or fine-tuning the hyper-parameters, here we only evaluate the pure models that have the minimum parameter sizes to converge in the training split.

Apart from supervised learning, other learning schemes are also useful for realistic applications of WiFi sensing. Here we evaluate two prevailing learning strategies on these models.

Evaluations on Transfer Learning The transfer learning experiments are conducted on NTU-Fi. We transfer the model from the HAR to Human-ID by pre-training the model in HAR (whole dataset) and then fine-tuning a new classifier in Human-ID (training split). This simulates the situation when we train the model using massive labeled data collected in the lab, and then use a few data to realize customized tasks for users. The human activities in HAR and human gaits in Human-ID are composed of human motions, and thus the feature extractor should learn to generalize across these two tasks. We evaluate this setting for all baseline models and the results are shown in Table IV. It is observed that the CNN feature extractor has the best transferability, achieving 96.35% on the Human-ID task. Similar to CNN, the MLP and BiLSTM also have such capacity. However, the RNN, CNN+GRU, and ViT only achieve 57.84%, 51.73%, and 66.20%, which demonstrates their weaker capacity for transfer learning. This can be caused by the overfitting phenomenon, such as the simple RNN that only memorizes the specific patterns for HAR but cannot recognize the new patterns. This can also be caused by the mechanism of feature learning. For example, the transformer (ViT) learns the connections of local patches by self-attention, but such connections are different between HAR and Human-ID. Recognizing different activities relies on the difference of a series of motions, but most human gaits are so similar that only subtle patterns can be an indicator for gait identification.

Evaluations on Unsupervised Learning We further exploit the effectiveness of unsupervised learning for CSI feature learning. We follow the AutoFi [63] to construct two parallel networks and adopt the KL-divergence, mutual information, and kernel density estimation loss to train the two networks only using the CSI data. After unsupervised learning, we train the independent classifier based on the fixed parameters of the two networks. All the backbone networks are tested using the same strategy: unsupervised training on NTU-Fi HAR and supervised learning on NTU-Fi Human-ID. The evaluation is conducted on Human-ID, and the results are shown in Table V. It is shown that CNN achieves the best accuracy of 97.62% that is followed by MLP and ViT. The results demonstrate that unsupervised learning is effective for CSI data. It yields better cross-task evaluation results than those of transfer learning, which demonstrates that unsupervised learning helps learn features with better generalization ability. CNN and MLP-based networks are more friendly for unsupervised learning of WiFi CSI data.

Convergence of Deep Models. Though all models converge eventually, their training difficulties are different and further affect their practical usage. To compare their convergence difficulties, we show the training losses of MLP, CNN-5, ViT, and RNN in terms of epochs in Figure 7. It is noted that CNN converges very fast within 25 epochs for four datasets, and MLP also converges at a fast speed. The transformer requires more epochs of training since it consists of more model parameters. In comparison, RNN hardly converges on UT-HAR and Widar, and converges slower on NTU-Fi. Then we further explore the convergence of RNN-based models, including GRU, LSTM, BiLSTM, and CNN+GRU in Figure 8. Though there show strong fluctuations during the training phase of GRU, LSTM, and BiLSTM, these three models can achieve much lower training loss. Especially, GRU achieves the lowest loss among all RNN-based methods. For CNN+GRU, the training phase is more stable but its convergence loss is larger than others.

How Transfer Learning Matters. We further draw the training losses of all models on NTU-Fi Human-ID with pre-trained parameters of NTU-Fi HAR in Figure 6. Compared to the training procedures of randomly-initialized models in Figures  7(c) and 8(c), the convergence can be achieved and even become much more stable. We can draw two conclusions from these results: (a) the feature extractors of these models are transferable across two similar tasks; (b) the fluctuations of training losses are caused by the feature extractor since only the classifier is trained for the transfer learning settings.

Poor Performance of Deep CNN on Widar. In Table III, a noticeable phenomenon is that the ResNet-18/50/101 cannot generalize well on Widar data, only achieving 17.91%, 19.47%, and 14.47%, respectively. In visual recognition, a deeper network should perform better on large-scale datasets  [74]. Then we have the question: is the degeneration of these deep models caused by underfitting or overfitting in WiFi sensing? We seek the reason by plotting their training losses in Figure 9. Figure 9(a) shows that even though the training accuracy has been almost 100%, the testing accuracy remains low, under 20%. Whereas, other networks (MLP, CNN, GRU) have similar training accuracy while the testing accuracy is increased to over 60%. This indicates that the degrading performances of ResNets are caused by overfitting, and different domains in Widar [34] might be the main reasons. This discovery tells us that very deep networks are prone to suffer from overfitting for cross-domain tasks and may not be a good choice for current WiFi sensing applications due to their performance and computational overhead.

Choices of Optimizer. During the training phase, we find that though Adam can help models converge at a fast speed, it also leads to much training instability, especially for the very deep neural networks. In Figure 10(a), we can see that ResNet-18 converges stably but ResNet-50 and ResNet-101 have fluctuating losses every 20-30 epochs. This might be caused by the dramatically changing values of WiFi data and its adaptive learning rate of Adam [104]. Then we consider changing the optimizer from Adam to a more stable optimizer, Stochastic Gradient Descent (SGD). In Figure 10(b), we find that the training procedure becomes more stable. This implies that if a very deep model is utilized in WiFi sensing, the SGD should be a better choice. If a simple model is sufficient for the sensing task, then Adam can enforce the model to converge better and faster.