Recovering Surveillance Video Using RF Cues

Deep generative models have been extensively investigated in image or video generation tasks. Variational Auto-Encoders (VAEs) (Kingma and Welling 2013) and Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) are the most popular models. After the first debut of GANs, many follow-up works are committed to solving problems encountered in original GANs. Arjovsky et al noticed the model collapse and unstable training process in GANs and proposed an improved version wGAN (Arjovsky, Chintala, and Bottou 2017) and wGAN-GP (Gulrajani et al. 2017). (Mirza and Osindero 2014) explored the performance of GANs in a conditional side. They first proposed cGAN to generate images controlled by class attributes. Besides, many other variants of GANs were proposed to improve the original GANs in different aspects. For example, DCGAN (Radford, Metz, and Chintala 2015) enables CNN to conduct feature extraction, StarGAN (Choi et al. 2018) allows image-to-image transformation among multiple domains, PatchGAN (Isola et al. 2017) introduces pixel-wise loss function. For human image generation, (Yang et al. 2018) proposed a pose-guided method to generate synthetic human video in a disentangled way-plausible motion prediction and coherent appearance generation. Similarly, Cai et al also proposed a two-stage human motion video generation method via GANs (Cai et al. 2018). Though GANs is a powerful tool to generate videos and have made exceptional breakthrough in many tasks, we will not employ it in this work since CSI2Video system has a unique ground-truth. Thereby, the Mean Square Error (MSE) loss would not impair the image quality.

CSI2Video is a cross-modal system that merges both wireless and visual information to produce real-time surveillance video. Specifically, the visual and wireless measurements are gathered spatially and temporally separated to obtain appearance and localization features respectively. In our system, the wireless information is extracted from standard IEEE 802.11n WiFi signals using commercial devices. Due to the different electromagnetic properties of the background environment and human bodies, WiFi signals have embedded rich spatial information of human localization and movements. However, the WiFi signals transmit at the same time may not be received simultaneously by the antennas considering the different propagation paths resulted from the signal penetrates, refracts and reflects. Fortunately, in WiFi communication system, channel state information (CSI) was adopted to describe the current channel condition between each transceiver ($\mathit{T}$) and receiver ($\mathit{R}$) pairs. This way, the spatial information of human body is encoded into a $N_{tx}\times N_{rx}\times N_{C}$ matrix where $N_{tx}$ and $N_{rx}$ denote the numbers of transmitting and receiving antennas respectively and $N_{C}$ denotes the number of orthogonal frequency division multiplexing (OFDM) subcarriers. Let $\mathbf{C}$ be the single CSI measurement where $\mathbf{C}\in\mathbb{C}^{N_{tx}\times N_{rx}\times N_{C}}$. For each continuous CSI measurements, the recorded CSI data can be represented as $\mathcal{C}=\left\{\mathbf{C_{0}}\cdots\mathbf{C_{i}}\cdots\mathbf{C_{n}}\right\}$. For human identity profile, it is used to give the network side information to determine which human instance should be displayed since the wireless measurement contains no explicit identity information. We choose video captures as our source of identity profile in this work. For each continuous video captures, the recorded frames can be denoted as $\mathcal{F}=\left\{\mathbf{F_{0}}\cdots\mathbf{F_{i}}\cdots\mathbf{F_{n}}\right\}$. Therein, $\mathbf{F}$ indicates one single frame where $\mathbf{F}\in\mathbb{R}^{H\times W\times\mathit{3}}$.

As depicted in Figure 2, CSI2Video consists of two parts-cross-modal mapping and video generation. The cross-modal mapping network takes CSI measurements as input and outputs Joint Heat Maps (JHMs) and Part Affinity Fields (PAFs) (Cao et al. 2017). The PAFs and JHMs will be concatenated by the selected video frames which contain human appearance features and then be feed into the video generation network. Finally, the video generation network merges both human pose features and appearance features then produces the synthetic surveillance video. In general, the CSI2Video converts the spatial information of the human body contained in the wireless signals into a form of ordinary RGB surveillance video.

The cross-modal mapping network takes CSI measurements $\mathcal{C}$ as input and outputs PAFs and JHMs. So, to train the cross-modal mapping network, we need a set of synchronized CSI measurements $\mathcal{C}$ and their corresponding PAFs and JHMs pairs. For convenience, we employed Openpose (Cao et al. 2017) to compute the PAFs & JHMs for training phase and denote Openpose as $\mathcal{O}(\cdot)$. Let $\mathcal{I}=\left\{\mathbf{I_{0}}\cdots\mathbf{I_{i}}\cdots\mathbf{I_{n}}\right\}$ be the frames used to compute body pose where $\mathbf{I}\in\mathbb{R}^{H\times W\times\mathit{3}}$ represents a single recorded frame. Since $\mathcal{C}$ is a 1D time-series with less points than $\mathcal{I}$, it can not contain as much information as $\mathcal{I}$. To ease the influence of unbalanced information density, we use a higher frequency to sample CSI measurement and pair multiple CSI measurements to one video frame. Let $K$ be number of CSI measurements which are corresponded to one video frame. Moreover, to feed a complex matrix to the neural network, we choose the amplitude of CSI measurement to represent the whole complex matrix. Therefore, the data pair used to train cross-modal mapping network can be represented as Equation 1 where $\mathbf{C_{m}}$ and $\mathbf{I_{m}}$ is assumed to have already been synchronized.

The video generation network takes two sources of input. The first source is generated PAFs ($\mathcal{P}$) and JHMs ($\mathcal{J}$) from the cross-modal mapping network which aims to reflect the localization and movement features. The second source is the visual captures for giving side information of human instances. Assuming $\mathcal{F}$ as a continuous video captures, however, we just need a few of them to give the video generation network cues about human appearances. Let $l$ be the number of selected frames from $\mathcal{F}$. Given PAFs $\mathbf{P_{i}}$, JHMs $\mathbf{J_{i}}$ and selected video frames $\mathcal{F}_{l}$, the video generation network outputs synthetic surveillance frames $\mathbf{S_{i}}$ as Equation 2. Specifically, we interpolate all input to $d\times H_{\mathcal{G}}\times W_{\mathcal{G}}$ where $d$ is the number of channels of each source of input.

We collected the data in an $8m\times 16m$ office room with 5 volunteers. During the experiment phase, zero to three volunteers were asked to perform walking, sitting, waving hands and several random movements concurrently in the perception area. We leveraged one logitech 720p camera and two Intel 5300 NICs equipped with three antennas each to record the CSI and video frame pairs. As shown in Figure 4, the NICs and RGB camera were placed at a height of 1m and 2m respectively. Antennas of each NICs were uniformly spaced at a distance of 2.6cm like common commercial WiFi routers and the two NICs were separated with each other about 6m away. Moreover, the camera was set to record $1280\times 720$p RGB video with a FPS of 7.5. For CSI measurement, we used an open-source tool to record (Halperin et al. 2011) and set the NICs to communicate with a bandwidth of 20 MHz centering in the 5.6 GHz WiFi band. In this condition, we recorded the CSI measurements of 30 subcarriers with a sampling rate of 100Hz.

Since the unstable nature of wireless communication, the sampling interval of the measured CSI signals are not strictly equal to 0.01s. Thus, before synchronizing the CSI measurements $\mathcal{C}$ and video frames $\mathcal{I}$, we conducted data cleaning to remove outliers and to resample the CSI measurements to exactly 100Hz. In our implementation, $K$ is set to 5. This means 5 CSI measurements are matched with one video frame. Consequently, we created 24082 pairs where 75% of it is used for training and 25% of it is used for testing. Specifically, to better fit our network structure, we interpolated the CSI measurements to an image-size tensor.

For video frames $\mathbf{F}_{l}$, we randomly selected them from $\mathcal{I}$ in our implementation for convenience consideration. Since we just suppose frame $\mathcal{F}$ contains human appearance information and take advantage of nothing beyond it, selecting $\mathbf{F}_{l}$ from $\mathcal{I}$ has no difference from using any other source and will not result in any negative influence on overall results. We set $l=1$ in our implementation.

In CSI2Video, two deep neural network structures are tailored. The cross-modal mapping network maps the CSI tensor to JHMs and PAFs and the video generation network makes use of JHMs, PAFs and video frames to generate synthetic surveillance video. Note that this is a proof-of-concept experiment that aims to show the ability to recover surveillance video from CSI measurements and can be further improved by carefully frame selection and manual annotations.

We train each of the networks for 20 epochs on the Pytorch framework. An Adam optimizer with $\beta_{1}=0.9$ and $\beta_{2}=0.999$ is adapted in all of the training process. The learning rate starts from 1e-6 for cross-modal mapping network and from 1e-3 for video generation network. They are divided by 10 for each 5 epochs. Moreover, we set the $\alpha_{J}=1$, $\beta_{J}=1$, $\alpha_{P}=1$, $\beta_{P}=0.5$, $\lambda_{J}=1$, $\lambda_{P}=1$ and $\lambda_{f}=1$, $\lambda_{b}=0.5$ in our experiment.

We evaluate the generated JHMs & PAFs and recovered video frames separately.

Figure 8 describes the PCKs of all 14 keypoints extracted from cross-modal mapping network. The mean [email protected] of all 14 points are about 70% which indicates the majority of keypoints will not diverge from ground truth away more than 20% of diagonal length of instance’s bounding box. This is an acceptable result since the spatial resolution of commercial Wireless signal is approximately 4cm. To further explore the gaps between vision-based approach, we presents the mPCK of both our method and Openpose on Table 1. The mean [email protected] drops 10% by replacing source from image to wireless pixel.

In Figure 7, we show our video generation results for single person scenario. The raw frames are recorded directly from the camera and the O-frames and C-frames are generated using JHMs and PAFs from Openpose and cross-modal network respectively. The O-frames have almost perfect quality except the face part. The reason for the blurred face perhaps because we just used an $256\times 128$ image as label. It could be further improved by using higher-resolution ground-truth. In comparison to the O-frames, the C-frames have a relatively low quality. The gap between O-frame and C-frame indicates that the wireless pixel cannot reflect the localization and movement as accurate as image pixels. However, the image quality of C-frame is quite enough to meet the requirements for behavior monitoring. And the image quality is supposed to be improved by training on more samples and manually annotating the ground-truth.

Figure 9 depicts an example of the multiple-people scenario. We can find that the image quality of it has no distinct difference from that of single person scenario while the appearances of generated human instances are almost random. This is because the network cannot match the human instance with their identities since the video frame only contains explicit information about who are in the RF perception field but cannot tell the identity of each one. We will address this issue in CSI2Video2 by obtaining human identity information using indoor identification and localization technology.

Then, we illustrate the results of objective evaluation metrics. For this purpose, we compute the IoU scores as Equation 10 on different thresholds as depicted in Figure 8. We observe that the recovered frames have a high IoU score and decline gently at low threshold values. The IoU begins to decrease rapidly from 0.55 to 0.75. This indicates IoUs concentrates in this interval. Additionally, to our best knowledge, this is the first attempt to generate RGB surveillance video using WiFi signals. For this reason, we choose Person in WiFi (Wang et al. 2019) and Mask-RCNN (He et al. 2017) as baseline. Specifically, Person in WiFi is an end-to-end body segmentation and pose estimation method using WiFi signals. Mask-RCNN is a camera-based segmentation method. Since we utilized Mask-RCNN to generate annotations in the training phase, we still need new manually annotated labels for Mask-RCNN evaluation. To evaluate the results, we randomly selected 250 samples and manually annotated them. Table 1 illustrates the performance between those three methods. Although those methods use completely different sources to generate masks, they are all toward fine-grained human body segmentation. The gap between video-based method is obvious. Incompletely reconstructed body part and overlapped body part may account for the gap. Although having an obvious gap compared to camera-based method, the IoU of masks from generated frames are comparable with that from WiFi signals directly. This indicates that the generated video does not lose information from WiFi signals, however, the wireless signals cannot contain as much information as video frame. Whilst this problem can be mitigated by adding more WiFi router pairs or more recorded OFDM subcarriers.