Invisible-to-Visible: Privacy-Aware Human Segmentation using Airborne Ultrasound via Collaborative Learning Probabilistic U-Net

The camera-based methods have been well investigated and have achieved high accuracy. However, privacy concerns should be considered for applications such as home surveillance. Cameraless human segmentation methods have also been investigated to address privacy issues. Wang et al. [29] proposed a method for estimating human segmentation images, joint heatmaps, and part affinity fields using Wi-Fi signals. They used three transmitting-receiving antenna pairs and thirty electromagnetic frequencies with five sequential samples. The channel state information [9] was analyzed to input networks. The networks comprise upsampling blocks, residual convolutional blocks, U-Net, and downsampling blocks. Although this method achieved privacy-friendly fine-grained person perception with Wi-Fi antennas and routers, the Wi-Fi signals are highly affected by the surrounding environment due to the multipath effect.

Alonso et al. [1] proposed an event camera-based semantic segmentation method. The event camera detects pixel information when the brightness changes, such as when the subject moves. They do not capture personal information more clearly than the RGB cameras because event cameras only capture changes in intensities on a pixel-by-pixel basis. In [1], event information from event cameras was formed as $6$-channel images. The first two channels were histograms of positive and negative events, whereas the other four channels were the mean and standard deviation of normalized timestamps at each pixel for the positive and negative events. They showed that an Xception-based encoder–decoder architecture could learn semantic segmentation from the $6$-channel information. Although this method achieves semantic segmentation from privacy-preserved event cameras, it is difficult to detect people who are not moving.

Irie et al. [12] proposed a method that generates segmentation images from sounds. They recorded sounds using $4$-channel microphone arrays. They used Mel frequency cepstral coefficients and angular spectrum from sounds for estimating segmentation images. This method can estimate human and environmental objects only from sounds. However, estimating segmentation images for non-sounding people is difficult in principle because this method analyzes the sound emitted from objects.

Airborne ultrasound could be used to detect non-sounding people. The positions of people can be detected by analyzing ultrasound echos reflected by them. Although the multipath effect influences detection, it is less effective than radio waves because the propagation speed of ultrasound is slower than that of radio waves.

Airborne ultrasonic sensors have been used to detect distance in various industries, such as the automobile [30] and manufacturing industries [7, 3, 13]. Ultrasonic sensors emit short pulses at regular intervals. The ultrasounds are reflected if there are objects in the propagation path. The distance of objects can be determined by analyzing the time differences between emitted and reflected sounds [2].

When sounds are captured using microphone arrays, the directions of objects can be detected in addition to their distances [19]. Sound localization using beamforming algorithms has been developed. A delay-and-sum (DAS) method is a common beamforming algorithm [23]. The DAS method can estimate the direction of sound sources by adding array microphone signals delayed by a given amount of time. By using ultrasonic transducers and microphone arrays, the positions of objects can also be estimated using the DAS with regarding a reflected position as a sound source. Although this method can detect the position of objects, it is difficult to obtain the actual shape of objects from echoes of a single pulse.

Hwang et al. [11] performed three-dimensional shape detection using wideband ultrasound and neural networks. Although the analysis of multiple frequencies can precisely detect the positions and shapes of objects, the sound system for emitting wideband frequency becomes large, and the number of data increases because of the high sampling rate required to sense wideband ultrasound. Therefore, we consider obtaining segmentation images from the positional information on reflected objects analyzed by narrowband frequency ultrasound using a neural network.

The hardware setup is shown in Figure 2. The transmitter comprised a function generator and an ultrasonic transducer at 62 kHz resonance. The ultrasonic transducer was driven with burst waves of $20$ cycles and $50\,\rm{ms}$ intervals at $62\,\rm{kHz}$ using the function generator. The receiver comprised a MEMS microphone array, an analog-to-digital converter, a field-programmable gate array (FPGA), and a PC. A $4\times 4$ grid MEMS microphone array, whose microphones were mounted on a $30\,\rm{mm^{2}}$ substrate at $3.25\,\rm{mm}$ intervals, was used. The analog signals captured by the microphones were converted to digital signals and imported into the PC through the FPGA. The distance between the microphone array and ultrasonic transducer was set to $30\,\rm{mm}$.

The diagram of the data preprocessing is shown in Figure 3. First, a band-pass filter with a center frequency of $62\,\rm{kHz}$ and bandwidth of $10\,\rm{kHz}$ was used for signals captured by the $16$ microphones. The filtered ultrasound signals were divided into blocks including direct and reflected waves. Then, we upsampled the ultrasound waves four times at each block to improve the accuracy of direction estimation. Following that, we produced ultrasound images from reflected ultrasounds via DAS beamforming. The beamformed signal $y$ can be defined as

$$y(t)=\sum_{m=1}^{M}x_{m}(t-\Delta_{m}),$$

(1)

where $t$ is the time, $M$ is the number of microphones, $x_{m}$ is the signal received by the $m$-th microphone, and $\Delta_{m}$ is the time delay for the $m$-th microphone, which is determined by the speed of sound and distance between the $m$-th microphone and observing points. We set the observing points in the range of $\theta=-45$–$45$ degrees in the azimuthal direction and $\phi=-60$–$60$ degrees in the polar direction. Then, we calculated beamformed signals and obtained the reflected directional heat maps. To reduce the noise from reflected waves from objects other than people, we calculated ultrasound heat maps $H_{\rm{us}}$ by subtracting a reference map $H_{\rm{ref}}$ from reflected directional heat maps $H$ as

$$H_{\rm{us}}(i,j)=H(i,j)-kH_{\rm{ref}}(i,j),$$

(2)

where $(i,j)$ is the pixel of the heat maps, and $k$ is the coefficient, which is determined by

$$k=\frac{H(i_{\rm{max}},j_{\rm{max}})}{H_{\rm{ref}}(i_{\rm{max}},j_{\rm{max}})},$$

(3)

where $(i_{\rm{max}},j_{\rm{max}})$ is the index of the maximum pixel of $H_{\rm{ref}}$. Notably, the reference map was calculated using the data without people, and $H_{\rm{us}}$ was normalized as

$$X_{\rm{us}}(i,j)=\begin{cases}0,&H_{\rm{us}}(i,j)<0\\ \frac{H_{\rm{us}}(i,j)}{\max(H_{\rm{us}})},&H_{\rm{us}}(i,j)\geq 0\end{cases}$$

(4)

when it was converted to ultrasound images $X_{\rm{us}}$.

Examples of segmentation and ultrasound images are shown in Figure 4. The segmentation images in the top row were annotated from RGB images. The ultrasound images, which represent the intensity of the reflected ultrasound at each pixel, had ambiguous shapes at the positions corresponding to a person. Furthermore, there were artifacts in the region without any people.

As shown in Fig. 4, the ultrasound images do not have the shape of a person, and their edges are very different from those of the segmentation images. When there is a lot of ambiguity in the input image or when the difference between input and output is large, it is better to use the probabilistic method than the deterministic method. Thus, we consider performing human segmentation based on probabilistic models. Kohl et al. [15] proposed a probabilistic U-Net combining VAE [14] with U-Net. This method learns prior and posterior networks, which output the latent spaces of input and input/segmentation images, respectively, to be close. The latent variables sampled from the latent space of the posterior network are added to the last layer of U-Net. This method deals with large discrepancies between the input and output images by sampling latent variables from the latent space output by the prior network during the inference phase. However, it is difficult to handle with the existing prior network because the difference in appearance between the ultrasound and segmentation images is huge. Thus, we propose collaborative learning probabilistic U-net based on probabilistic U-Net. Details of the proposed method are described below.

In this section, we explain the probabilistic U-Net. The architecture of the probabilistic U-Net is shown in Figure 5(a). During the inference phase, the input image $X_{\rm{in}}$ is input into the prior network and U-Net. The prior network, which is parameterized by $\omega$, encodes the input image and outputs the parameters of the mean $\mu_{\rm{prior}}(X_{\rm{in}};\omega)$ and variance $\sigma_{\rm{prior}}(X_{\rm{in}};\omega)$. Then the data $z$ was randomly sampled from the prior distribution $P$ as

$$z\sim P(\cdot|X_{\rm{in}})=\mathcal{N}\bigl{(}\mu_{\rm{prior}}(X_{\rm{in}};\omega),\rm{diag}(\sigma_{\rm{prior}}(X_{\rm{in}};\omega))\bigr{)},$$

(5)

and broadcasted to the N-channel feature map to match the shape to the segmentation image. The feature maps are concatenated at the last layer of the U-Net, and the output image is estimated by convoluting the feature maps from the prior network and U-Net.

During the training phase, the posterior network is added to encode the information about the segmentation image. The posterior network parameterized by $\nu$ encodes the data that contains the input image $X_{\rm{in}}$ and segmentation image $X_{\rm{seg}}$. The parameters of the mean $\mu_{\rm{post}}(X_{\rm{in}},X_{\rm{seg}};\nu)$ and variance $\sigma_{\rm{post}}(X_{\rm{in}},X_{\rm{seg}};\nu)$ of the posterior distribution $Q$ are obtained using the posterior network. Then, the data $z$ is sampled from the posterior distribution $Q$ as

$$z\sim Q(\cdot|X_{\rm{in}},X_{\rm{seg}})=\mathcal{N}\bigl{(}\mu_{\rm{post}}(X_{\rm{in}},X_{\rm{seg}};\nu),\rm{diag}(\sigma_{\rm{post}}(X_{\rm{in}},X_{\rm{seg}};\nu))\bigr{)},$$

(6)

and combined to the feature map at the last layer of the U-Net. The loss function consists of two terms. The first term is cross entropy to minimize the error between the estimated image and ground truth. The second term is KLD to close the distribution between the prior distribution $P$ and posterior distribution $Q$. The losses are combined as a weighted sum,

$$L(X_{\rm{seg}},X_{\rm{in}})=\mathbb{E}_{z\sim Q(\cdot|X_{\rm{seg}},X_{\rm{in}})}\left[-\log P(X_{\rm{seg}}|X_{\rm{out}}(X_{\rm{in}},z))\right]\\ +\beta D_{\rm{KL}}(Q(z|X_{\rm{seg}},X_{\rm{in}})\parallel P(z|X_{\rm{in}})),$$

where $X_{\rm{out}}(\cdot)$ outputs the estimated segmentation image and $\beta$ is the weight parameter.

Although the appearance of ultrasound and ground truth segmentation images differ significantly at the edges, the appearance of the other parts, particularly positions, is relatively similar. Therefore, it is important to learn latent space by focusing on the difference in edges. In probabilistic U-Net, prior distribution $P$ and posterior distribution $Q$ are penalized using KLD. Reducing the distance between the distributions to estimate edges with high accuracy by comparing the distributions obtained after the spatial dimensions decreased at the prior/posterior network. Because the ultrasound and segmentation images are roughly matched other than edges. Thus, we propose a method to use mean squared error (MSE) of means and variances instead of KLD.

The errors between the standard normal distribution calculated by KLD and MSE are shown in Figure 6. The MSE loss was calculated as

$$\rm{MSE}=(\mu-\mu_{0})^{2}+(\sigma-\sigma_{0})^{2},$$

(7)

where $\mu_{0}=0$ and $\sigma_{0}=1$ are the mean and variance of the standard normal distribution. Since the value around the error of 0 for the MSE loss changes more rapidly than that of the KLD (Fig. 6), the MSE loss is more sensitive than the KLD. Therefore, converging to an optimal result by KLD is difficult because of the small gradient of loss landscape, since the ultrasound and segmentation images are roughly matched other than edges. Thus, we propose a method that uses MSE loss of the means and variances to penalize the differences of the latent spaces. Additionally, KLD regularization is added to the posterior distribution to approach a standard normal distribution.

The proposed network is illustrated in Figure 5(b). The comparison method between the prior and posterior distributions at the training phase distinguishes the probabilistic U-Net. The loss function $L$ of the proposed method is as follows:

$$L=\alpha L_{\rm{VAE}}+(1-\alpha)L_{\rm{MSE}},$$

(8)

where $\alpha$ is the weights adjusting the scale. The first term $L_{\rm{VAE}}$ is

$$L_{\rm{VAE}}=\mathbb{E}_{z\sim Q(\cdot|X_{\rm{seg}},X_{\rm{us}})}\left[-\log P(X_{\rm{seg}}|X_{\rm{out}}(X_{\rm{us}},z))\right]\\ +\beta D_{\rm{KL}}(P_{0}(z)\parallel Q(z|X_{\rm{seg}},X_{\rm{us}})),$$

where $X_{\rm{us}}$ is the ultrasound image and $P_{0}(z)$ is the standard normal distribution. The second term $L_{\rm{MSE}}$ is

$$L_{\rm{MSE}}=\frac{1}{N}\sum_{n=1}^{N}(\mu_{\rm{prior}\it{,n}}-\mu_{\rm{post}\it{,n}})^{2}+\frac{1}{N}\sum_{n=1}^{N}(\sigma_{\rm{prior}\it{,n}}-\sigma_{\rm{post}\it{,n}})^{2},$$

(9)

where $N$ is the dimension of the latent vector.

We first describe the performance of the proposed CLPU-Net and compare it with other methods. Then, we described the comparison of the accuracy by distances between the sensor and person. Then, we explain the comparison of the accuracy by rooms. Finally, quantitative results are described.