CoPlay: Audio-agnostic Cognitive Scaling for Acoustic Sensing

Achieving our two objectives simultaneously is hard because just tweaking the signal amplitude in the time domain will cause noisy distortion in the frequency domain. To understand the effect of inordinately tweaking the amplitude, consider amplitude modulation (AM) as an interpretable example. AM is a technique used in telecommunications to transmit information by varying the amplitude of a carrier wave. In Fig. 2, the high-frequency carrier is multiplied by the low-frequency base signal, i.e. amplitude modulated. It causes the frequency spectrum of the signal to spread out with noise. This is because the noisy side bands, located above and below the carrier frequency, are created by the modulation process. In this case, the change is closed-form; the side frequencies are $f_{c}+f_{m}$ and $f_{c}-f_{m}$. (Derivatives can be found in Appendix A). Building upon this observation, our cognitive scaling is namely multiplying the signal with a more complex window function, leading to various sidebands in the frequency. Therefore, this effect of altering signals necessitates the utilization of deep learning in this work; closed-form signal processing methods or traditional machine learning can hardly dynamically control the distortion during generation.

In the field of acoustic sensing, two commonly used types of signals are sine waves and FMCW chirps. These signals serve different purposes and are utilized in various applications within acoustic sensing systems.

Sine waves: A sine wave is a single-frequency oscillation that varies with time according to the mathematical function $A\sin(2\pi ft+\phi)$, where $A$ is the amplitude of the sine wave, $f$ is the frequency, $\phi$ is the initial phase. They are commonly used in acoustic sensing for tasks such as signal testing, calibration, and as reference signals. They provide a simple and well-defined waveform that is easy to generate and analyze.

FMCW chirps: Frequency Modulated Continuous Wave (FMCW) is a technique where the frequency of a continuous wave changes linearly over time. Basically, they are repeated chirps defined as $A\sin(2\pi(f_{a}t+\frac{f_{1}-f_{0}}{2T}t^{2}))$, where $f_{0}$ and $f_{1}$ are the starting and ending frequency of the chip; $T$ is the time duration of one chirp. They are widely used in applications like radar and sonar. In acoustic sensing, FMCW signals offer advantages such as improved range resolution and the ability to distinguish between multiple targets. They are particularly useful in applications requiring precise distance measurements and target discrimination.

To begin, we establish a formalization of the model input, output, and training objectives in loss function.

Input and output: Let $x$ be the sensing signal, i.e. a high-frequency sine or chirp. Concurrently, an App plays music $z$. Then the mixer mixes them within the maximum range of $[-m,m]$ allowed by the number of DAC bits. In this paper, we use signed 16-bit PCM, resulting in a range of $-2^{15}\sim 2^{15}-1$. For computation convenience, we set $m=2^{15}-1$. In general, our goal is to generate a cognitively-scaled sensing signal $\hat{x}$ conditioned on the music $z$; and the $\hat{x}$ is optimal when satisfying the following objectives defined in the loss.

Loss: The first objective is to maximize the sensing amplitude within the bandwidth left by music. In other words, the amplitude of the mixed signal $\hat{x}+z$ in time domain should be close to the maximum allowed amplitude $m$. To restate, it is to minimize loss $q(\hat{x}+z,m)$, namely the difference between the magnitude of them. However, this would cause consequential frequency distortion as explained in section 3.1, which deteriorates the receiver-side algorithms since they generally rely on accurate frequency (either FFT convolution or correlation convolution). Therefore, we define another loss $p(x,\hat{x})$ as the difference between $x$ and $\hat{x}$ in frequency domain; it is the difference between their L-2-norm of FFT coefficients. In summary, our goal is to minimize these two loss terms simultaneously as an optimization problem.

Next, we can further customize the loss for sine and chirps. The L-2 norm of FFT coefficients is then rearranged as

where $c$ are the magnitude of the complex number coefficients in FFT, $FFT(x)=(c_{i},...)$, $FFT(\hat{x})=(\hat{c_{i}},...)$. And $f_{c}$ is the target frequency bin containing the ultrasound frequency. $N$ is half of the window size since we only take the one symmetric half of the FFT output. The magnitude is normalized by the window size and the maximum allowable amplitude of input $x$, i.e. $m$. The normalization guarantees that the two variable terms in $p$ are standardized and unbiased. We call the first half target-loss and the second half recovery-loss. The intention here is to encourage a large value in the target frequency bins, and a close-to-zero value in the others.

Secondly, $q$ is the L-2 norm of $\hat{x}+z$ in time domain normalized by $m$, named as amplitude-loss.

For FMCW chirp, we also want the multiple frequency bins to have even amplitude so that it is well correlated. So, we have an additional variance-loss for FMCW using the standard deviation with zero correction:

Finally, the overall loss equals to the weighted summation of these two terms, where $\alpha,\beta,\gamma$ are hyperparameters.

Note that, our loss may not guarantee that $\hat{x}+z$ falls within [$-m,m$]. Therefore, we need to embed this constraint into model layers, as introduced in the following section 3.4 per customized link function.

In practice, we first split the sensing signal and music into small windows and run the optimization for each window. For chirps, the window size should be the size of one chirp. In this paper, we use 18kHz sine wave and 18k-20kHz FMCW signal, with a window size of 512 and a sampling rate of 48kHz. These are common settings and factor in all considerations per efficiency and performance. And our model can be extended to other settings as long as training settings are the same as those in testing.

Next, we establish a model using a non-AR seq-to-seq network. As depicted in Fig. 3, the inputs are music $z$ and sensing $x$ and the output is adapted sensing $\hat{x}$. The model architecture derives from a non-causal WaveNet backbone, followed by an anti-aliasing (sinc) kernel and a custom link function for normalization with agnostic music as bias.

We use four types of concurrent audio to study model’s generalization across agnostic types of music. They are piano music, hot songs, bass music, and speech datasets. Furthermore, for each type, we select two different datasets.

Piano: One dataset is the Beethoven dataset (Mehri et al., 2016), consisting of piano sonatas. It has a total duration of 10 hours and serves as a benchmark music dataset for audio generation. We interpolate the data to a sample rate of 48kHz and cast the 8-bit quantization into a 16-bit signed integer. Another piano dataset is the YouTubeMix dataset (Samplernn., 2017) with higher-quality recordings, with a total duration of 4 hours. And we format it in the same way as the Beethoven dataset.

Hot songs: We select two non-stop playlists of hot songs, featuring top-ranking Billboard songs from 2019 and 2022(YoutubeMix, 2019, 2022), spanning various genres like Pop, Rock, and Hip-hop. Both playlists are around two hours long, sourced from YoutubeMix adhering to the same format.

Bass: To ensure the inclusion of low-frequency audio, we had two bass playlists (YoutubeMix, 2024, 2023a), lasting for one hour and two hours, both of which are downloaded from YoutubeMix under the same data format as above.

Speech: Besides musical audio, we add two conversational speech datasets. They are a mental health podcast featuring Selena Gomez(YoutubeMix, 2023c) and a Conan O’Brien podcast(YoutubeMix, 2023b). Both last over one hour long and are downloaded by YoutubeMix in the same format.

All datasets are then cut into equal-length clips, paired with same-length sensing signal clips as defined in section 3.3.

Generalization across datasets and music types: In total, we have 8 datasets, with two datasets for each of the four audio types: piano, hot songs, bass, and speech. When the model is trained and tested with the same dataset, we refer to it as intra-dataset experiment. When training and testing with different datasets from the same type, it is a cross-dataset experiment. Lastly, to assess a more out-of-distribution case, we train on one dataset from one type and test on another dataset from another type, termed as cross-type experiment. Note that, in intra-dataset experiments, we avoid randomly splitting the dataset for testing that is non-realistic in usage. Instead, we take the last 10% of the video as testing set. The rest is then randomly split into 80% training and 20% validation. Fig. 5 shows the average error for an extensive run of these experiments. The results for both sine and chirps are separately plotted as bar charts for different groups of music types. To better plot the error, the y-axis is log-scaled testing loss multiplied by $1e5$.

In conclusion, the model manifests the capability of generalizing to different types of music or speech, since the cross-dataset error is similar to that of cross-type data error for both sine and FMCW chirps. While intra-dataset usually outperforms the others because the distribution between testing and training is close, the intra-dataset results in chirp are a bit counter-intuitive and it takes 10x more epochs to converge. We think the reason could be that chirp is more complex and needs more data to train, while intra-dataset trains on only 6̃0% of cross-dataset/cross-type experiments. Besides, it is worth mentioning that the cross-type results in the speech group are also good. This is a particularly significant generalization result, since the speech category stands out as the most distinguishable type of audio, compared with music of piano, hot songs, and bass.

Decompose the loss: The overall error mentioned above comprises multiple loss terms as defined in the section 3.3. Hence, we decompose it into individual loss term values to assess the output per each objective represented by the corresponding loss term. Table. 1 demonstrates the average sub-loss on hot songs datasets. They are training loss including three sub-loss values for sine and an additional variance-loss for FMCW, which we defined above to even the amplitude in the target bandwidth. By default weights, each sub-loss ranges from 0 to 1. We observe that target-loss and amplitude-loss are the bottlenecks, aligning with the intuition that minimizing frequency distortion and maximizing the amplitude are opposed objectives, which is a common issue in multi-task learning. This table serves as a reference for tweaking the weights of loss terms to specific downstream task requirements. For instance, near-field sensing tasks can sacrifice amplitude for minimal distortion.

Varying music volume: Although it is possible that a high volume of music might degrade our model performance, Fig. 6(b) shows that the overall error does not increase with music volume. In detail, we train with the same sub-loss experiment and test with increasing levels of music volume. The x-axis indicates the ratio of sensing signal volume and music volume. The default ratio is 1:1, and 0.6:1.4 is when music is roughly double in power, equivalently 7.3db louder. In summary, CoPlay is stable when training with agnostic music even in unseen volume. This could be attributed to the normalization in the definition of our amplitude-loss and recovery-loss.

Ablation of model architecture: In terms of the model design, we justify our choice by comparing the model performance under different model backbones. We considered three models namely CNN, RNN, and Wavenet with the same number of recursive layers. Fig. 6(a) shows that Wavenet slightly outperforms CNN and both surpass RNN. But CNN has a limited reception field so we chose Wavenet which is better at overcoming this issue with dilated convolution layers. Furthermore, the sample RNN uses LSTM layers which is less efficient to train in comparison.

Ablation of multi-task loss terms: The overall loss consists of sub-loss terms, each of which represents different objectives of optimization (section 3.3). By default, they are assigned equal weights. So, we ablate the weights to assess the performance when we emphasize on certain objective. In detail, recovery-loss is set to one by default, the same as other sub-losses. Hence, we also run the same ablation setup but increase the weight of recovery-loss from 1 to 1.2, 1.4, and 1.6, making it the largest sub-loss, i.e. making minimizing frequency distortion the most significant objective; this is a practical use case when certain sensing systems, like near-field tasks, can compromise the power for minimal frequency distortion. In Fig. 6(c), the error roughly goes down with the increase in weight, which serves as a reference for tuning this ratio for the downstream tasks in practice.

Ablation of sinc filter: Finally, we conduct an ablation to evaluate the effectiveness of the sinc filter as defined in section 3.3. It’s also the same experiment set up as the above ablation study. As shown in Fig. 6(d), both the sine and chirp models exhibit improved performance with the presence of sinc kernel, especially the sine wave model. We attribute this enhancement to the narrow target main lobe in the spectrum of sine, thus more side lobes are to be suppressed. The sinc kernel helps significantly reduce these lobes, resulting in a lower numerical loss. In summary, this ablation study validates the efficacy of the sinc filter, and we incorporate this essential layer in all of our experiments.

Training procedure in 3D frequency domain: Although the numerical loss can indicate the model performance, we seek a more comprehensive perspective by visualizing the model output in frequency spectrum. This helps confirm that the output aligns well with expectations. In Fig. 7, we take the FFT of a sample window output $\hat{x}$ and screenshot it with the increasing training epochs. The rise of the yellow peaks indicates that it converges to an 18kHz sine wave or 18k-20kHz FMCW as the training epoch increases. The side lobes are much weaker than the main lobes, which manifest good SNR. (Moreover, we further substantiate the SNR by qualitative study in section 5)

Ranging profile with cross-correlation: Likewise, certain sensing algorithms employ cross-correlation (Xcorr) to obtain the ranging profile instead of utilizing FFT. Nevertheless, they basically use equivalent computation components: convolution or even DFT. For instance, an accelerated computation of Xcorr is to perform a frequency-domain Xcorr, which is faster than that in the time domain (Dr. Erol Kalkan, 2023), because shifting a signal in the time domain is equivalently scaling it in the frequency domain. Therefore, in addition to plotting the FFT spectrums of the generated signal, Fig. 8 shows the Xcorr correlation output for the cognitively-scaled signal. By rolling the signal in the time domain, the distinct peak in Xcorr output changes with the number of shifts as expected. In summary, this justifies that our adapted signal works well even if the receiver-side sensing algorithm uses Xcorr.

We refer to CoPlay as a mixer algorithm, by expecting that our cognitive scaling can be applied to the buffered audio stream in the speaker. So, we run training and inference on small windows of audio clips. In this paper, we test with window size of 512, which is a common setting for acoustic sensing and also meets the computation efficiency requirement. To elaborate on the computation cost, our model has a size of 5.5 KB, forward/backward pass size of 0.98 MB, and an input size of 0.07 MB. The estimated total size of training is 1.05 MB only, which indicates the inference could easily fit into the RAM of commercial smartphones or other mobile devices. Despite that it was trained on an RTX 8000 GPU, it can run inference for roughly 625 fps on mobile devices, equivalently $1/625=1.6ms$ per frame. On the other hand, the buffer interval is roughly $512/48000Hz=10ms$, which leaves quite enough time for inference on one frame. Therefore, it can run in real time with no perceptible delay. Furthermore, the entire audio is actually pre-known by the speaker in many cases. For instance, Java sound API can load the audio file into memory and then buffer the sliding windows into input stream. That’s saying, the speaker can run the inference upfront and before playing the processed signal. Moreover, there is a lot of literature on model compression and inference acceleration to further improve the efficiency, which is beyond the scope of this work.

We also plot the roofline in Fig. 9, illustrating our model on RTX 3080 GPU in comparison with other popular models; ours is much more memory-efficient than even the mobile models like MobileNet, SqueezeNet, and DenseNet, because of its small number of parameters of dilated layers. We also show dashed rooflines from other popular mobile SoCs(System-on-a-Chip) and GPUs as a comparison of their peak performance and max memory bandwidth, which validates that our model would work on most chips, especially mobile chips like Snapdragon 888 and Kirin 9000.

In the field study, we evaluate our cognitive scaling mixer algorithm in two tasks: respiration rate detection and hand gesture recognition. In each task, we play inaudible ultrasound (sine or FMCW) from a mobile phone (IOS or Android) and then analyze received signals for recognition.

Respiration monitoring is a well-explored application in acoustic sensing, manifesting its high resolution that can detect subtle millimeter-level motions. The target is to get breath per minute BPM by detecting peaks in the waveform of chest movement during inhalations. Therefore, in the following results, we show the mean absolute error MAE of BPM and visualize chest movement in a waveform.

Hand gesture recognition is another important use case showing that acoustic sensing can leverage patterns in received signals for profound motion recognition. In this study, we classify three gestures: swipe, push, and pinch, which are essential in commercial gesture control systems. In the following result section, we show the classification accuracy and the confusion matrix as well.

Hardware: We run acoustic sensing on two mobile phones: a Samsung smartphone and an iPhone, representing the mobile operation systems of Android and IOS, because their built-in algorithm handling overflow in the speaker mixer is different. As discussed above, Android turns to clip, IOS turns to scale down. Both phones are placed on the desk around 40cm away from the seated user facing the phone; one phone at a time.

The ground truth for respiration detection is from the Go Direct Respiration Belt, a belt with a force sensor (Belt, 2023). The user wears it around the chest or abdomen. It also has a built-in breath rate calculation every 10 seconds. The sample window for the calculation is 30 seconds. To get the ground truth label for gesture recognition, we play a guiding video on a monitor and ask the user to follow the video to conduct specific gestures at a guided time every 3 seconds.

Data collection software: We developed our own data collection App for both Android and IOS. The recorder operates at a sample rate of 48k HZ. The recording format is 16-bit PCM. Both the transmitter and receiver are set to use the mono channel. And the volume of concurrent music is around 50db-60db. These are commonly supported audio parameters in mobile phones.

There is also a data collection program running on a laptop for synchronous ground truth data recording using the respiration monitoring belt. It calls the Python SDK of GoDirect (Belt, 2023) and runs as a Flask Restful API on a Macbook. The API is invoked by the data collection mobile App to start the recording of sensing and ground truth synchronously. The hand gesture guiding video also runs an API on the laptop to start playing the video synchronously.

Sensing algorithm: As for the receiver side sensing algorithm, we select two classic ones for the sine and FMCW signal separately. Note that we aim to show the effectiveness of CoPlay from the perspective of downstream task accuracy here, while many other factors could affect the accuracy, including the receiver-side sensing algorithm. So, the two selective ones are a baseline to show the relative effectiveness of our CoPlay V.S. baselines; Improving the receiver-side sensing algorithm is not the focus of this study.

As discussed above, almost all receiver-side algorithms are based on the Fourier transform in some way. Therefore, for the respiration rate detection, we play a sine wave signal at 18kHz and extract that frequency bin in the received signal, whose amplitude is proportional to the chest-to-device distance. Next, we apply peak detection on the amplitude waveform to get BPM. We use the basic Scipy peak detection function with default parameters; it finds all local maxima by simple comparison of neighboring values. Besides, to eliminate noisy motion other than the breath -for example, body or head rotation- we apply a band-pass filter before peak detection. The filter selects out 8-22Hz because the human breath rate is typically 12-18Hz. Specifically, we use the same window size of 30s and interval of 10s as the Vernier belt ground truth. The beginning ¡30s are padded.

For the gesture recognition task, we transmit FMCW of 18k-20kHz and then dechirp the received signal with cross-correlation to get a range profile. At last, each window subtracts its previous window to eliminate the static environment and let the motions stand out. Note that nearby motions other than the hand would degrade sensing performance. So, we ask the users to try their best to stay still, since robustness under noise is not the focus of this study; we control these conditions to isolate the effectiveness of CoPlay only. Finally, the range profile is input into a simple CNN classifier (Document, 2023) as a single-channel 2D feature map. The CNN is a simple 2-conv-layer network with max pooling, ReLU, plus three linear layers at the end. Then by training with Adam optimizer and cross-entropy loss, we got a 3-class classifier. Besides, to avoid overfitting on our small dataset, we augment the feature map by adding Gaussian noise in the training dataloader. This regulation helps the training loss converge fast and stable. The classifier is trained in a leave-one-user-out manner and the following results are the average accuracy.

In the study, we first brief the participants on our IRB and study procedure without telling them the order and content of the audio to be played. Then we conduct 2 sessions of data collection; each has 6 sub-sessions. Upon finishing each sub-session, we ask the user to fill in a qualitative questionnaire about the music quality they heard. In total, each user collects 12 minutes of data and we have around 2 hours total data from 12 users.

For each user, the 2 sessions of data collection cover the two tasks of respiration and gesture detection separately. Within each session, there are 6 sub-sessions that are 6 minutes in total. The first three sessions run on the Android phone, playing 1) sensing signal only, no-music 2) audio + cognitively-scaled sensing 3) audio + sensing (cause clipping). The other three sessions run on the iPhone, playing the same audio except that the third case will cause down-scaling instead. Sub-session 1 and 3 are baselines to compare with our method in sub-session 2. Sub-sessions 2 and 3 use the same audio clip, which consists of samples from the datasets we used in the micro-benchmark section. It concatenates three 20-second clips from each of the three types of audio, including piano, hot songs, and speech. At the beginning of the first session, the user starts by putting on the belt and sitting in front of the device. Then they click the button on the phone screen to start recording. In between the sub-sessions, we switch the phone. Next, for the second session, they can take off the belt. Likewise, they will also click the button to start recording and playing the guiding video. Upon finishing every 3 sub-sessions, the user fills in the qualitative questionnaire about music quality and volume.