Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction

To navigate autonomously, traditionally a robot builds a map via 3D reconstruction (i.e., SLAM) and then plans a path using the map [20]. Recent works have developed navigation policies that make navigation decisions in a previously unmapped environment from egocentric observations directly without relying on mapping [4, 7, 21]. Some recent efforts have developed audio-visual simulation platforms [11, 18, 22] that enable embodied agents to both see and hear. In the AudioGoal navigation task [11, 17], the agent must navigate to the source of sound in an unknown environment. The state-of-the-art audio-visual navigation models train policies with reinforcement learning (RL), requiring millions of samples. Inspired by recent work in learning potential functions for interaction-free navigation [8], we propose to predict acoustic fields for interaction-free audio-visual navigation.

In robotics, microphone arrays are often used for sound source localization [23, 24, 25]. Past studies fuse audio-visual cues for surveillance [26, 27], speech recognition [28], human robot interaction [29, 30], and robotic manipulation tasks [31]. These solutions typically depend on analytical solutions for computing the direction of sound, whose performance deteriorates under strong reverberation and noise [32]. Recent works propose learning-based sound localization [33, 34], which however require collecting real data for training. In this paper, we show that we can transfer models trained in simulation directly to the real world.

Benefiting from recent large-scale datasets of real-world 3D scans [35, 36] and supporting simulators  [5, 37], recent works have shown success in enabling embodied agents in simulation. Transferring the model trained in simulation to the real world is thus of great interest. The mostly widely used approaches are domain randomization [38, 39], system identification [15, 40], and transfer learning and domain adaptation [41]. Most sim2real research studies transferring a policy from simulation to the real world based on visual input. Recent work [16] does sim2real transfer for audio-visual navigation by applying data augmentation empirically, which does not account for the effect of frequencies on sim2real.In this work, we systematically evaluate the sim2real gap by collecting data and identifying the spectral discrepancy.

We first introduce the simulator that we are using. SoundSpaces 2.0 [18] is a state-of-the-art audio-visual simulation platform that produces highly realistic audio and visual rendering for arbitrary camera and microphone locations in 3D scans of real-world environments [36, 42, 35]. It accounts for all major real-world acoustics phenomena: direct sounds, early specular/diffuse reflections, reverberation, spatialization, and materials and air absorption.

In audio-visual navigation, the goal is to navigate to a sounding object in an unknown environment by seeing and hearing. The location of the sound source is not known and needs to be inferred from the received audio. At every step, the agent needs to sample an action from {MOVE_FORWARD, TURN_LEFT, TURN_RIGHT, STOP}. If the agent issues the STOP action within a 1m radius of the goal location, the episode is considered successful.

There are different instantiations of the audio-visual navigation task, each with its goal specification. For example, AudioGoal [11] requires navigating to a static target in a discretized environment, while Dynamic AudioGoal [43] requires navigating to a moving sound source. We target the Continuous AudioGoal navigation benchmark introduced in SoundSpaces 2.0 [18], which generalizes the state space to be the continuous environment.

Transferring a navigation policy trained in simulation to the real world is not trivial due to many domain gaps between the simulation environment and the real world, which include the visual discrepancy, the physical dynamics discrepancy, the robot actuation discrepancy and—specifically in this task—the acoustic discrepancy.

We focus on investigating the acoustic discrepancy and to bridge other domain gaps (e.g., visuals and physics), we take a hierarchical approach that disentangles navigation into high-level path planning and low-level motion planning. This has a few benefits: 1) disentangling the policy makes it possible to utilize existing SLAM algorithms on the real robot to abstract away domain gaps other than the audio. 2) disentangling the policy makes the intermediate output more interpretable and easier to debug 3) specifically in this work, posing the high-level planning as a supervised prediction task makes it easier to measure the sim2real difference because we can evaluate the performance by collecting real measurements without repeatedly running robots.

The key challenge here is to formulate the proper waypoint prediction task that could survive the sim2real transfer. One existing approach [12] predicts the exact location of the audio goal directly, which is however an ill-posed problem since the environment geometry is unknown. For example, when the audio goal is in another room, the received audio reveals the direction to the door rather than the exact direction of the goal. Instead of predicting the global audio goal location, we propose to predict the local acoustic field (sound pressure field) centered around the agent, which not only better captures the direction of the sound but also is more predictable from the visual observation of the environment.

Defining the waypoint prediction task however does not address the audio discrepancy directly. SoundSpaces 2.0 renders the audio propagation as a function of the geometry of the environment, the material properties, and source/receiver locations based on a bidirectional ray-tracing algorithm [44]. While it produces realistic audio renderings, there remains some difference between how sound propagates in the real world vs in simulation. It is known that ray-tracing-based algorithms yield worse performance with lower frequencies due to wave effects [19]. This implies the model needs to be aware of this spectral difference for sim2real. Thus we introduce a frequency-adaptive prediction strategy to help the model better transfer to the real world in Sec. III-E.

We define acoustic field prediction as predicting the top-down sound pressure field of $L\times L$ centered at the agent, given the egocentric depth image of $128\times 128$ pixels and a one-second binaural audio.

To tackle the acoustic field prediction problem, we first present a model that uses both audio and visual observations (see Fig. 2). The motivation behind using the visual sensor is that the visual observation of the surrounding environment can be useful in inferring the geometry of the environment, which affects the acoustic field. For example, walls often act as the boundary of the acoustic field. More specifically, we use a pre-trained ResNet [45] to extract features from the input image and reshape it with a $1\times 1$ Conv layer into a 1d vector of size $512$. For the input binaural audio, we first process the waveforms with Short-Time Fourier Transform (STFT) to convert the time-domain signal into the frequency domain. We then use a 2D Conv net to encode the features and then tile and concatenate with the visual feature. Lastly, we feed the final output through one linear layer and reshape the prediction into the size of the target acoustic field $L\times L$.

With this acoustic field prediction model, we then construct a hierarchical navigation pipeline (see Fig. 3) to perform audio-visual navigation, which executes the following steps: 1) sampling a long-term goal; 2) navigating to the long-term goal; and 3) making the stopping decision.

Existing audio-visual navigation models use all frequencies in the input audio. However, sound spans across a wide range of frequencies, and due to imperfect geometric simulation techniques, the acoustic gap varies as a function of frequencies. Models trained with all frequencies assuming them equally reliable would have lower performance when deployed on a real robot.

Given this observation, we first systematically examine how the gap changes as a function of the frequency. The idea is simple: with a given frequency band $[F_{1},F_{2}]$, we first train an acoustic field prediction (AFP) model in simulation using only that band, then test it on real-world data of the same sound and same band, and calculate the prediction error. We equally divide all frequencies into $N$ subbands ($N=5$ based on hyperparameter tuning), and we show the distribution of errors over the frequency bands in Fig. 4, where error is defined as the distance between the predicted max and ground truth goal location. As expected, the lower frequencies tend to yield larger prediction errors. However, the error does not monotonically decrease as frequency increases likely due to simulation errors. We also trained a model that uses all frequencies, which has a distance error of $0.86m$, underperforming the best frequency band.

With this measurement, the most intuitive idea would be just to take the frequency band that has the least sim2real error and train a model with that band. However, this will not work for real-world scenarios where some sounds span across many frequency bands while others only occupy a very narrow range of frequencies. To take that into account, we propose a frequency-adaptive prediction strategy that uses the best frequency band based on both the measured error and the energy distribution of the received audio.

Assume we divide all frequencies linearly into $N$ bands. Given a received audio $A_{r}$, we first convert it into the frequency domain and divide it into these $N$ bands. Based on the measured errors, we have a weighting function that assigns weights to these bands based on their sim2real errors:

$$p(i)=(\frac{1}{e_{i}})^{\alpha},i\in[1,...,N],\vspace{-0.05in}$$

(1)

where $e_{i}$ is the error in Fig. 4. For each subband $i$ of the input, we then compute another weight based on the energy of the band normalized with respect to the highest energy:

$$q(i)=(\frac{r_{i}}{r_{m}})^{\beta},i\in[1,...,N],\vspace{-0.05in}$$

(2)

where $r_{i}$ is the energy of that band and $r_{m}=\max_{i}r_{i}$. We basically assign higher weights to frequency bands that have more energy. Lastly, we take the product of these weights:

$$w(i)=p(i)\times q(i),i\in[1,...,N]\vspace{-0.05in}$$

(3)

We take band $i$ with the highest $w(i)$ to produce the final prediction. Both $\alpha$ and $\beta$ are hyper-parameters, and we perform a grid search to find the best values on validation.

Intuitively, what the weighting function does in Eq. 3 is: if the input sound has a fairly equal distribution of energies over all subbands, it will take the best band from $p(i)$ that has the lowest sim2real error. If the input sound has a very skewed energy distribution, it will prioritize taking the band where the audio has the most energy. In this way, we factor into both the measured difference and the spectral distribution of individual sounds.

For the size of the acoustic field, we set $L$ to $9$ with a grid resolution of $0.5m$, i.e., $4.5m\times 4.5m$ centered around the agent based on our ablations. $\alpha$ and $\beta$ are set to $5$ and $0.8$ respectively based on the validation performance.

We train the predictor with Mean Squared Error (MSE) loss till convergence. For optimization, we use the Adam optimizer [47] with a learning rate set of $0.001$.

SoundSpaces 2.0 supports computing the impulse response $I(s,r)$ between the source location $s$ and the receiver location $r$ as a function of the 3D environment but does not have direct API support for rendering the acoustic field. To compute the field, given a $(s,r)$ pair, we first sample a grid centered at the receiver location of size $L\times L$, and for each grid point $p$, we compute $I(s,p)$ which results in $L^{2}$ number of RIRs per receiver location. However, these RIRs are represented in the form of waveforms instead of single numbers. To best represent the sound pressure at each single point, we take the maximum amplitude of the waveform.

For sampling the source/receiver locations and environments, we utilize the existing audio-visual navigation episodes [11], which provides configurations of the environment and source/receiver locations. This dataset uses scenes from the Matterport3D dataset [36], which contain scans of real-world environments such as apartments, offices, and even churches. We sample 500 episodes per environment for the 57 training environments in the navigation dataset. We also perform a similar operation to curate the validation and test set. In total, we collect 1.1M/52K/52K samples for train/val/test. Along with these acoustic fields, we also render the RGB-D images at the corresponding locations. See examples in Fig. 6.

To measure the sim2real error, we collect real audio measurements to evaluate the trained model’s performance. For that, we use a 3Dio microphone to capture the binaural audio with a smartphone serving as the speaker output. We align the real-world parameters closely with those in our simulator, such as the height of the speaker and receiver. Since the simulator employs a mono receiver, the two-channel audio data we gather is transformed into mono format by averaging the amplitude values across both channels. This process is repeated for ten distinct speaker positions (8 different directions w.r.t the agent and two data points for when the speaker is near the agent). We also downsample the acoustic field resolution from $9\times 9$ to $3\times 3$ so that we can collect more data in more environments.

For the source of the sounds, we use two types of sounds: white noise and normal sounds. To compute the sim2real errors in Fig. 4, it is important for the sound to have uniform distribution across all frequencies, and we use white noise for that. For evaluating the final frequency-adaptive acoustic-field prediction model, we choose 7 unheard sounds that have varying spectral distributions and play them as the source. For each sound, we collect 10 data points. We split them equally into validation and test for hyperparameter searching.

We first demonstrate the effectiveness of our navigation system on the challenging Continuous AudioGoal navigation benchmark [18], where the agent moves in a continuous unseen environment to find the location of a ringing telephone sound. For metrics, we use the common Success Rate (SR), success weighted by inverse path length (SPL), and soft SPL. An episode is considered successful when the agent issues the stop action within 1 meter of the goal. SPL [7] is defined as $\mathrm{SPL_{i}}=S_{i}\cdot l_{i}/\mathrm{max}(p_{i},l_{i})$, where $i$ denotes the index of the episode, $S=1$ when the episode is successful and $S=0$ otherwise, $l$ denotes the length of the shortest path between the agent and the audio goal, and $p$ denotes the length of the actual path taken by the agent in the episode. Soft SPL is a variation of SPL where $S_{i}=1$ for all $i$.

We compare with the following models: DDPPO [21]: an end-to-end reinforcement learning policy trained with distributed proximal policy optimization. Direction Follower [9]: this model predicts the direction of the audio goal and navigates with the same waypoint planner. We stop the agent automatically when it is within a 1-meter radius of the goal. Gan et al. [12]: this model predicts the (x,y) location of the audio source and navigates using a waypoint planner. The agent stops whenever it reaches its predicted location or the closest navigable point. Beamforming [48]: classical beamforming method that calculates the direction of arrival of the sound and navigates with the same waypoint planner.

To further justify our model design choice, we also compare with the following ablations of our own model. AFP w/ predicting max: this model does not predict the whole acoustic field. Instead, it predicts a single point that represents the highest point of the local acoustic field. AFP w/ audio-only: this model only takes in the audio input, which tests whether the full model uses the visual information when predicting the acoustic field.

Results are shown in Tab. I. Our model strongly outperforms all baselines and ablations. Compared to DDPPO, our model is more efficient due to its hierarchical nature since the DDPPO model often gets stuck with obstacles and corners. Direction Follower and Gan et al. predict the goal direction/location directly, which is however ill-posed when goals are in some other room at a distance from the robot. As a result, their navigation performance is also pretty poor. For the Beamforming baseline, similar to ours, it also predicts the local direction of arrival of the sound; however, since it is not robust to reverberation and noise, it performs poorly. Lastly, the two ablations perform comparably to baselines but also underperform the full model, showing it is both beneficial to predict the full acoustic field and leverage visual sensors to understand the environment.

We show the comparison of trajectories with these baselines in the same episode in Fig. 5, where our model is more efficient in reaching the goal. We also visualize the acoustic fields of both the ground truth and prediction in Fig. 6. Initially, the model predicts high values at the corner (in the direction of the goal), and as the agent gets close to the goal, it predicts high values at the center of the field.

Here we evaluate our frequency-adaptive acoustic field prediction (FA-AFP) model on the collected real acoustic field data. We compare our method to the random baseline and ablations of our approach. We consider three ablations: “All-freq AFP” uses all frequencies for prediction. “Best-freq AFP” uses the best frequency band shown in Fig. 4 and “Highest-energy AFP” uses the band where the received audio has the highest energy. We measure the performance of different prediction errors with the angle and distance of the predicted max location on the acoustic field. We train our models on 73 sounds and test on 7 unheard sounds.

The results are shown in Tab. II. We show that compared to the random prediction, our All-freq AFP model reduces the prediction error drastically. If we always use the best frequency for prediction, it helps lower the angle error a bit but not the distance error. Using the frequency band with the highest energy brings down the prediction error more. Our frequency-adaptive prediction model (FA-AFP) improves the performance even further, showing the importance of intelligently selecting a frequency band for prediction.

In Fig. 7, we show examples of the collected acoustic field and the predicted acoustic field for multiple directions and sounds. Note that the acoustic field is only sampled at a $3\times 3$ grid centered at the robot to reduce the cost of collection. Our predictions are consistently accurate across examples.

Finally, to validate the whole navigation pipeline, we deploy our navigation policy on the real robot platform (described in Sec. V). When deploying on the real robot, one thing that differs from the previously collected real data is that the robot also makes some low-frequency noise while running. To address this issue, we collect recordings of the robot noise and perform data augmentation by adding the noise to the received sound during training to improve the model performance.

We conduct 20 navigation examples with various source/receiver distances and directions and show that our robot can navigate the sounding object with a 75% success rate. See the supplementary video for both success and failure cases. We also tried to deploy the best-performing baseline DDPPO, which however failed all the test scenarios, which is likely due to the significant physical sim2real gap since that model trains with RL end-to-end. We show one navigation step example in Fig. 1, where the model predicts the acoustic field correctly.