Mechatronic generation of datasets for acoustics research

Central to the task of designing an experiment is the ability to accurately specify an acoustic scene. The volume of information required to fully characterize such scenes makes concise representation challenging, thereby increasing the tedium of creating and iterating experiment designs. Our solution is to provide an object-oriented application programming interface (API) in Python, modeled loosely after pyroomacoustics [9]. Any audio or robot device in MARS is dubbed a component, each of which offer a range of actions that can be requested via instructions. A first pass of the experiment can be done without calling upon the actual equipment, so that logs are provided nearly instantly. We find that this streamlines the process of iterating experiment designs.

Remote access to a mechanized room poses several safety concerns. In particular, the possibility for user-defined experiments to damage equipment must be addressed. A digital twin of the system is developed in Gazebo: a simulation toolbox and physics engine commonly used in robotics [12], such that experiments may be simulated in advance to check for collisions. This tool also provides a visualization of the physical space, as shown in Figure 2. We anticipate this tool to be instrumental in the design and execution of a new class of robot-enabled audio experiments.

We operate the MARS prototype using a virtual interface, which lets us design and run experiments remotely. Experiments may involve a large number of devices, or have a duration spanning multiple days, thus a monitoring tool is provided to track experiment progress, report system status, and stream a camera feed of the recording space.

To convert descriptions of experiments into audio data, MARS must offer positioning, playback, and recording capabilities. Our prototype does this by providing an extensible API that defines such behavior for an arbitrary set of devices.

The positioning of microphones and loudspeakers within an acoustic scene is done using multiple robots, which can range from simple linear actuators to complex pulley-driven systems. These robots must move to requested positions at specified times while avoiding collision. Collision avoidance in multi-robot systems is well studied [13] and can be highly sensitive. Fortunately, MARS controls a known environment where motion planning can be done in advance. Additionally, for the sake of repeatability, the motion provided by MARS is mostly described by the user, so there is little need for advanced decision-making and path-planning algorithms. These factors make MARS suitable for a centralized architecture, which can be more efficient than a decentralized one [14].

We acknowledge the need for MARS to provide access to state-of-the-art equipment. To this end, MARS is built atop the Robot Operating System (ROS): a collection of open-source robotics software that offers a publisher-subscriber communication architecture over IP [15]. To support device integration with MARS, we offer modular client and server objects that request and execute instructions, respectively. The use of client/servers in the handling of an experiment is shown in Figure 3. A single component can run many servers in parallel, each of which passes instruction data to an arbitrary callback function. This implementation is extensible: Updating a server to fit a new device only requires changing the instruction type and callback implementation, so devices relevant to audio research such as the Tympan [16] can be added with little overhead. Using ROS, wireless devices can be interfaced easily, making MARS suitable for research on cooperative listening with IoT devices.

Although the modified TCP protocol provided by ROS guarantees data integrity and order of arrival, large latency and jitter is observed, particularly on the low-power microcontrollers that drive many of the robots within MARS. This does not pose an issue for sequentially executed instructions, as order of execution is easily maintained. To support timestamped instructions, the prototype controller transmits messages in advance, which gives components ample time to receive and execute instructions according to their internal hardware clocks. Assuming synchronized hardware clocks across the network, this guarantees that robot motion occurs on a shared timeline. The effects of clock drift are minimized by running a local Network Time Protocol (NTP) server [17]. This approach allows for the creation of dynamic acoustic scenes, even on busy wireless networks.

Given that MARS is designed to provide spatial audio, we aim to record and validate the positions of robots in the environment. Recording this data can be achieved using ROS tools, namely rosbag. Validation requires ground-truth position data, which can be found using a camera system and visual fiducial markers [18, 19]. With the ground-truth positions known, position error can be calculated and used to interrupt experiments that cause the MARS system to behave erratically. Doing so serves as a form of quality control, wherein only high-quality, precise data is captured.

Complex experiments have the potential to involve a massive number of devices, each of which logs its own performance locally and writes audio files to internal storage. MARS offers the ability to consolidate this data onto the host machine, where it can be labeled and uploaded for the user to access. Data upload can be requested concurrently to experiment execution, allowing for validation and feature extraction even as an experiment is ongoing. This minimizes the downtime between requesting an experiment and receiving data.

MARS was used to take a dense spatial sampling of an acoustic scene, to demonstrate robustness. The acoustic head simulator was placed in various poses by the rail and turret. Over the course of around fifty hours, 40,000 three-channel audio files were collected at a sampling frequency of 48 kHz without human supervision or intervention.

Using the densely sampled MARS data, we verify the repeatability of MARS for static scene creation using four measurements corresponding to the same head pose. The recorded data was highpass filtered to remove background noise. The value of the maximum entry in the normalized cross-correlation (NCC) between two signals is used as a metric for similarity between recordings. To evaluate repeatability for multiple instances of the experiment, the NCC between the first recording and each of the others is calculated, as shown in Figure 4. The maximum entries of the NCCs had an average value of 0.98 with a standard deviation of 0.01. A maximum NCC value of 1.00 corresponds to perfect similarity between two signals, thus our results indicate highly repeatable performance.

To verify repeatability for dynamic scenes, an experiment was run where the spiderbot carried its speaker payload along a trajectory as audio was being played concurrently. Recordings were taken from a fixed microphone across ten repeated runs of the experiments. Across nine comparisons to a reference recording, the maximum NCC was 0.93 on aver age, with a standard deviation of 0.04. We observe that the repeatability was only marginally inferior when motion was introduced.

The dynamic experiment was repeated with the fixed acoustic head simulator in place of the microphone. The interaural time difference (ITD) remained constant across separate runs of the experiment, demonstrating reliable collection of spatial audio.