Robo-Platform: A Robotic System for Recording Sensors and Controlling Robots

A device might have several accelerometers, gyroscopes, and magnetometers. Each type of sensor is stored in a separate file. IMU (accelerometers and gyroscopes) and magnetic field sensors are considered separate sensor groups, and their data are stored in different folders (Fig. 5). The application can record pre-calibrated and raw, uncalibrated measurements. These sensors usually do not have any requirements. Other types of sensors, such as barometers, are not considered in the current version of the algorithm.

The Android platform initially provided GPS measurements as the source of location data. These measurements are stored in a specific file. However, raw GNSS has also been available in recent versions. For the raw GNSS readings, both Navigation and Measurement Messages are stored in separate files to allow the recovery of the device’s geographical location.

Most portable devices provide multiple cameras. Additionally, each physical camera can provide different output streams, e.g., one for preview and another for image processing and storage. According to the Android Camera2 API [12], a logical camera in a device that supports multiple camera outputs represents several physical cameras. In this case, the user can select any combination of physical cameras belonging to a specific logical camera group but not from the other groups.

Instead of saving a video stream for each physical camera, timestamped images are stored in a designated folder. A separate text file containing all image names and their corresponding timestamps is also stored. Since most devices are equipped with auto-calibration algorithms, a locking algorithm is triggered at the start of the recording session to lock the focal length and other parameters (white balance and exposure). Moreover, if the device supports lens distortion correction, this capability is also disabled for capturing high-rate raw images. Available parameters can be stored in the calibration file and retrieved for post-processing.

The platform also supports recording external ADC channels from a microcontroller via the USB. The microcontroller fills its output buffer when an ADC channel reports a new value. Then, the Android platform initiates an ADC report command, receives the ADC measurements as a raw array of bytes, and publishes the result, timestamped with the received time. Therefore, the data acquisition rate depends on the communication latency and the microcontroller’s configured sample rate.

The microcontroller device can also store the calibration data, such as the number of ADC channels, resolution in bits, and the sample rate. The Android platform can retrieve and store the calibration information after establishing a communication channel.

Fig. 5 shows the proposed folder structure for the recorded dataset. Each recording session creates a root dataset folder. Measurements from related sensors are stored in separate files in the same folder. The raw GNSS sensor records its observations in multiple files. Each physical camera stores a text file for timestamped image paths and a folder of all images. Each row of the text files that store readings from multiple sensors of the same type also includes a sensor identifier.

Table II summarizes the data format for each type of sensor. All measurements are stamped with the time the sensor value becomes available with nanosecond accuracy. All text files also contain a header describing the content of the file. Some data must be interpreted by the user. For example, what ADC values represent depends on the physical quantities the external sensors measure.

In Table II, the square bracket means the fields are optional. In the case of raw acceleration ($\frac{m}{s^{2}}$), angular velocity ($\frac{rad}{s}$), and magnetic field ($\mu T$), the first three columns are raw measurements before the optional sensor biases. For a comprehensive explanation of raw GNSS fields, please refer to [13].

In addition to the Android wireless server, a desktop controller server is presented. The logic is the same as the Android server explained before.

In addition to the Arduino microcontroller board, the proposed system can communicate with a custom AVR board through the USB channel. There are several hardware and software USB implementations for AVR boards. The proposed design is powered by the OBDEV’s V-USB software library [14]. The AVR module receives USB commands, decodes them, and executes the requested task. It can also load buffers with data (ADC sensor and configuration) and send them to the mobile application on demand. This software-based implementation can be very cost-effective since it obviates the need for an external USB controller chip and extra components.

Another popular option is to use third-party embedded boards such as an Arduino board. In this case, the USB communication incorporates Arduino’s standard serial transfer.

Sample datasets are recorded using various devices and sensor configurations. Then, the average and standard deviation of each sensor’s period is calculated using the timestamps of all similar measurement files and reported in Table IV. According to the results, most motion sensors can output data at about 100 Hz with acceptable accuracy (relatively small jitter). The external ADC can also operate at 100 Hz, which is beneficial in capturing rapid fluctuations in the measured quantity. The camera in this experiment can capture about three frames per second, much lower than a video captured at 30 fps. However, recording in this format is more reliable and necessary for some applications such as AR. Finally, GPS data exhibit the lowest frequency (approximately 0.25 Hz), which is expected with consumer GNSS sensors.

This section aims to answer the question, can modern state estimation methods process the noisy and sporadic sensor data from a mobile phone to reconstruct the scene and estimate the device pose? Several datasets are captured using the proposed method and fed into four visual and inertial state estimation algorithms. This process is repeated for other related datasets, and the tracking duration is calculated and reported as the percentage of the successful tracking time to the duration of a whole sequence.

ORB-SLAM3 [18] is a robust SLAM algorithm that supports several sensor configurations and camera models. The monocular visual and visual-inertial settings of this algorithm are considered in this experiment. Another popular state estimator is VINS-Fusion [19], used in the monocular visual-inertial configuration. Finally, Matlab’s asynchronous Extended Kalman Filter (EKF) state estimator [20] is used to fuse measurements from the accelerometer, gyroscope, magnetometer, and GPS sensors and estimate the pose of the device when no visual data is available.

EuRoC [4] and TUM’s visual inertial dataset [15] are popular SLAM datasets and benchmarks. The reference sequences Machine Hall 3 from the former and the Corridor-4-512 from the latter are considered in this experiment. The Public Event dataset [16] provides challenging sequences recorded using a DAVIS device. Only intensity images and IMU measurements of the sequence boxes_6dof are used in this study. The sequence urban25-highway from the Complex Urban Dataset [17] provides stereo images, inertial measurements, GPS data, and magnetic field readings. These datasets are recorded using a dedicated sensor system and provide a reference. ADVIO [6] is a relevant public dataset that records data from a configuration of multiple devices and sensors, including an iPhone mobile device. Finally, a dataset is recorded using the proposed Android application. Fig. 8 presents sample images of several sequences in this dataset. Table III describes the datasets used in this experiment.

These state estimation algorithms require calibration parameters and a correctly formatted dataset. Calibration is the process of estimating sensor model parameters (camera intrinsics and distortion and IMU noise and bias) and the spatiotemporal relation between sensors (e.g., camera-IMU transformation and time offset). SLAM and AR datasets usually provide these parameters by calibrating their sensors or sharing a calibration dataset that can be used to calculate these parameters manually. The sensor model of a calibrated dataset must match that of the SLAM algorithm. For instance, while ORB-SLAM3 supports the distorted pinhole and fisheye [21] camera models, a dataset that only provides a unified projection model [22] should be recalibrated for the correct parameters. A separate calibration dataset is recorded to estimate the calibration parameters for the proposed dataset using the Kalibr Toolbox⁷⁷7https://github.com/ethz-asl/kalibr.

The dataset’s output formats vary for different providers. Although the ROSBAG⁸⁸8https://wiki.ros.org/rosbag format tries to mitigate this problem by presenting data from various sensors as standard messages passed to different components, the installation of ROS is necessary for using it. Additionally, ORB-SLAM3 supports several other directory and file structures, including that of the EuRoC dataset. For Matlab’s EKF state estimator, measurements from the inertial, magnetic field, and GPS sensors must be aligned using the timestamps. Datasets recorded in unsupported formats must be converted, e.g., aligning separate accelerometer and gyroscope readings in the ADVIO and proposed datasets. Additionally, timestamped images should be extracted from the ADVIO videos. The bag_creater tool of the Kalibr Toolbox can create ROSBAG files from each sequence. The GitHub repository of the proposed system also provides several scripts for this post-processing step.

After these steps, the visual (V) and visual-inertial (VI) configurations of ORB-SLAM3, visual-inertial VINS-Fusion, and the inertial (I) Matlab’s state estimator are executed on each sequence. Tracking time, $\tau$, is the difference between the first and last timestamps of the estimated trajectory. This time is divided by the total duration of that sequence, $T$, to calculate the percentage time of tracking. Each method-sequence combination is repeated five times, and the median percentage is reported in Table V. A dash symbol (-) in this table indicates a tracking failure or unsupported sensor configuration (sequence koohestan-park)⁹⁹9Video: https://youtu.be/BTQ4yLB1bak.

Although ORB-SLAM3 and VINS-Fusion can track EuRoC and TUM-VI sequences with acceptable accuracy, their performance degrades in challenging conditions. VINS-Fusion and inertial EKF can follow almost all sequences with a high percentage. However, the estimation error always overgrows in the latter case, and the same is true for the VINS-Fusion in challenging situations. This issue is particularly severe for purely IMU-based sequences because, without the magnetic field and GPS measurements, inertial EKF fails to correct IMU errors. The accuracy of the methods is not reported here because it assesses the limitations of algorithms rather than datasets.

The visual-inertial ORB-SLAM3 fails to track most sequences, while its visual configuration shows poor performance for the challenging sequences. Interestingly, VI ORB-SLAM can better follow the sequences recorded by the proposed platform. The sequence advio-19 shows a sudden jump in the V ORB-SLAM results because it fuses a few loop-closing poses at the end of the sequence with the initial estimates.

Tracking results strongly depends on the limitations of each method. The optical-flow feature tracking and association pipeline of VINS-Fusion is sensitive to fast-paced motion, which happens frequently in the low frame rate of the proposed sequences. On the other hand, although the descriptor-based feature-tracking ORB-SLAM3 can cope with high baseline images, it is sensitive to lighting and illumination changes in the environment, which is the case in the sequence iaun-corridor4. Current SLAM algorithms are optimized for several datasets recorded with high-quality sensors and optimal calibration; however, their performance degrades rapidly with noisy data and calibration parameters. The reverse is also true: knowing the limitations of a SLAM algorithm, it is possible to record an ideal dataset that favors a particular method.

Based on this discussion, we need robust general state estimation algorithms capable of processing low-quality data without requiring the user to go through the complex calibration procedure. The optimal hybrid method is a delicate balance of simple, low-precision, fast algorithms such as an EKF for high-frequency and noisy data and accurate but slower methods for processing sporadic and feature-rich data such as images.

Four communication types in the proposed system are the Arduino USB serial, software-based V-USB implementation for the custom AVR board, Wi-Fi, and Bluetooth. To test the latency of each network type, a packet containing the current timestamp identifier (a number) is sent from the Android device to the destination device (the microcontroller board or wireless server). This message is echoed back to the Android device to compare its arrival timestamp with the original message. This operation is repeated ten times for 100 timestamps (a total of 1000 messages), and the average timestamp difference is reported as the latency of the channel.

To measure throughput, 1000 packets of a predefined size are sent from the Android device to the target. Each packet contains a specific pattern of bytes. This packet is decoded in the target, and the number of successfully received bytes is sent to the Android device. The channel throughput is the number of successfully transmitted bytes divided by the duration of this test. The buffer size is increased for all modes of communication except the V-USB. For this configuration, the buffer size is kept at 64 bytes for all experiments because the current implementation of the proposed system cannot support buffers greater than or equal to 256 bytes. In this case, a train of $N_{s}$ consecutive 64-byte packets is sent to the target before a synchronous response for the number of received bytes is requested. Here, $N_{s}$ is the buffer size divided by 64, e.g., four in case of a buffer of 256 bytes.

For this experiment, an Android phone (Samsung Galaxy A20) establishes a Wi-Fi connection with a laptop controller server through an access point (Wi-Fi router) while it is directly paired with the desktop system for Bluetooth communication. Three types of Arduino devices considered here are Arduino Uno, Due, and Mega 2560. They are connected to the phone via a USB cable and operate at 115200 bits per second. The custom V-USB-powered AVR board is an Atmega32 with a clock frequency of 16 MHz.

Table VI reports the results of this experiment. The Arduino board exhibits the lowest latency, while, as expected, Bluetooth communication demonstrates the highest value among these channels. The custom board almost has the same latency as the Wi-Fi connection and is in the mid-range. Although not explicitly shown in the table, the latency results for other buffer sizes are consistent. However, wireless communication shows a higher variance when changing the buffer size.

According to the table, throughput increases linearly with buffer size. The custom V-USB microcontroller has the highest throughput for small buffer sizes, while the Wi-Fi channel can provide high throughput for large buffers. The throughput of the Bluetooth connection is always lower than that of Wi-Fi, as expected. Also, the throughput of the V-USB board is consistently higher than that of the Arduino board for smaller buffers but changes slowly with an increase in buffer size. A future implementation of this board that increases the actual buffer size might improve the results. Based on these experiments, only the high-end and more expensive Arduino boards (such as Arduino Mega 2560) can provide enough memory for large buffer sizes.

In conclusion, the Arduino board is the preferred option for its low latency and high throughput using large buffer sizes in high-end boards, while the V-USB board is suitable for low-cost solutions. Moreover, when a Wi-Fi wireless connection is available, it is preferred over Bluetooth, as it offers high throughput and moderate latency.

Finally, a ground vehicle and a quadcopter are manually controlled using the proposed system. In the first experiment, a Samsung Galaxy J5 device is attached to a V-USB-powered Atmega8 AVR board, which provides low-level control signals that are sent to a toy car’s electrical DC motor through a driver (Fig. 9). This car is controlled by two digital signals, one to enable/disable the device, and the other for forward/backward-rotation motion. These commands are issued wirelessly using the companion desktop application. The Android application processes the commands and sends the final driver signals to the microcontroller to be applied directly to the outputs¹⁰¹⁰10Video: https://youtu.be/BTQ4yLB1bak.

Next, an Arduino board generates the PWM signals to control a quadcopter. The Android application receives the current state (the four PWM strength values) from the Arduino board, control commands from the wireless server, and IMU readings to calculate the device’s orientation and update its state. These three inputs are combined to generate the desired PWM strength sent to the Arduino and directly applied to the device’s PWM outputs. In both applications, the Android phone and microcontroller board are rigidly attached to the mobile robot and move with it. These experiments show that the proposed versatile platform can manually control various robotic systems.