ExtPerFC: An Efficient 2D and 3D Perception
Software-Hardware Framework for Mobile Cobot

In order to build a more reliable software-hardware framework for mobile cobots, we define our design goals are:

• •

  Reusability and simple integration into systems with mixed versions of OS and middleware: Linux & ROS.

• •

  Distributed computing with load balancing awareness.

• •

  State-of-the-art ML models deployed with optimal configurations of cameras and processing devices.

Hardware is often compatible with a specific Linux kernel, and a specific ROS distribution is only provided to a specific Linux version. For this reason, selecting hardware concurrently with selecting Linux kernels implies narrowing down options in choosing a ROS distribution for developers. Unfortunately, not all hardware works well with the same Linux kernel, leading to using various ROS versions in the same system. Therefore, calling the same APIs from different ROS distributions may cause backward incompatibility issues due to a slight change in the function prototype and the underlying implementation of that supported API. We adopt message conversion [27] between multiple communication protocols to implement the message translator between ROS versions.

The driver incompatibility problems can be solved by using suitable Linux kernels supporting these devices’ drivers. However, it may raise backward incompatibility between a certain ROS distribution and APIs from other ROS distributions. Indeed, we encountered these compatibility issues with the Baxter robot (Ubuntu 14.04 and ROS Indigo) when executing motion APIs on Linux machines with Ubuntu versions other than Ubuntu 14.04. Therefore, an OS bridge between APIs from different ROS distributions is needed.

With the increasing requirements of computational tasks, a single computer system may no longer fit into a robotic application since the failure of that computer would cause the entire system to crash. We adopt distributed computing with load balance awareness between computers by periodically broadcasting the status to their peers. With this scheme, we can obtain load balancing at run-time. Two aspects must be considered for robots with multiple vision cameras: (1) bandwidth to acquire data and (2) processing power to perform detection or recognition tasks. We propose a method for selecting the optimal configuration at run-time for the combination of cameras, ML models, and devices.

As illustrated in Fig. 2, we structure the software-hardware framework as four different layers: hardware layer, OS layer, software framework, and application layer.

1. 1.

   Hardware Layer includes distributed computers, sensors (cameras, tactile sensors, LiDAR sensors), IMU, actuators, and ML accelerators and how they interface with each other by USB and Ethernet ports.

2. 2.

   OS Layer contains an OS and device drivers that support connecting devices at the hardware layer.

3. 3.

   Software Framework is the core contribution of this work. It bridges different Linux and ROS distributions, selects optimal configurations of sensors and ML models to available hardware to operate in real-time, and supports basic functionalities of the robot system, including map building, localization, navigation, motion planning, and arm moving for grasping. Most of the components are implemented on top of ROS and OpenVINO.

4. 4.

   Application Layer allows users to control and monitor the robot via a GUI, as shown in Fig. 4, which is written using the PyQt5 toolkit. The users can manually control the robot’s arms and joint positions, navigate the mobility base, and program the robot using Python. Also, native simulations such as RViz can be used to monitor the robot.

Problem Formalization: We need to distribute processing of data from $n$ sensors $S=\{s_{1},s_{2},...,s_{n}\}$ into the distributed computers. Each distributed computer could contain a CPU, a GPU, and a VPU, and we denote these processing units as $d_{i}$s in $D$. The image stream acquired from a sensor comes from either a USB bus if that sensor is connected to a device via a USB port or from the network via an Ethernet switch, in which a device may ask the neighbor device to share the image stream. Also, we define each configuration as $c_{i}=[s_{i},d_{j}]$, where $s_{i}$ is from $S$ and $d_{j}$ is from $D$.

We first sort both $S$ and $D$ lists in descending order in terms of input image sizes and computational power, respectively, as depicted in sort_by_image_size(S) and sort_by_comp_power(D). We then establish configurations for sensors and devices by nested-looping through $S$ and $D$ and check the following booleans, as described below:

If sensor $i$ interfaces to device $j$ via Ethernet or USB, we can group them as one configuration, as configuration $M$ (with USB port interface) and configuration $N$ (with Ethernet interface) in Fig. 3. However, if the sensor $i$ does not directly connect to the corresponding device $j$, we will find an already-connected device as an intermediate device $k$ to share the sensing data stream with device $j$. Thus, in this case, we group that sensor with the processing device $j$ and the intermediate device $k$ as one configuration as configuration $P$ in Fig. 3.

Note that after completing grouping each configuration, we also have to remove the configured elements from $S$ and $D$ lists: delete_from_list(s_i, S) and delete_from_list(d_j, D), and hence recursively call the hardware_selection(S, D) function until either $S$ or $D$ list is empty. The pseudocode of the algorithm is presented in Alg. 1.

Deploying the robot in an unknown dynamic environment with high uncertainty in robot perception (e.g., cameras, LiDAR sensors) requires a robust simultaneous mapping and localization (SLAM) algorithm to attain the highest map-building and localization accuracy. Moreover, building an environment map and localizing within that map in a robot system is essential to enabling path planning and path execution and later avoiding obstacles in robot navigation. In the Baxter robot, we examine the reconstructed maps by using various ROS-based SLAM algorithms on the VLP-16 LiDAR sensors and prioritize GMapping for its highest accuracy and availability in multiple ROS versions.

The main goal of GMapping is to compute the posterior over maps and trajectories $p(x_{1:t},m\text{ }|\text{ }z_{1:t},u_{0:t})$ given the estimated posterior $p(x_{1:t}\text{ }|\text{ }z_{1:t},u_{0:t})$, where $x_{1:t}$ are robot poses up to time $t$, $z_{1:t}$ is the observation up to time $t$, and $u_{0:t}$ is the control sequence. The GMapping is modeled as:

In specific, the posterior $p(x_{1:t}\text{ }|\text{ }z_{1:t},u_{0:t})$ is treated as a particle that is associated with a proposal map using the Rao-Blackwellized particle filter (RBPF). Meanwhile, the posterior over maps $p(m\text{ }|\text{ }z_{1:t},x_{1:t})$ is built given the robot poses $x_{1:t}$ and observations $z_{1:t}$. As the robot obtains observations and control measurements, the RBPF for GMapping is iterative, repeating the following steps: (i) sampling, (ii) importance weighting, (iii) resampling, and (iv) map estimating [21].

To navigate the robot in an environment, we reconstruct the global cost map and local cost map, where obstacles appear. The trajectory from source to destination is generated while minimizing the path cost in both maps. In this work, we inherit these ROS-based features and translate them into our control command in the application layer (our APIs).

The main objective of arm motion planning is to find a trajectory from the end-effector source position to the desired positions avoiding collision and minimizing the path cost and time complexity under the constraint that every point in the trajectory must have an inverse kinematic solution. In dealing with efficient arm motion planning, many methods have been used [13, 14], such as AtlasRRT and CBIRRT2. Most of them are well-supported by MoveIt!. Thus, we reuse them for the implementation of our work.

To control the robot, we not only command from one computer but also command from other computers for synchronization tasks. Furthermore, working on multiple computers requires considerable steps like login using SSH, editing source code, and uploading and executing code. These steps are repetitive and time-consuming, becoming a burden for developers. We build a graphic user interface (GUI) application, as shown in Fig. 4, in this framework to eliminate these burdens while developing robotic software.

The key idea is that this GUI application can run on a remote computer (Windows or Linux) and is able to connect to available on the robot so that it can execute specific tasks on each computer remotely. For example, tucking and untucking robots must run programs on the Bater robot, while moving the arm from one position to another can be conducted on another computer. Yet, another advanced tool is the program manager, allowing developers to edit Python code on a remote computer and execute their own customized code. This GUI application eventually saves the programming time of developers by getting rid of repetitive routines, thus making robotic software development more productive.

To allow the remote GUI applications to execute a command or a Python script, a gateway on the robot system is needed. This gateway talks directly to other computers via ROS message and ROS bridge adaption layer. In specific, it operates as a translator between the GUI application and the entire robot system by translating the commands from GUI to robot system commands and delivering commands to specific computers. In the meantime, it acquires all information about the robot system from distributed computers and periodically sends it to the connected remote computer.

To perform real-time detection and recognition of objects with a limited richness compared with those in open datasets like ImageNet and MS COCO datasets, we adopt two strategies: transfer learning [22] and single state detection [16]. We first transfer knowledge from a rich domain feature into a sparse domain feature, which represents our dataset. Herein, we then mathematically model the overall concept of transfer learning and address the two questions: (i) what to be transferred between models and (ii) how to transfer that knowledge.

Let $P(Y|X)=f(X,\beta)$ where $f$ is the task function. The minimizer for the trainable parameters, $\beta$, is written in terms of the loss function, $L(\cdot,\cdot)$, and the task function, $f$, as follows:

With respect to DL and computer vision concepts, we divide the task function into two components: feature extraction (backbone) and detection (head), such that $f(X,\beta)=(f^{D}\circ f^{F})(X,\beta^{D},\beta^{F})$ where $f^{D}$ and $f^{F}$ are task function for detection and feature extraction, respectively, and $\beta^{D}$, $\beta^{F}$ are parameters for detection and feature extraction, respectively. The analogous minimizer for $\beta_{t}^{F}$ and $\beta_{t}^{D}$ is:

Since features in the source domain are more generalized and sufficiently cover our target domain, we assume that the feature space in the source domain and target domain are similar. However, our target labels are different ($\mathcal{Y}_{s}\neq\mathcal{Y}_{t}$) since we retrain the models with in-lab objects (cone, cube, and sphere). Here, we utilize two transfer learning strategies: (1) instance transfer, where the marginal distribution of source features is different from that of target features, and (2) feature representation transfer, where we fit the source feature domain into the target feature domain (Fig. 5).

To implement instance transfer, we transfer $\left(\beta_{s}^{D},\beta_{s}^{F}\right)\rightarrow\left(\beta_{t}^{D},\beta_{t}^{F}\right)$, where $\beta_{s}^{D}$ and $\beta_{s}^{F}$ are resultants from source task functions, and fine-tune $\left(\beta_{t}^{D},\beta_{t}^{F}\right)$ using Eq. 2. For feature representation transfer, we separate the source task into two components (backbone and head) and transfer the entire source task’s backbone into the target task. In specific, we transfer $\beta_{s}^{F}\rightarrow\beta_{t}^{F}$, and train $\beta_{t}^{D}$ using Eq. 2. We also train with randomly initialized weights as a third strategy for accuracy comparisons in Sec. VI.

We obtain depth images and RGB images simultaneously from the Intel RealSense D435i camera, which also well-handles the depth image creation process, including camera calibration, image rectification, and disparity computation.

As the whole predicted bounding box also covers non-detected objects, we average the depth of the bounding box would incur estimation errors. We, therefore, scale the bounding box small enough at the center of the box, then calculate the estimated depth of the object, $D$, by averaging depth values of each pixel in the scaled region as follows:

Note that, in this framework, we choose the scaled region as $20\times 20$ pixels to guarantee optimal selection for computational power while maintaining the correctness of depth estimation.

To verify the correctness of our proposed method, we first collect data from three in-lab objects: cones, cubes, and spheres. We then label them with annotations in Pascal VOC format and split our custom dataset into three sets: training set (70%), validation set (20%), and test set (10%).

To evaluate how well the transfer learning strategies perform during the training periods, we calculate the validation loss, average precision (AP), and mean average precision (mAP) for MobileNetv1 [23], MobileNetv2 [24], SqueezeNet [25], VGG-16 [26], and YOLOv7 [17]. We train each model with three different strategies, as illustrated in Fig. 5. Data augmentation is used in a preprocessing procedure to enrich the training dataset, including rotation, cropping, and color distortion. We also evaluate the detection performance on test sets using AP and mAP calculated based on multiple intersections over union (IoU) thresholds.

For each test image, the IoU is defined as $IoU={(B_{p}\cap B_{t})}/{(B_{p}\cup B_{t})}$ where $B_{p}$ and $B_{t}$ are predicted bounding box and ground truth bounding box, respectively. If an IoU of a prediction is greater or equal to the predefined threshold value, that prediction is classified as a true positive (TP); otherwise, it is counted as a false positive (FP). Since the testing stage produces multiple detections among the classes, we first sort the confidence scores of all predictions in descending order as we need the trend of recall scores ($P$) to ascend while the trend of recall scores ($R$) descends in the P-R curve. Hence, we compute $P$ and $R$ for predictions of proposal performance for each class, $P=cTP/(cTP+cFP)$ and $R=cTP/GT$, where $cTP$ and $cFP$ denote cumulative TP and cumulative FP, respectively, and $GT$ is each class’s ground truth. Thus, the AP for each class is computed as $AP_{j}=\int_{0}^{1}\text{p}(r)$, where $j$ denotes for class ”j”. However, the values we obtain are discrete; we defined the interpolated area under the P-curve along the R-axis in the P-R curve in terms of recall levels, $r_{j}$, as:

The IoU thresholds range from 0.01 to 1.00 with a step of 0.01. After evaluating detection proposals on all IoU thresholds, we calculate the mAP for each model as below:

where $c$ is the number of classes in our training set.

The AP and mAP results on multiple IoU thresholds ranging from 0.01 to 1.00 with a step of 0.01 are shown in Fig. 7.

Training Performance: We train MobileNetv1, MobileNetv2, SqueezeNet, VGG-16, and YOLOv7 on the NVIDIA GTX 3090 (24 GB) GPU with the three strategies mentioned in Sec. V with 1000 epochs. The trained models start to converge at the $200^{th}$ epoch and finally converge at the $800^{th}$ epoch, taking approximately 39 minutes. Note that YOLOv7 is trained with a different loss function than other models, so we only compare YOLOv7 in prediction performance. Fine-tuning gives the lowest loss among the three methods, and VGG-16 obtains the best results among models. Indeed, VGG-16 learns the best with fine-tuning transfer learning strategy, as shown in Fig. 6.

The demonstration video includes two scenarios: (1) the robot grasps an object while estimating the depth of the detected object (Fig. 10), and (2) the robot performs SLAM while following a person using a face recognition module (Fig. 11) running on the Intel NC2: https://youtu.be/q4oz9Rixbzs.