DARE: AI-based Diver Action Recognition System using Multi-Channel CNNs for AUV Supervision

Since on-demand reprogramming of the AUVs is difficult in dynamic and uncertain underwater environments, they require constant supervision from the diver to perform tasks. Traditional methods of communicating and controlling the AUVs utilize waterproof tablets, keyboards, mice, or joysticks directly connected to the AUVs. These interfaces, however: 1) are expensive to waterproof and deploy in underwater environments, 2) require the diver to be close to the AUV, and 3) are cumbersome and unwieldy to operate. Therefore, due to lack of effective technologies of underwater radio and wireless communication, one way for the diver to send commands to an AUV is by means of hand gestures [19]. In response, the AUV can utilize AI to recognize the diver’s hand gesture and interpret the corresponding command, as seen in Fig. 1. Sometimes the AUVs are also required to follow the diver by recognizing diver’s posture. It is desired that such diver action recognition system implemented on an AUV should be fast to act in time-critical situations, such as changing environments, risky diver locations, and limited oxygen supply.

There exist several challenges in building a robust, reliable and real-time underwater diver action recognition system, as highlighted in Fig. 2. These challenges are discussed below.

Remotely operated vehicles (ROVs) are common underwater robots which are directly controlled by a human operator from the surface via cable connections. These vehicles, however, are limited in their deployment and autonomy due to expensive equipment, short range, unwieldy cables, and complex operation [20]. On the other hand, autonomous underwater vehicles (AUVs) are cable-free, equipped with advanced sensing and control technologies [21], provide longer range autonomy and are becoming increasingly more intelligent and cost effective. Thus, they have become useful to serve as advanced assistants for a variety of underwater tasks. However, despite the advances in AUV technologies, several underwater tasks require efficient diver-AUV collaboration. The key requirement for diver-AUV collaboration is the capability to dynamically reprogram the AUV’s task parameters. In underwater environments, radio and wireless communications between the diver and AUVs[22] are not efficient. Traditionally, divers used tactile devices to send commands using waterproof tablets, keyboards, mice and joysticks; however, these interfaces are expensive and inconvenient [19]. Therefore, the underwater diver action recognition problem has become an increasingly important area of research to enable efficient underwater diver-AUV interaction.

There have been some initial attempts to address the diver action recognition problem, which requires: 1) the creation of a database of diver actions in diverse and realistic underwater environments, 2) the use of cost-effective but information-rich underwater sensors to capture diver motions, and 3) a computationally efficient and reliable diver action recognition model. Initial studies focused on the classification of diver actions were based on images captured in ideal swimming pool environments [23] with controllable lighting and background conditions. These were later extended to include diver actions in more realistic underwater ocean environments [24]. In both cases, images were captured using a single monocular RGB camera. While this sensing solution is cost-effective, this system has several limitations including blind spots from a single point-of-view, low image resolution, and limited depth information which can make the system unreliable.

The diver pose recognition problem is also crucial for diver tracking application [25]. Traditionally, acoustic sensors are used to locate the diver’s position relative to the water surface for autonomous surface vehicles; however, such sensors cannot identify specific diver poses (e.g., orientations and motion types). Similarly, information-rich sonar, ultra-short baseline acoustic localization, and stereo-cameras have also been used for AUV diver tracking. These sensors ensure the AUV closely follows the diver underwater; however, diver action recognition models were not developed to identify specific diver poses present in the captured images[26]. At the same time, research has also focused on detecting the number of divers present in the view of the AUV camera in various underwater environments [24, 27]. Researchers have also addressed the diver arm motion recognition problem using wireless acoustic networks in a simulated underwater environment [28]; however, there is no real-world data for validation. Furthermore, these networks might not always be available for implementation in the ocean environments.

Common underwater recognition models utilize time-frequency feature extractors with k-nearest neighbor and SVM classifiers to identify mammals and inanimate objects on the ocean floor [29, 30, 31]. Researchers performed gesture recognition by fusing convex hull method and SVM classifier [32]; however, very few gestures and no poses were considered. As such, the method might not scale well to larger and more complex set of diver actions. On the other hand, deep transfer learning methods have been applied to underwater fish species classification [33], human motion recognition in terrestrial environments [34] and diver gesture recognition[35]. While these methods provide good average performance, high individual class performance is not guaranteed.

However, for the diver action recognition problem it is of critical importance that each diver action is recognized with high accuracy for safe and reliable operation in underwater environments. To the best of our knowledge, the diver action recognition problem is still an active area of research.

To address the challenges and limitations discussed above, this paper develops a robust, reliable and computationally efficient Diver Action Recognition System, called DARE, to identify the diver hand gestures and full-body poses in real-time; thus, enabling the AUVs to reliably interpret diver commands. DARE has been trained based on a recently created rich database of diver hand gestures and poses, called Cognitive Autonomous Diving Buddy (CADDY) [36] dataset, which contains data generated in real uncertain ocean environments.

DARE is built upon deep transfer learning architecture for robust classification of a multitude of diver actions under various underwater uncertainties and complex scenarios. First, DARE enables fusion of information available from the left and right images of the stereo camera system on AUVs. For this purpose, the deep transfer learning method is extended to a multi-channel framework that: 1) extracts the relevant features from each stereo image individually using the convolution layers, and 2) optimally fuses these features together using fully connected neural networks.

Second, DARE provides high classification performance of each individual class. Typically, a single monolithic neural network classifier is trained after the convolution layers; however, for problems with large number of classes, such networks cannot guarantee a high minimum performance for each individual class. Therefore, instead of considering all diver actions simultaneously, DARE trained a decision tree of fully-connected neural network classifiers, where each network is tailored to discriminate between groups of diver actions. This ensures high individual class recognition performance at the bottom of the tree, thus yielding superior reliability.

The main contributions of this paper are as follows:

• •

  Development of DARE, to facilitate computationally efficient, robust, reliable and highly accurate diver action recognition, under various environmental, diver, and sensing uncertainties. It consists of:

  • –

    a multi-channel deep transfer learning architecture for fusing stereo pairs of camera images and

  • –

    a hierarchical tree structured classification scheme to yield high individual class recognition performance.

• •

  Training and testing of DARE using the CADDY dataset consisting of a large number of diver action images in real-life challenging underwater environments.

• •

  Comparative evaluation with baseline architectures.

The rest of the paper is organized as follows: Section II describes how CADDY dataset was created and organized for both gestures and poses. Section III provides the details of the deep transfer learning tree structure approach for classifying the diver gestures and poses. Section IV presents the data analysis setup, compares the performance with other existing techniques, and discusses the results. Finally, Section V presents the conclusions and plan for future work.

Hand gestures in the CADDY dataset use the CADDIAN gesture-based language[38]. The diver signals a specific hand gesture or performs a sequence of gestures to command the AUV in real-world underwater missions. In order to aid the classifier in distinguishing between different hand gestures, the divers wear gloves with special features that have a different color strip on each finger, a white square on the back of the hand, and a white circle on the front of the hand. Without these features on the gloves, it is not only difficult to distinguish between different hand gestures but also difficult to differentiate between the diver’s body and his hands in dim surroundings even with human eyes.

Table I shows a total of 16 different gestures and the associated commands, such as start, up, down, and take a photo. Overall, the CADDY dataset contains $9239$ stereo pairs, i.e., $9239\times 2=18,478$ images of the diver gestures.

Pose images were extracted from video sequences showing the diver’s movement. The purpose of classifying diver’s pose is to track diver’s movement and position such that the diver is always facing the camera on the AUV. Table II shows $3$ different poses: i) turning horizontally with chest pointing downward to the floor, ii) turning clockwise or counterclockwise vertically, and iii) free swim. Overall, the dataset contains $12,708\times 2=25,416$ images of the diver poses.

In this paper, DARE is trained using both gesture and pose datasets, thus it is capable of recognizing $20$ different classes consisting of $16$ gestures, $3$ poses, and $1$ true negative (or null) class. The null class contains $7190$ stereo pair which include miscellaneous situations like diver idle, diver missing, gestures missing, improper gestures or the transition between two gestures. Thus, there are a total of $(9239+12,708+7190)\times 2=58,274$ images which are used for training and validation of DARE.

DARE is built upon the deep transfer learning approach utilizing CNNs in a bi-channel configuration (i.e., one for each image of the stereo pair of the AUV camera).

Multi-channel CNN approaches have been applied in other domains that have multiple high-dimensional input data streams for robust and accurate classification [39, 40]. In multi-channel frameworks, data from each input stream is processed independently before fusion and classification. In the context of stereo camera system in the CADDY dataset, we employ a bi-channel approach to enable feature extraction from two different perspectives, which: i) implicitly capture the distance of the diver from the AUV, and ii) allow for better filtering of noise, biases, and other uncertainties by comparing correlated details between stereo images. Thus, a bi-channel CNN approach enables DARE to take full advantage of stereo images and provide robust classification.

The training of deep CNNs requires a large dataset (on the order of tens of thousands of images per class) in order to adequately train the weights of the network without overfitting. However, the collection of such large amounts of data is often cost-prohibitive and requires expensive computational resources to process. In order to overcome these limitations, we employ the use of pre-trained deep transfer learning models[41]. These models were trained on large-scale and diverse benchmark datasets, and their structure can be transferred to solve a different, smaller, but also relevant problem. In a transfer learning approach, the weights of the pre-trained convolutional layers are directly utilized to extract the features of the input images, and only the fully-connected neural network layers are retrained. This approach therefore only requires minimal data for re-training and thus shorter computation times, while still yielding high classification accuracy.

In this paper, DARE is constructed using three basic pre-trained transfer learning models: ResNet18 [42], AlexNet [43], and VggNet16 [44]. The architectures of these models are summarized in Table III. In general, these models are composed of an input layer, a series of convolutional stages that extract abstract and relevant features from the images, and finally a fully connected neural network that fuses the features and makes a classification decision. The input images from the camera are down-sampled to match the input size requirements of the considered transfer learning model.

Both channels are identical to each other and consist of the same stages as the underlying transfer learning model. These stages typically consist of a convolutional layer with a specified filter size that extracts the features, an activation layer that augments the features via nonlinear transformations to help increase separation, and finally a pooling layer that downsamples the features to ensure they are compact to maintain generality while reducing computation times. An example of the feature extraction process from the AlexNet is shown in Figure 4. The final features of each channel are then flattened and concatenated before being sent to the tree topology classifier for robust information fusion and decision making. For the sake of completeness, a brief overview of the most common CNN layers in each stage of a deep transfer learning model [45] is presented below.

DARE consists of hierarchical arranged tree-structured fully-connected neural network classifiers which are custom trained for systematically identifying individual classes, as shown in Figure 3. Since CADDY dataset has a large number of classes, a single neural network classifier may not yield satisfactory individual class recognition performances although it may still provide good overall average performance. The classification tree is constructed as follows. Each node of the tree represents a neural network classifier which is trained to separate certain groups of classes. The branches correspond to the outputs of the classifiers and the final decisions of individual classes are predicted at the leaf nodes.

In the classification tree, the class groups separated at each node are formed based on similar categories (e.g., number of hands, hand orientation, number of fingers, etc.) The root node at level $0$, called RootNet, is trained to classify the three major categories: gestures, poses and true negatives (i.e., images with no, improper or wrong gestures). Subsequently, the two classifiers at level $1$, GNet10 and PNet11, further classify the gestures and poses identified from RootNet, respectively. It is noted that the back and front faces of the hands with gloves have different features with the white square shape on the back and white circle on the front. Moreover, the gestures are made with either one or two hands. Thus, GNet10 is constructed as a quinary classifier to classify the following five gesture types: one hand (back face), one hand (front face), two hands closed (back face), two hands open (back face), two hands open (front face). On the other hand, PNet11 is constructed as a ternary classifier to separate the three different diver poses: turning horizontally, turning vertically and free swim. Next, at level $2$, the one hand gestures are further classified using the finger/thumb pointing direction and the number of fingers. Thus, GNet20 is constructed as a ternary classifier to classify the one hand (back face) gestures into three categories: thumb pointing up/no finger, finger pointing down/front, and finger(s) pointing up. Similarly, GNet21 is constructed as a quaternary classifier to classify the one hand (front face) gestures into four categories: thumb pointing down, four/five fingers, three fingers/two fingers and a thumb, and one/two fingers.

Finally, at level $3$, six binary classifiers are constructed: GNet30, GNet31, GNet32, GNet33, GNet34 and GNet35, which classify no finger vs thumb; finger pointing downward vs finger pointing forward; two fingers pointing up vs one finger pointing up; four vs five fingers pointing up; three fingers vs two fingers and a thumb pointing up; and two fingers vs one finger pointing up, respectively.

As previously mentioned, each node of the tree is a fully-connected feed-forward neural network classifier. The input layer to each of these classifiers is the Multi-FM. The architecture of the hidden layers of each of these classifiers is the same as the fully-connected hidden layers of the pre-trained transfer learning model that the multi-channel feature extraction stages are based on. For example, AlexNet has two identical fully-connected hidden layers. Each hidden layer has $4096$ neurons that are connected to each input from the previous layer, a ReLU activation layer, and a $50\%$ dropout layer. Thus, if the convolution layers of AlexNet are used for multi-channel feature extraction, then these hidden layers are used in each node classifier of the tree. These hidden layers are then followed by a fully-connected output layer that has the same number of neurons as number of classes or class groups separated at that node, and a softmax layer to calculate the probability of a sample belonging to a class. The weights of the fully-connected network are retrained using the cross-entropy loss function at the output.

The tree-structured classifier simplifies the 20-class underwater diver action recognition problem by constructing several focused neural network classifiers arranged in a hierarchical manner. It not only boosts the individual class recognition performance but also improves the overall performance, while producing a robust and reliable decision in real-time.

In this paper, we used several measures to evaluate the performance of DARE against other existing deep learning architectures. For each individual diver action, we compute the micro F1 score and balanced individual class accuracy. Then, we compute the macro F1 score and the overall correct classification rate (CCR). Each of these measures are defined in terms of the number of true positives, false negatives, true negatives, and false positives for each diver action.

For each class $i\in\{1,\dots,n\}$, where $n$ is the number of classes, let $TP_{i}$ be the number of true positives, which is the number of samples of class $i$ that are correctly classified. Let $FN_{i}$ be the number of false negatives, which is the number of samples of class $i$ that are misclassified. It should be noted that the total number of samples belonging to class $i$ is $TP_{i}+FN_{i}$. Also let $TN_{i}$ be the number of true negatives, which is the number of samples belonging to any class $j\neq i$ that are not classified as class $i$. Finally, let $FP_{i}$ be the number of false positives, which is the number of samples belonging to any class $j\neq i$ that are misclassified as class $i$.

F1 Score: The F1 Score is the harmonic mean of precision and recall. A high precision for diver action $i$ suggests that there are few false positives relative to true positives (i.e., other diver actions are not misclassified as action $i$), whereas a high recall suggests that there are few false negatives relative to true positives (i.e., diver action $i$ is not misclassified as another action). The precision and recall for class $i$ are computed as:

Then, the individual class (micro) F1 score is computed as:

and the average (macro) F1 score is computed as:

Balanced Individual Class Accuracy: The balanced individual class accuracy (BACC) is the mean of the true positive rate and true negative rate. This measure is useful in inherently unbalanced one-verses-the-rest class performance comparisons, as other accuracy measures do not normalize the true positive and true negative predictions. The individual class true positive rate $TPR_{i}$ is the same as the recall in Eq. (5), and the individual class true negative rate is computed as:

Then, the balanced individual class accuracy is computed as:

Correct Classification Rate: The CCR is the overall accuracy of the classifier, which is computed as: