Two-stream Multi-level Dynamic Point Transformer for Two-person Interaction Recognition

Most recent two-person interaction recognition approaches heavily rely on RGB-D data (Liu et al., 2019). These methods can be broadly classified into two major groups: skeleton-based interaction recognition and hybrid feature-based interaction recognition.

Skeleton-based methods are centered around utilizing skeleton data for interaction reasoning. This type of data contains 3D positions of body joints and is typically estimated from depth images. Various approaches within this category attempt to generate features based on each person’s joints and their interactive joints to represent the motion relation. For instance, in (Yun et al., 2012), an approach is proposed to use movements between all pairs of joints, joint distances, and velocity information as the motion information for interaction recognition. Another study, (Wu et al., 2018), extracts a human interaction feature descriptor encompassing the directional, static, and dynamic properties of the skeleton data. They subsequently employ a linear model with a sparse group lasso penalty enhancement to facilitate the interaction recognition task. Meanwhile, some researchers transform the interaction problem into a single-person action recognition problem. In (Bloom et al., 2016), a method is proposed to decompose a two-person interaction into two individuals’ actions, followed by separate classification of each person’s action. However, a significant challenge in using skeleton data is the robustness of the skeleton estimation, which is still not a trivial matter. Inaccurate and incomplete estimated data can adversely affect recognition performance, thereby posing a serious limitation to the effectiveness of skeleton-based methods.

Hybrid feature-based methods leverage combined features from different channels to recognize interactions. For instance, (Xia et al., 2015) integrates motion features, skeleton joints-based postures, and local appearance features from both depth and RGB data to reason about interactions. Similarly, (Trabelsi et al., 2017) combines the distance property of the 3D skeleton with dense optical features extracted from RGB and depth images jointly to predict the interaction class. On the other hand, (Gemeren et al., 2014) merges information from joints with poselets to select important frames and captures motion and appearance features for each interactive person from the bounding boxes where the human subjects are located. Although features from multimodal sources can provide valuable information for recognition, these methods often require substantial amounts of data, affecting the network’s efficiency. Additionally, many of these methods involve using RGB images as part of the input, which could expose excessive personal privacy information. Consequently, these methods may not be suitable for practical applications where some users prefer unobtrusive monitoring.

Point clouds contain abundant 3D geometrical information. PointNet (Qi et al., 2017a) is a groundbreaking work that employs deep learning techniques to address static point cloud problems, including object part segmentation, classification, and semantic scene segmentation. The primary concept behind this approach is to utilize a symmetric function constructed with shared-weight deep neural networks to extract information from the points. Building on this, PointNet++ (Qi et al., 2017b) is a subsequent work that captures spatial features from local partitions and then hierarchically merges them to form a frame-level global feature. The subsequent 3DGCN (Lin et al., 2020) and IGCN (Liu et al., 2022) improve the performance of the model by defining variable kernels and interpolating kernels for point cloud feature extraction, respectively.

Compared to static point cloud analysis, learning on dynamic point clouds presents more challenges. Recently, various point cloud-based approaches have been devised to address video-level human action recognition tasks. For instance, in the 3DV method (Wang et al., 2020), 3D points are initially converted into regular voxel sets to capture motion information. These voxel sets are then abstracted back to a point set, serving as the input to PointNet++ (Qi et al., 2017b) for action recognition. Similarly, SequentialPointNet (Li et al., 2021) flattens a point cloud video into a new data type named hyperpoint sequence, followed by feature mixing on the hyperpoint sequence to extract appearance and motion information. The advantage of using point clouds is that they can be directly derived from depth images, thereby mitigating the exposure of excessive texture and color information found in RGB images and effectively safeguarding privacy. Moreover, depth images aid in distinguishing between background and people, and employing advanced point cloud processing techniques can reduce computational costs and enhance efficiency. Taking inspiration from these methods, our paper proposes an effective point cloud-based network for two-person interaction recognition.

In video data processing, it is common to convert the video into a sequence of frame-by-frame images for further analysis. Despite the extensive research conducted in the area of human activity recognition (Sargano et al., 2017), the impact of frame rates has received relatively little attention until recently. (Harjanto et al., 2016) is the first work to focus on this aspect. According to their findings, a lower frame rate (with fewer frames) results in a shorter running time, but it sacrifices the amount of information available for feature extraction, which can compromise the discriminability of the extracted features. On the other hand, a higher frame rate captures more temporal information, but it comes at the cost of longer processing times and potential redundancy. This excess redundancy can negatively affect the networks by introducing distracting effects. As a result, employing a simple frame sampling method to evenly or consecutively select frames with either a lower or higher frame rate may not be optimal. The frame rate and the sampling method of the video have a significant impact on subsequent tasks, and it is essential to carefully consider and choose the appropriate frame rate and sampling technique for optimal performance.

The Transformer architecture (Vaswani et al., 2017) was initially developed to address language-related problems, such as text classification, machine translation, and question answering, by utilizing a self-attention mechanism to learn the relationships between elements in a sequence.

In recent years, the computer vision community has embraced transformer networks and adapted them for various vision-related tasks, achieving remarkable success. Various approaches have been created to tackle different computer vision problems, including image generation, object detection, segmentation, image recognition, video understanding, and visual question answering (Khan et al., 2022). For instance, (Dosovitskiy et al., 2021) introduces the Vision Transformer (ViT) for image classification. The method splits an image into multiple fixed-size patches and utilizes their embeddings along with positional information as input for a transformer. The transformer’s output is then used for classification. MViT (Fan et al., 2021a) introduces the concept of multiscale in ViT, constructing the model by gradually expanding the channel capacity and reducing spatio-temporal resolution. This approach addresses the pyramid structure through a multi-headed attention mechanism for recognition tasks in both images and videos. TimeSFormer (Bertasius et al., 2021) presents a convolution-free model proposing divided attention based on ViT. By extending ViT to video processing, TimeSFormer treats videos as sequences of patches extracted from frames. SequentialPointNet(Li et al., 2021) is a model for 3D human action recognition, processing point cloud sequences through two modules: the intra-frame appearance encoding module for spatial structure and the inter-frame motion encoding module for temporal changes. Transformer is used in the temporal position embedding layer to capture temporal features. P4Transformer (Fan et al., 2021b) employs a point 4D convolution to understand spatio-temporal structures and a transformer mechanism for grasping comprehensive appearance and motion details across videos, using self-attention on localized features instead of explicit tracking. Building on the success of these methods, we integrate a transformer into our network to enhance the information contained in global and partial features, thereby improving performance in two-person interaction recognition.

We introduce a novel frame sampling technique called Interval Frame Sampling (IFS) (refer to Figure 3). IFS comprises two steps, with the first step executed before the training phase to sample $p$ frames per interaction video from the original videos. To elaborate, given a point cloud video, we evenly separate all frames into $p$ intervals and then randomly select one frame from each interval. This step balances the advantages of uniform and random sampling to reduce redundancy while retaining highly discriminative information. We denote $p$ as the Top frame rate in our method, usually selecting a large value to ensure that the interaction samples contain ample information. During the training phase, the second step is performed in each epoch. In this step, we apply the same sampling method again to evenly divide the updated video from the first step into $q$ intervals. We then randomly select one frame from each interval, resulting in a total of $q$ frames as a training instance. Essentially, this means the frames representing the same interaction sample are likely to vary from epoch to epoch. We refer to $q$ as the Bottom frame rate in this work.

Our frame selection technique offers several advantages. Firstly, by setting the interval size in the second step appropriately (not too large compared to the total training epochs), our network can effectively access a significant portion of the information contained within the $p$ frames. As mentioned earlier, $p$ can be set as a large number to provide our network with ample information. Moreover, the two rounds of interval sampling within IFS help eliminate redundant information that may be present in consecutive interaction frames. This ensures that the network receives only essential and non-redundant data. Additionally, IFS contributes to shorter processing times, as our network only utilizes $q$ frames per interaction sample during training. Here, $q$ can be set as a much smaller value compared to $p$. Lastly, the characteristic of IFS introducing slight changes to the same interaction sample in each epoch’s training offers positive data augmentation-like effects. This variation enables the network to focus on learning discriminative partial information from different frames during the training phase. Overall, these advantages make IFS a valuable addition to our network, enhancing its efficiency and performance in two-person interaction recognition.

The frame features learning module outputs three types of features: local-region features, frame spatial features, and frame temporal features, which are obtained through a series of feature extraction steps (refer to Figure 4). This module takes a point cloud set ${x_{1},x_{2},\dots,x_{n}}$ as input, where each $x_{i}\in\mathbb{R}^{3}$ represents the 3-dimensional coordinates of a point within the point cloud frame.

Specifically, the spatial feature extraction component consists of two levels of set abstraction (SA) (Qi et al., 2017b). At each abstraction level, a sampling layer performs iterative Farthest Point Sampling (FPS) to select a fixed number of points as centroids from the input points. Subsequently, a grouping layer constructs local regions around these centroids using a ball query algorithm. Following that, a modified PointNet layer extracts local spatial features from each region. In the modified PointNet layer, the coordinates of all points in each local region are transformed into local coordinates relative to the centroid point. Additionally, the distance between each point and its corresponding centroid is included as an additional point feature to enhance the network’s performance in dealing with rotational motions. To further focus on learning crucial features, a Convolutional Block Attention Module (CBAM) (Woo et al., 2018) is applied to perform inter-feature attention on the point features. It is important to note that CBAM is not used in the first abstraction level within this work.

At the end of this layer, the feature vectors are combined with the coordinates of the corresponding centroid point, forming a spatial feature that represents a local region, which is referred to as local-region features. Subsequently, a Multi-Layer Perceptron (MLP) and a Max Pooling operation are applied to capture a spatial feature representation for the entire frame, which we call the frame spatial feature. The frame spatial feature extraction operation can be formulated as below:

In Equation 1, $FS_{t}$ is the frame spatial feature of the $t\text{-th}$ point cloud frame ($PC_{t}$). $e^{t}_{j}$ is the abstracted feature of the $j\text{-th}$ local region from the $t\text{-th}$ frame. SA is the set abstraction operation.

After extracting the frame spatial features, a frame temporal feature extraction structure is constructed to capture temporal clues for each frame. In this step, a temporal positional feature vector, which contains the temporal positional information for each frame, is generated using a sinusoidal positional encoding technique (Vaswani et al., 2017). Each dimension of the vector is calculated as follows:

In Equation 2 and  3, $t$ is the temporal position. $2j$ and $2j+1$ represent the dimension serial number. $m3$ is the size of a frame spatial feature. Each dimension of the sinusoidal positional vector corresponds to a sinusoid with a period of $10000^{2j/m3}\times 2\pi$. For a given input feature, this technique can be used to generate an absolute positional vector.

The generated temporal positional feature vectors have the same dimension as the frame spatial features; hence these two can be summed. After the summation, a MLP is applied to extract frame temporal features:

In Equation 4, $FT_{t}$, $FS_{t}$ and $TP_{t}$ are $t\text{-th}$ frame temporal feature, frame spatial feature, and temporal positional feature, respectively.

With frame-level features generated, we propose a two-stream module to aggregate them into global and partial representations (the whole architecture with the following transformer classification module can be seen in Figure 5).

We employ a transformer to perform self-attention on the aggregated global and partial features. This step aims to enhance each feature’s information by allowing them to learn important relationships from other related features. Consequently, the partial features can gain a global view, and the global feature can capture more fine-grained partial motion and appearance details.

The transformer applies a multi-head structure to improve its performance. It consists of $n$ heads and receives the integrated feature $S$ as the input. Specifically, within an arbitrary head $h_{i}$, the transformer uses three learnable matrices, $M_{ki}$, $M_{vi}$ and $M_{qi}$, to transform each sub-feature within $S$ into three vectors, representing a key, a value, and a query. All the keys, values, and queries are then packed together into matrices $K_{i}$, $V_{i}$, and $Q_{i}$, respectively:

After that, an attention weights matrix $A_{i}$ containing the relationships between each pair of sub-features within $S$ is computed by using an attention function. Next, an updated feature vector $U_{i}$ is generated by computing the cross product of the attention matrix $A_{i}$ and the values $V_{i}$:

The same calculation is performed in every head parallelly. $n$ different updated feature vectors are generated. Eventually, a final updated vector $U$ is computed by concatenating all of these n vectors:

The updated vector after the transformer has the same shape as the input feature $S$. It consists of one updated global feature and $ts$ updated partial features, representing the enriched information from the self-attention process. In addition to the multi-head structure, the transformer incorporates several other components, including residual connections, LayerNorms, and linear layers. Moreover, to enhance performance, $b$ identical transformer blocks are stacked together, enabling the model to capture more complex and higher-order relationships within the data. Following the self-attention process, a subsequent MLP layer compresses these transformed features into a single feature. Subsequently, another MLP layer utilizes this single feature, combined with the original global feature, as the input to predict the interaction class of the video. This residual-like structure contributes to making the model more stable and improves its ability to effectively learn and represent complex interaction patterns.

The overall process of our Two-stream Multi-level Dynamic Point Transformer is depicted in Algorithm 1. Steps 1 to 5 pertain to the pre-processing data stage, primarily involving the conversion of the depth video into a collection of point cloud frames, downsampling of the point cloud frames, and the execution of IFS Step I. From Step 6 onwards, the algorithm is executed in loops during the training process and only once during the testing process. Step 6 corresponds to IFS Step II. Steps 7 to 11 represent the frame features learning module as described in Section 3.2. Steps 12 to 25 depict our two-stream multi-level feature aggregation module, responsible for obtaining both global and partial features, as discussed in Section 3.3. Finally, the last step is the classifier that utilizes Transformer in Section 3.4, employed to obtain the final recognition results.

The Top and Bottom frame rates are set to 50 and 24, respectively, in the Interval Frame Sampling (IFS). That is saying 24 frames per video are used during training. We initially randomly sample 2048 points from each point cloud frame. Then, 512 points are further sampled from these 2048 points using a Farthest Point Sampling (FPS) method. Within the frame features learning module, FPS is applied again to select 128 centroids in the first set abstraction level and 32 centroids in the second level. In these two levels, the ball query radiuses are set to 0.06 and 0.1. In the partial stream of the two-stream feature aggregation module, a point cloud video (containing 24 frames) is split into 6 consecutive temporal segments with an overlap ratio of 0. There are 4 frames in each temporal segment from which a partial feature is generated. We stack 5 identical transformer blocks together within the following transformer, each with 18 heads. We train our network for 90 epochs in total. Adam (Kingma and Ba, 2015) is selected as the optimizer with a batch size of 32. We set the learning rate to start with 0.001 and decay with a rate of 0.5 every 10 epochs. A same data augmentation method from (Wang et al., 2020) is applied in this work.

We compare our network’s performance to the state-of-the-art methods on the two large-scale 3D two-person interaction datasets, the interaction subsets of NTU RGB+D 60 and NTU RGB+D 120. The comparison results can be seen in Table 1 and Table 2. Impressively, our network achieves remarkable results, consistently outperforming most other methods in the four standard evaluation settings of these two challenging datasets.

• •

  ST-LSTM (Liu et al., 2016): The authors propose Spatio-temporal LSTM with trust gates for 3D human action recognition.

• •

  GCA-LSTM (Liu et al., 2017): The authors propose Global Context-aware Attention LSTM Networks for 3D action recognition.

• •

  SPDNet (Huang and Gool, 2017): The authors construct a Riemannian network structure with Symmetric Positive Definite (SPD) matrix nonlinear learning in a deep model for visual classification tasks.

• •

  2S-GCA-LSTM (Liu et al., 2018): The authors propose a Two-stream Global Context-aware Attention LSTM network for human action recognition.

• •

  ST-GCN (Yan et al., 2018): The authors propose Spatio-temporal Graph Convolutional Networks for automatic learning of spatio-temporal patterns in data.

• •

  AS-GCN (Li et al., 2019): The authors introduce Actional Links in Graph Convolutional Networks to capture potential dependencies directly from actions, thus completing skeleton-based action recognition.

• •

  2S-AGCN (Shi et al., 2019): The authors propose Two-stream Adaptive Graph Convolutional Networks for skeleton-based action recognition. The topology of the graph can be learned end-to-end by the BP algorithm.

• •

  FSNET (Liu et al., 2020b): The authors focus on online action prediction for streaming 3D skeleton sequences. The authors introduce expanded convolutional networks to model the dynamics by sliding windows on the time axis.

• •

  ST-GCN-PAM (Yang et al., 2020): The authors propose a method based on Spatial-temporal Graph Convolutional Networks, which uses the pairwise adjacency matrix to capture the relationship of person-person skeletons.

• •

  MS-AGCN (Shi et al., 2020): The authors propose Multi-stream Attention-enhanced Adaptive Graph Convolutional Neural Network for skeleton action recognition. Graph topologies can be learned uniformly or individually based on data in an end-to-end manner, and this data-driven approach increases the flexibility of the model.

• •

  3DV-PointNet++ (Wang et al., 2020): The key to 3D Dynamic Voxel is to compactly encode the motion information in the depth video into a regular voxel set via a temporal rank pool, which is then transformed into point cloud data for processing.

• •

  LSTM-IRN (Perez et al., 2022): The authors propose an Interaction Relational Network that uses minimal prior knowledge about the structure of the human body to drive the network to identify the related body parts and use LSTM for inference.

• •

  SB-LSTM (Chiu et al., 2021): The Stacked Bidirectional LSTM uses two vectors to encode joint dynamics and spatial interaction information.

• •

  GeomNet (Nguyen, 2021): This is a method of two-person interaction recognition by using 3D skeleton sequences. Its key idea is to use Gaussian distributions to capture statistics on $R^{n}$ and those on the space of Symmetric Positive Definite (SPD) matrices.

• •

  SequentialPointNet (Li et al., 2021): The authors propose a frame-level parallel point cloud sequence network; specifically, the authors flatten the point cloud sequence into hyperpoint sequences and then use the proposed Hyperpoint-Mixer module to process the spatial and temporal features of the frames.

• •

  2s-AIGCN (Gao et al., 2022): The authors propose the Attention Interactive Graph Convolutional Network (AIGCN), which employs an Interactive Attention Encoding GCN (IAE-GCN) to extract interactive spatial structures and an Interactive-Attention Mask TCN (IAM-TCN) to discern temporal interactive features.

• •

  3s-EGCN-IIG (Ito et al., 2022): The authors propose a method for recognizing interactions between two individuals by employing factorized convolution and analyzing the distances between joints.

• •

  JointContrast (Zhang et al., 2023):The authors propose the Interaction Information Embedding Skeleton Graph Representation (IE-Graph) model, which utilizes unsupervised pre-training.

A comprehensive ablation study is performed to extensively test the effectiveness of various aspects of our network design. All tests are conducted on NTU RGB+D 60 under the cross-view evaluation setting.