Free-Form Composition Networks for Egocentric Action Recognition

Compared with third-person views, egocentric action recognition relies on modeling the fine-grained hand-object interactions. Early methods mainly explore motion cues to distinguish egocentric actions. In particular, the optical flow-based global [20] and local [21] descriptors are designed to classify a variety of interactive actions. The motion descriptors are further combined with CNN-based appearance [22, 23] and depth [24] information to improve the action representations. Noticing that the attention mechanism is efficient to locate the region of interest from cluttered background, the gaze estimation [25, 26] is utilized to direct deep learning models to concentrate on the informative region in hand-object interaction videos. LSTA [27] extends traditional LSTM with the spatial attention based neural unit in order to concentrate on the action-related portion of the video sequence. In addition, the visual and audio features are fused by multi-scale temporal-binding [28] to obtain complementary information from egocentric actions.

According to the structure of egocentric actions, recent methods utilize two independent branches to separately recognize the associated verb and noun. The twin stream network [3] utilizes the hand segmentation to locate the action-related object, and then classifies the verb and noun by CNNs. To exploit the correlation between the verb and noun branches, the symbiotic attention [7, 8] uses the spatial locations and discriminative object features to focus on the occurring interactions. Based on the multi-task framework, H+O [5] estimates the hand and object poses in the 3D space from 2D RGB images, and then models the hand-object relationships with LSTM to recognize the action class and the action-related object. Several datasets [2, 6] also provide annotated 3D hand poses and 6D object poses to facilitate the study of egocentric actions. To avoid labor-intensive annotations, IPL [4] leverages the motion cues to learn the verb class, and then guide the noun classification by extracting the action-related object. Compared with the rigid combination of a verb and a noun, our method is able to simultaneously extract the features of multiple verbs, prepositions, and nouns from an egocentric action video, and then reassemble these features to generate new training data for the rare and unseen actions.

Compositional learning has attracted increasing attention by synthesizing new samples based on the representations of partial images [29, 30, 31]. Compositional models are also utilized to ameliorate the detection of partly occluded objects with non-occluded portions [32, 33]. For visual question answering [34, 35], the questions are first decomposed into syntactic sub-modules, which are then regrouped by question-related compositional networks.

Currently, a variety of compositional models are proposed to focus on the data scarcity issue in human-object interaction (HOI) detection. External lexical knowledge [36] is embedded into graph convolutional networks to generate new action-object pairs for zero-shot HOI. The target objects in images are also replaced with similar intra-class instances in other images to improve the diversity of training data [37]. IDN [38] and VCL [39] extract shared action and object features from HOI images via the transformation function in the multi-branch network, and then regroup these action and object features to compose novel HOI samples in the feature space. In order to detect rare and unseen categories in the open world, ATL [40] and FCL [41] combine verb representations and novel object representations to generate large-scale HOI samples. ICompass [42] and GCA [43] further explore multiple verb-noun pairs from a single image via cross-attention mechanism and generative adversarial networks. Compared with existing HOI models decomposing/composing still images with the rigid template of a verb and a noun, we devise a flexible form to decompose/compose egocentric action videos with multiple verbs, prepositions, and nouns.

Graphical models are widely used to reason the long range relationships in multiple computer vision tasks, including semantic segmentation [44, 45], scene graph generation [43, 46], and group activity recognition [47]. By performing convolutional operations on the graph-structured data, graph convolutional networks (GCNs) [48] are adopted to learn the temporal evolution of human body skeletons. ST-GCN [49] adopts the spatial and temporal convolutions to model the evolution of human poses. 2s-AGCN [50] increases the flexibility of graph models and learn both the static and dynamic information from skeleton sequence. Besides skeleton data, GCNs are also employed to model the relationships between object proposals extracted from the action video. STRG [51] constructs the space-time region graph with object bounding boxes to reason the human-object and object-object relationships. LSTR [52] takes 3D tubelets as graph nodes, and models the long-term temporal dynamics via GCNs to localize human actions.

Compared with GCNs operating on a graph with its edge fixed as a similarity scalar, graph neural networks (GNNs) updates the node and edge features alternatively by message passing procedure. dNRI [53] predicts the states of the human body joints and their relations at every point in time based on GNNs and all the previous states. Similarly,  [54] makes use of historical pose tracklets to predict the human poses in the following frame, which serves as a robust estimation in challenging scenarios. In addition to the human pose estimation, GNN-based models are combined with the pose-graph optimization to deal with the problem of camera pose estimation [55]. Different from above graphical models operating on the complete graph, MUSLE [56] explores compact sub-graphs to capture discriminative patterns of human actions for classification. In this paper, we adopt the edge features in GNNs to characterize the verbs and prepositions, and then distinguish different verbs/prepositions by learning from corresponding sub-graphs.

We present an overview of the primary framework of the Free-Form Composition Networks (FFCN) in Figure 2. This framework is designed to combine semantic elements from various videos to generate new instances corresponding to specific action classes. To accomplish this, we start by considering each video frame, which contains $N$ instances, including hands and objects, and establish a graph with $N$ nodes to represent these instances (the red and yellow nodes separately represent hands and objects). Subsequently, we use Graph Neural Networks (GNNs) to concurrently enhance the features of both nodes and edges in the graph. The relationships between verbs, prepositions, hands, and objects are closely intertwined within the edge features of the graph, which capture substantial relational information among different nodes. We then partition the graph into two sub-graphs, namely, a verb graph and a preposition graph, to extract the representations of verbs and prepositions from the edge features. Recognizing that action descriptions may not always comprise just one verb and one preposition, we introduce temporal and spatial decomposition modules to distinguish multiple verb and preposition features from a sequence of verb and preposition graphs.

The detailed procedure to extract different verb representations is shown in Figure 3. Since the number of verbs varies between different action classes, we consistently extract ${n_{v}}$ verb representations from each video, where ${n_{v}}$ indicates the largest number of verbs appearing in an action description. Similarly, the preposition representations are extracted from the sequence of preposition graphs. To extract the object representations corresponding to the nouns in the action description, we employ vanilla CNNs. With the above-mentioned shared verb/preposition/noun representations, we then integrate those verb, preposition, and noun representations across different classes to generate new training samples in the composition branch, with a focus on the rare classes. That is, the verb, preposition, and noun representations are concatenated in terms of their order appearing in the action description.

It is important to note that the quantity of semantic components (verbs, prepositions, and nouns) varies significantly across different action classes. For the rare classes, which may contain ${n_{v}}$ verbs, ${n_{p}}$ prepositions, and ${n_{n}}$ nouns, we take the $d$-dimensional features corresponding to verbs, prepositions, and nouns and individually reduce their dimensions to $d/{n_{v}}$, $d/{n_{p}}$, and $d/{n_{n}}$ using a multilayer perceptron (MLP). After this dimension reduction, these features are then concatenated based on their order of appearance in the action description, resulting in the creation of new samples with a dimension of $3d$. This ensures that the features of all generated training samples are normalized to the same dimension.

In this subsection, we define the process of generating a graph as follows: Consider a video comprising $T$ frames, each containing $N$ instances. We represent all these instances’ feature information as $X=(x_{1}^{1},\cdots,x_{N}^{1},x_{1}^{2},\cdots,x_{N}^{2},\cdots,x_{1}^{T},\cdots,x_{N}^{T})$, where $x_{i}^{t}$ represents the feature representation of the $i$-th instance in frame $t$. These instance representations can take various forms, such as 1) bounding box coordinates that include the center position, height, and width of each bounding box, and 2) appearance data obtained from the backbone CNNs. In this study, we choose to use the bounding box coordinates as the instance features for extracting verb and preposition characteristics. This choice is motivated by two reasons: firstly, these coordinates complement the widely adopted RGB-based methods, and secondly, verbs and prepositions have a closer association with the spatial positions of hands and objects than with RGB information. For example, two action videos of “pour juice” and “pour water” contain the same verb “pour”, but they may have different appearance features. The zero vectors are adopted as the features when there are fewer instances detected in each video frame. Therefore, we have the ability to build a graph denoted as $G=(V,E)$ where the graph nodes and edges symbolize the identified instances and their interconnections. To enhance this graph, we employ a graph neural network (GNN). In the process of message passing at step $s$, we represent the node features as $h_{i,t}^{s}$ and the edge features as $f_{ij,t}^{s}$. The architecture of the GNN, which concurrently improves both the graph nodes and edges, is structured as follows:

Here, the notation $[\cdot,\cdot]$ signifies the concatenation operation, while $h$ and $f$ correspond to the intermediate hidden states of the graph node and edge, respectively. Additionally, $\phi$ stands for the multilayer perceptrons (MLPs).

We utilize the output of the GNNs, which are the graph edge features, to gain insights into the interactive relationships among various instances. Recognizing that the verb and preposition characteristics can be respectively attributed to the hand-object and object-object relationships within egocentric videos, we proceed to break down the refined graph model into two distinct components: the verb graph and the preposition graph. This allows us to independently capture the features related to the verb and preposition. Within the verb graph, we establish connections between the hand node and all the object nodes to extract information regarding hand-object relationships. In contrast, in the preposition graph, we interconnect the object nodes themselves to capture the interactions among objects. This approach differs from most existing graph-based models, which predominantly focus on exploring the features of graph nodes. In our case, we leverage the robust edge features to extract semantic components, namely the verb and preposition features.

In this subsection, we formulate the temporal decomposition as follows. Given the fact that the action description usually consists of multiple verbs and prepositions, which correspond to different segments in the video. Meanwhile, the dilated convolution is usually adopted to aggregate multi-scale contextual information, and increase the receptive field by stacking multiple layers. Therefore, we propose to explore dilated convolutions on the verb/preposition graph edges in the time dimension to extract the features of different verbs/prepositions from the corresponding video segments.

Given ${N_{e}}$ edges in the verb graph, the features of the $k$-th edge across $T$ frames are denoted as ${F_{k}}=({f_{k1}},{f_{k2}},\cdots,{f_{kT}})$, which are fed into several layers of dilated 1D convolutions. In this paper, we use four layers of dilated convolutions with dilation factors $d=1,2,3,4$ at each layer. All these layers have the same number of convolutional filters with the kernel size 3. Each layer utilizes a dilated convolution with the ReLU activation function, and the output is refined by the next layer. Inspired by MS-TCN [57], we also employ a residual connection [58] and the operations at each layer are formulated as follows:

where ${H_{l}}$ indicates the output of the $l$-th layer, $*$ is the convolution operator, ${W_{1}}\in{R^{3\times D\times D}}$ are the weights of the dilated convolution with $D$ filters and kernel size 3, ${W_{2}}\in{R^{1\times D\times D}}$ are the weights of the $1\times 1$ convolution, and ${b_{1}},{b_{2}}\in{R^{D}}$ indicate the bias. The optimized graph edge features can be obtained from the last dilated convolution layer and the features of the $k$-th edge across $T$ frames are then aggregated as follows:

where the function ${\phi_{verb}}$ indicates the MLP networks. All the edges in the verb graph share the same dilated temporal convolution layers, in order to extract the verb feature from the graph edges belonging to the same video segment. This way, all the verb graphs across $T$ frames are aggregated along the time dimension into a single graph model, where the edge features are denoted as ${Q_{v}}=[{q_{1}},{q_{2}},\cdots,{q_{{N_{e}}}}]$. Similarly, the preposition graphs across $T$ frames are temporally aggregated into a single graph with the edge features ${Q_{p}}$.

In this subsection, we formulate the spatial decomposition as follows. Though the verb and preposition graphs are utilized to explore the verb and preposition features based on the temporal evolution of hand-object and object-object relationships, the verb and preposition graphs usually contain irrelevant objects due to the complex background, which may degrades the reasoning from spatial-temporal relationships. Instead of using a simple averaging pooling operation to aggregate the graph edge features, we propose to adaptively learn from the relationships between action-related instances to extract the verb and preposition features.

For the verb features, we apply the graph convolutional networks (GCNs) to encode the interactive relations between different instances as follows:

where $A$ represents the adjacency matrix with ${N_{e}}\times{N_{e}}$ dimensions, ${Q_{v}}$ indicates the edge features in the verb graph with $D\times{N_{e}}$ dimensions, and $W$ is the weight matrix with the dimension $D\times D$. Following the same strategy in [50], the elements $\theta$ in the adjacency matrix $A$ are trainable parameters in the networks and updated in the training process. Therefore, the output of each graph convolution layer $Z$ is in ${N_{e}}\times D$ dimensions, and multiple convolution layers can be stacked together. After the message passing within the verb graph, the final outputs of the GCNs then indicate the optimized relationships between different instances, which are aggregated by average pooling to obtain the verb feature. Notably, we always adopt a residual connection to facilitate the gradient flow. We can also extract the features of all the verbs and prepositions appearing in the action description in the same way. In addition, we adopt vanilla CNNs to extract the object features in each video frame, and then fuse the features of the same object across all the frames by MLPs to represent the nouns.

In this subsection, we outline the process of creating new action samples as follows: We start with verb, preposition, and noun features, and from these, we generate features for new action samples, which are aimed at enriching the training data for the less common classes in the composition branch. To construct these samples for rare actions, we follow the sequence of verbs, prepositions, and nouns as they appear in the infrequent action descriptions. We concatenate the corresponding features and employ MLPs to reduce the dimension of these features, effectively representing the rare action samples within the feature space. These generated samples are then used for training in conjunction with the original training samples. It’s worth noting that both the composition and decomposition branches share the same weights in the fully connected and classifier layers. Finally, we simultaneously minimize two loss functions: ${L_{d}}$ for the decomposition branch and ${L_{c}}$ for the composition branch, expressed as $L={L_{d}}+\lambda\cdot{L_{c}}$, where $\lambda\geq 0$ serves as a hyper-parameter to balance the two distinct branches.

• •

  Something-Something V2 (SS-V2) [59] is a large-scale hand-object interaction dataset, which contains 174 action categories created in a crowdsourcing manner. The videos in the same class are captured by performing the same action with various objects, in order to make the methods identify human actions regardless of the interactive objects. It provides the action descriptions mainly consisting of multiple verbs and prepositions as well as video annotations, including bounding boxes and their identities. Overall, it includes 168,913 training videos and 24,777 validation videos. The number of videos in different action classes follows a long-tailed distribution ranging from 115 to 4,081.

• •

  H2O [2] is a recently introduced egocentric interaction dataset containing 36 action classes of two hands manipulating objects, which are captured by four participants in three different environments. The action labels are verb-noun pairs composed by 11 verb classes and 8 noun classes. This dataset includes 571,645 RGB-D frames with 344,645 training frames, 73,380 validation frames, and 153,620 testing frames. It also provides detailed annotations, containing hand and object poses separately in 3D and 6D space, object meshes, camera parameters, and point cloud data. We focus on the action recognition task, and provide the top-1 accuracy on the validation videos.

• •

  EPIC-KITCHENS-100 (KITCHENS) [1] is a large-scale egocentric video dataset, which includes 100 hours of daily activity videos captured in kitchens. The performed activities are unscripted and close to real-world data, which makes the number of videos unbalanced between different classes. All the videos are split into train/validation/test with a ratio of 75/10/15. In total, there are 89,977 video clips of fine-grained interactions annotated with 97 verb classes and 300 noun classes. It also provides annotations of hands and objects appearing in the video. Following the standard evaluation strategy, we separately predict the verb and noun classes of each video, and then obtain the action class by fusing the verb and noun classification results. We provide the top-1 accuracy on the validation videos.

We implement the proposed free-form composition networks (FFCN) based on PyTorch [60], and sample 16 video frames from each clip. We adopt SGD [61] with the momentum value of 0.9 and the weight decay value of 1e-4.

On the Something-Something V2 dataset, we respectively report the settings in the standard, object-independent, and few-shot classification tasks. In the standard and object-independent protocols, each mini-batch contains 64 samples. The basic learning rate is set to 0.05, and divided by a factor of 10 after 15 epochs. A total of 25 epochs are utilized to train the networks. We generate new training data for 30 tail classes, and compose 10 samples in each mini-batch. In the few-shot protocol, each mini-batch contains 16 samples and the learning rate is set to 0.05 with a total of 8 epochs. We compose new training data for all the classes, and the number of composed samples is equal to original training samples in each class. We set the weight $\lambda{\rm{=}}0.1$ to balance two branches in the final loss function.

On the H2O dataset, each mini-batch contains 12 action samples. The basic learning rate is set to 0.005, and decreased by 0.1 at the 20-th epoch with a total of 80 epochs. We generate new training data for each action class, and compose 10 action samples in each mini-batch. We utilize the 3D hand poses and 6D object poses provided by this dataset as the features of each instance to learn the verb features, and employ the VGG-16 [62] to learn the noun features from the region of the objects. We set the weight $\lambda{\rm{=}}0.2$ to balance the decomposition and composition branches.

On the EPIC-KITCHENS-100 dataset, each mini-batch contains 256 videos. The basic learning rate is set to 0.001, and divided by a factor of 10 at the 20-th and 40-th epochs. A total of 50 epochs are utilized to train the networks. We compose new samples for the tail classes with fewer than 100 training samples, and generate 20 samples in each mini-batch. We use the bounding box positions of hands and objects to learn the verb features, and adopt the VGG-16 [62] to extract the noun features. We set the weight $\lambda{\rm{=}}0.1$ in the final loss function.

Initially, we evaluate our proposed method using the standard approach, which involves a dataset consisting of 168,913 training videos and 24,777 validation videos. The performance comparison against state-of-the-art methods is presented in Table I. To ensure a fair assessment, we separately compare our Free-Form Composition Network (FFCN) with two different categories of existing methods that employ bounding box locations and video sequences as input to the networks. Both FFCN and STIN [18] utilize bounding box locations as input and can be integrated with RGB-based methods to enhance accuracy. In this scenario, FFCN outperforms STIN in both settings. Additionally, I3D [63] is a widely used RGB-based method for video action recognition. When combined with the same method, FFCN+I3D ($67.3\%$) surpasses STIN+I3D ($60.2\%$) by $7.1\%$. As TDN achieves better results than I3D, we incorporate FFCN with TDN, resulting in an accuracy of $74.8\%$, which is $7.8\%$ higher than TDN and notably superior to recently proposed alternatives.

Following the evaluation protocol outlined in [18], we also assess the performance of our FFCN in an object-independent scenario. In this setup, the training set consists of combinations of nouns and verbs that do not appear in the testing set. The training and validation sets contain 54,919 and 57,876 videos, respectively, encompassing a total of 174 action classes. As demonstrated in Table II, FFCN ($54.3\%$) outperforms STIN ($51.3\%$) by exclusively utilizing bounding box locations of hands and objects to capture the features of verbs and prepositions present in the action descriptions. Most existing methods rely on whole video frames as input to distinguish between different action classes, partially relying on scene appearance information rather than the motion characteristics between hands and objects. The proposed FFCN can be easily combined with RGB-based methods through simple late fusion, resulting in an improved performance. By fusing FFCN ($54.3\%$) with I3D ($51.9\%$), the final accuracy reaches $66.9\%$. When combined with TDN ($62.5\%$), we achieve the highest accuracy of all, at $70.5\%$.

To further validate the effectiveness of composing new training data, we test the performance of FFCN in the task of few-shot action recognition. We also utilize the protocol in [18] to separate the training data into the base and novel sets. Our networks are first trained on the 88 base classes, and then finetuned on the other 86 novel classes with 5 or 10 training videos in each class. As illustrated in Table II, we separately compare the accuracies of 5-shot and 10-shot action recognition on the 86 novel classes. With only 5 training samples in each action class, our FFCN ($30.9\%$) exceeds STIN ($25.8\%$) by $5.1\%$ based on the locations of hands and objects. Combined with RGB-based models, FFCN+I3D ($38.7\%$) outperforms I3D ($23.5\%$) by $15.2\%$, and FFCN+TDN ($43.1\%$) outperforms TDN ($30.8\%$) by $12.3\%$. In the 10-shot action recognition, our FFCN also improves the performances of RGB-based models by more than $15\%$. The above results in the few-shot learning validate that our FFCN can augment the training data to obviously improve the accuracies in egocentric action recognition.

We also test our FFCN on the H2O dataset, and the comparison results are illustrated in Table III. RGB-based methods are widely used to recognize hand-object interactions, such as I3D, SlowFast, and X-ViT. Moreover, TAR achieves excellent result ($90.9\%$) in ECCV 2022 challenge on action recognition from egocentric camera. HTT adopts the locations of 3D hand joints and the RGB features of objects together to obtain a promising result ($86.4\%$). The accuracy of our FFCN is $93.3\%$, which outperforms other methods by a clear margin with only location information. Because the location- and RGB-based methods are complementary in representing hand-object interactions, we combine our FFCN ($93.3\%$) with recently proposed X-ViT ($82.8\%$) to further increase the accuracy to $95.0\%$. We also fuse the results of FFCN and I3D, and the accuracy is improved to $95.9\%$, which is the best performance on the H2O dataset.

Different from the above two datasets designed to directly predict the action labels, the EPIC-KITCHENS-100 dataset expects the methods to separately predict the verb, noun, and action labels of each video. Our FFCN mainly utilizes the coordinates of hands and objects to extract the motion features in the video without the appearance information, so we separately report the prediction accuracies of the verb and action in FFCN and the accuracies of the verb, noun, and action by integrating FFCN with a RGB-based method. The comparison results are illustrated in Table IV. Because of the promising performance, SlowFast and TSM are both widely used to recognize egocentric actions. In first person videos, the active objects are usually surrounded by distracting objects, so IPL and SOS present two kinds of methods to capture the action-related objects in the video. Based on the transformer architecture, X-ViT exploits the spatial-temporal attention in action videos, and OMNIVORE jointly trains a single model from multiple modalities, such as images, videos, and 3D data, to achieve outstanding results in different classification tasks. All aforementioned methods are RGB-based models, and extract the video features from the appearance information. Our FFCN utilizes the locations of hands and objects to predict the verb and action labels, and the mean accuracies are separately $56.1\%$ and $52.2\%$. Integrated with X-ViT, the verb and action accuracies are increased to $71.5\%$ and $60.5\%$. Our FFCN is also fused with OMNIVORE, and the final verb and action prediction accuracies are $73.6\%$ and $69.1\%$, which are separately $4.1\%$ and $19.2\%$ higher than OMNIVORE.

The proposed free-form composition networks mainly consist of two branches, i.e., a decomposition branch and a composition branch. To demonstrate the effectiveness of two branches for egocentric action recognition, we separately report the results of the decomposition branch and two branches together. The accuracies corresponding to two kinds of settings are shown in Table V. On the something-something V2 dataset, the mean accuracy is $51.3\%$ based on the decomposition branch, and the result is increased to $52.9\%$ through generating new training data for the tail actions with the composition branch. We can see that some tail actions in this dataset can not be composed by the videos of other action classes. For instance, the verb “cover” only appears in the action “covering something with something”, so we can not extract its visual feature from other actions. We thus gather 50 action classes (i.e., SS-V2-Subset) with shared verbs and prepositions/adverbs from something-something V2 to evaluate the effectiveness of two branches. The number of training samples in each action class is illustrated in Figure 4, where the class label is consistent with original dataset. Based on the decomposition branch, the mean accuracy is $55.1\%$ using only the locations of hands and objects. Then, the accuracy is improved to $64.7\%$ by generating the features of new training samples for 20 action classes with the fewest training data. H2O is a relatively small dataset, and the number of videos for each class in the training set ranges from 12 to 25. We thus generate new training samples for all the action classes. After applying data augmentation, the mean accuracy is increased from $91.7\%$ with only the decomposition branch to $93.3\%$ with two branches together. The EPIC-KITCHENS-100 dataset expects the methods to separately predict the verb, noun, and action labels. It is worth noting that the two-branch architecture is designed to directly predict the action label. To test the effectiveness of the compositional branch, we only compare the action prediction accuracies whether or not generating new training data for the tail actions. The average accuracy is increased from $50.5\%$ to $52.2\%$ through data augmentation.

We also compare the performances of each action class whether or not composing new training samples. The comparison results on the SS-V2-Subset are shown in Figure 5. The class label and the number of training samples in each class are both consistent with Figure 4. With only the decomposition branch, none of the validation samples is correctly recognized in some tail classes, such as 136, 60, and 137. After composing new samples for 20 tail classes, the accuracies of 45 out of 50 actions are improved, and the result of action class 15 is increased from $50.0\%$ to $83.5\%$. We further compare the results of the decomposition and composition branches on the H2O dataset. As shown in Figure 6, the decomposition branch of our FFCN obtains promising results, and the accuracies of most classes are $100\%$. After composing new samples with two branches, the results of five classes are increased by a clear margin, and the accuracy of the action “close chips” is increased from 0 to $50.0\%$. But some actions in this dataset are still easily confused with each other. For example, the action “take out espresso” is usually misclassified into the actions of “take out cappuccino” and “put in espresso”.