ERA: Expert Retrieval and Assembly for Early Action Prediction

A full-length action sequence can be represented as a set $S=\{s_{t}\}_{t=1}^{T}$ containing $T$ frames, where $s_{t}$ denotes the frame at the $t$-th time step. Following previous works [15, 22, 30], $S$ is divided into $N$ independent segments, with each segment containing $\frac{T}{N}$ frames. A partial sequence consists of a set of frames $P=\{s_{t}\}_{t=1}^{\tau}$, with $\tau$ being the last frame in any one of the $N$ segments, i.e., $\tau=i\frac{T}{N},i=\{1,2,...,N\}$. The task of early action prediction is to predict the class $c\in\{1,2,...,C\}$ of the activity that the partial sequence $P$ belongs to, and different observation ratios $\frac{\tau}{T}$ of $P$ are tested.

As shown in Fig. 2, our ERA module consists of candidate experts contained within multiple Expert Banks and a non-expert block. Considering that convolutional architectures have been shown to be effective for the early action prediction task [30, 54], the experts are implemented as convolutional kernels. For ease of notation, we describe our method in a 2D convolutional kernel setting, even though it can be generalized to 1D, 3D or graph convolutions as well. This ability to generalize to other types of convolutions is important, as existing early action prediction architectures often use various types of convolutions, such as 3D convolutions [45] or graph + 2D convolutions [13].

Let an input be $X\in\mathbb{R}^{N_{in}\times N_{h}\times N_{w}}$, where $N_{in},N_{h}$ and $N_{w}$ represent the channel, height and width dimensions of the input feature map. Note that here we omit the batch dimension for simplicity. Assume that, in the backbone model, input $X$ is processed by a convolutional filter $W_{conv}\in\mathbb{R}^{N_{out}\times N_{in}\times b_{h}\times b_{w}}$, where $N_{out}$ represents the number of output channels, and $b_{h}$ and $b_{w}$ represent the height and width of the convolutional kernel. We aim to replace $W_{conv}$ with our ERA module, for better performance on early action prediction.

Specifically, we design our ERA module to also ultimately produce weights $W_{ERA}$ of the same shape ($N_{out}\times N_{in}\times b_{h}\times b_{w}$) as $W_{conv}$, which can be seen as $N_{out}$ kernels (each of shape $N_{in}\times b_{h}\times b_{w}$) that respectively produce each of the $N_{out}$ output channels. More specifically, in our ERA Module, we split the $N_{out}$ channels (and therefore also kernels) into two parts: $d$ expert channels and $N_{out}-d$ non-expert channels, where $d$ is a hyperparameter. To allow our $d$ expert channels to specialize in subtle cues, we would like each expert to be trained on only a subset of the data, thus we introduce $d$ Expert Banks containing $M$ candidate experts each, and retrieve only one expert from each Expert Bank per sample, such that the other $M-1$ candidate experts in the bank are unused for this sample. The $N_{out}-d$ non-expert kernels (collectively defined in a non-expert block $W_{nonexpert}\in\mathbb{R}^{(N_{out}-d)\times N_{in}\times b_{h}\times b_{w}}$) are shared over all samples, and thus tend to learn general patterns. We utilize a combination of both non-expert and expert kernels, because the usage of non-expert kernels to capture general patterns, is complementary with our experts that specialize at capturing subtle cues for discriminating between similar samples, and their combination leads to improvements on early action prediction.

We define the $i$-th expert in the $p$-th Expert Bank as $E_{i}^{p}$, where $E_{i}^{p}$ contains convolutional kernel weights $m_{i}^{p}$ and a key $k_{i}^{p}$. The key $k_{i}^{p}$ is used for matching with the most suitable samples, and represents the area of expertise of this expert, as it determines the samples that the expert will be retrieved for. Meanwhile, the expert kernel $m_{i}^{p}$ acts as a specialized mechanism to process the discriminative subtle cues on the input features that match the key $k_{i}^{p}$. The expert key $k_{i}^{p}$ and kernel $m_{i}^{p}$ are model parameters that are trained in an end-to-end manner. For each expert $E_{i}^{p}$, $m_{i}^{p}\in\mathbb{R}^{N_{in}\times b_{h}\times b_{w}}$ and $k_{i}^{p}\in\mathbb{R}^{K}$, where $K$ represents the dimensionality of the key, and $K<<N_{in}\times N_{h}\times N_{w}$ for efficiency.

This step transforms the feature map $X$ into a query (vector) $q^{p}$ of the dimensionality $K$.

Next, conditioned on $q^{p}$, we retrieve the most suitable expert from the $p$-th Expert Bank for discriminating subtle cues within the feature map $X$. Recall that each expert $E_{i}^{p}$ holds a key $k_{i}^{p}$ which represents its area of expertise, while $q^{p}$ is the representation of the feature map $X$. The degree of suitability of the expert $E_{i}^{p}$ on the feature map $X$ can thus be obtained by calculating the matching score between $k_{i}^{p}$ and $q^{p}$. We calculate the matching score $s_{i}^{p}$ for each expert $E_{i}^{p}$ using dot product between the query $q^{p}$ and the key $k_{i}^{p}$ as:

where $Argmax_{i}$ returns the index $i$ belonging to the largest element in the set $\{s_{i}^{p}\}_{i=1}^{M}$, and the returned index $I^{p}$ represents the index of the retrieved expert. We take the highest matching score ($s_{I^{p}}^{p}$) within the set, as it will come from the key $k_{I^{p}}^{p}$ representing an area of expertise that matches the query $q^{p}$ the best. Thus, the corresponding expert $E_{I^{p}}^{p}$ is the most suitable expert to be applied to the feature map $X$, and is retrieved from the $p$-th Expert Bank in this step.

It is worth mentioning that, using this key-query mechanism, the input feature maps that are highly similar (i.e., with similar $q$ values) will tend to have high matching scores with the same key and retrieve the same expert. Crucially, this leads to the experts having to discriminate between highly similar input samples, pushing each expert to specialize in exploiting subtle cues for distinguishing between those similar samples to tackle early action prediction.

Above, we only show the operations on the $p$-th Expert Bank, but the same process is conducted for all $d$ banks to retrieve $d$ experts, which is shown in Fig. 2. Notably, this process (Eq. 1, 2, 3) across $d$ Expert Banks can be done in parallel, so it is efficient.

Finally, to form the full convolutional block $W_{ERA}$, we further assemble the non-expert block $W_{nonexpert}$ and the expert block $W_{expert}$ (shown in Fig. 2) as:

The final assembled block $W_{ERA}$ will be applied to the feature map in a similar manner to a traditional convolutional kernel $W_{conv}$. Notably, as $W_{ERA}$ can directly replace $W_{conv}$, our ERA module is a plug-and-play module that can replace the basic convolutional layer.

To apply the ERA module to other types of convolutions that are used in early action prediction architectures, only minor changes need to be made. For 1D and 3D convolutions, we change the shape of $m_{i}^{p}$ and $W_{nonexpert}$ according to the corresponding 1D or 3D kernel. As for graph convolutions, since many graph convolutions (as used in [45]) are implemented based on traditional convolutions with additional parameters and steps to account for adjacency information, thus we can also implement our method by replacing the contained convolutional kernel with our ERA module in these scenarios.

Usually, the aggregated gradients $\bar{g}$ of a loss $\mathcal{L}$ w.r.t a model parameter $w$ is computed by averaging over the entire batch with batch size $B$:

The non-expert parameters are updated using $\bar{g}$ in Eq. 6, which trains the non-expert parameters to contribute towards classifying all samples, resulting in the learning of general patterns that apply to more samples, as opposed to subtle differences that occur only in a small subset of the data. If all parameters in a network are non-experts, this results in the network having sub-optimal performance with respect to subtle cues [20] and leads to worse performance on early action prediction.

In contrast, in our ERA module, not all experts are selected by each sample, as each expert is only retrieved for its most suitable samples. When backpropagating using the loss $\mathcal{L}$ on experts, the aggregated gradient $\bar{g}_{i}^{p}$ for the expert kernel weights $m_{i}^{p}$ thus becomes:

where $\mathcal{N}(k_{i}^{p})$ denotes the set of samples in the batch that select expert $E_{i}^{p}$ (with key $k_{i}^{p}$ and kernel $m_{i}^{p}$), i.e., $\mathcal{N}(k_{i}^{p})=\{j\text{ s.t }I_{j}^{p}=i\}_{j=1}^{B}$, where $I_{j}^{p}$ refers to the index of the selected expert in the $p$-th Expert Bank ($I^{p}$) for the $j$-th sample in the batch. The samples in $\mathcal{N}(k_{i}^{p})$ are likely to be very similar, with only some subtle differences, due to their close proximity to $k_{i}^{p}$ in the feature space.

If we train the expert $E_{i}^{p}$ using the gradient $\bar{g}_{i}^{p}$ as in Eq. 7, the expert is only updated using samples that are closer to this expert’s area of expertise, i.e., samples which are in $\mathcal{N}(k_{i}^{p})$. Thus, as compared to $\bar{g}$, much more emphasis is placed on learning to distinguish between these similar samples in $\mathcal{N}(k_{i}^{p})$ only, which pushes the expert to learn to exploit subtle differences in these samples, as opposed to general patterns that generally hold across all data.

Experts in our ERA modules, together with all other network parameters, are end-to-end trainable using backpropagation. However, due to the uneven distribution of samples across experts, some experts might be selected by more samples and be better trained than others, possibly causing imbalanced training that limits the performance of our ERA module. To mitigate this effect, we design an Expert Learning Rate Optimization (ELRO) method that optimizes the training among experts, leading to improved early action prediction accuracy. For ease of notation, in this section, we only use one ERA module, although this method can also work for multiple ERA modules. We introduce a set of expert learning rates $\beta=\{\beta_{i}^{p}\}_{i\in\{1..M\},p\in\{1..d\}}$ as additional parameters, where each element $\beta_{i}^{p}$ is a scalar that balances the training of a corresponding expert $E_{i}^{p}$ during backpropagation. Instead of using manual tuning to adjust the large set of $\beta$, we update $\beta$ using a meta-learning approach in an end-to-end manner.

The core idea of meta-learning [9, 47] is about “learning-to-learn”, which in our case is learning to optimize the learning rates $\beta$ for improved training of experts. This meta-optimization of $\beta$ is conducted over two steps. Firstly, we simulate training on a training set while using the current $\beta_{i}^{p}$ values to balance updates for each expert $E_{i}^{p}$ respectively, to obtain a virtually updated interim model. Next, we evaluate the performance of this interim model on a validation set, and the gradients of these validation losses will provide feedback on how we can adjust $\beta$ to a more optimized $\beta^{\prime}$ (which improves training of experts, and results in better performance on unseen validation samples). Finally, we then use the meta-optimized $\beta^{\prime}$ values for balancing expert updates during actual model training, which yields improvements in performance.

An illustration of our proposed ELRO method is shown in Fig 3. Specifically, in each iteration, we draw two batches of training data that are non-overlapping, which we call training samples $\mathcal{D}_{train}$ and validation samples $\mathcal{D}_{val}$. Then, the following three steps are employed to update the model parameters.

(1) Virtual Training. We simulate the training on $\mathcal{D}_{train}$ by virtually updating all the model parameters other than $\beta$ as follows:

where $w$ represents the non-expert parameters, $\mathcal{E}$ represents the set of experts $\{E_{i}^{p}\}_{i\in\{1..M\},p\in\{1..d\}}$, $\alpha$ is the learning rate (which is a fixed hyperparameter), and $\mathcal{L}$ refers to the supervised loss for the early action prediction task. Note that here the update of each expert $E_{i}^{p}$ (which includes key $k_{i}^{p}$ and kernel weights $m_{i}^{p}$) is scaled by $\beta_{i}^{p}$.

(2) Meta-Expert Optimization. In this step, we evaluate the performance of the virtually updated model (consisting of $\hat{w}$ and $\hat{\mathcal{E}}$) on $\mathcal{D}_{val}$. The gradients w.r.t each expert learning rate $\beta_{i}^{p}$ provide feedback on how $\beta_{i}^{p}$ should be tuned for the virtually updated model to generalize better to unseen samples, as follows:

Only $\beta$ is updated in this step, and other parameters ($\hat{w}$ and $\hat{\mathcal{E}}$) remain fixed. Note that $\nabla_{\beta_{i}^{p}}$ takes gradients with respect to $\beta_{i}^{p}$ as used in Eq. 9. This means that, by tuning $\beta_{i}^{p}$ in Eq. 10, the newly updated expert learning rate $\beta_{i}^{p}{}^{\prime}$ can provide better training for expert $E_{i}^{p}$ if Eq. 9 is performed again.

(3) Model Training. After we obtain the set of meta-optimized expert learning rates $\beta^{\prime}=\{\beta_{i}^{p}{}^{\prime}\}_{i\in\{1..M\},p\in\{1..d\}}$, we can perform actual model training by updating model parameters $\mathcal{E}$ and $w$ on $\mathcal{D}_{train}$ as:

In this step, the meta-optimized $\beta^{\prime}$ balances the training of experts such that performance on unseen samples is improved. This concludes one iteration of ELRO, where we have obtained updated parameters $w^{\prime}$, $\mathcal{E}^{\prime}$ and $\beta^{\prime}$. An outline of this algorithm is shown in the Supplementary Material.

We train our model using a cross-entropy loss $\mathcal{L}_{CE}$ on the early action prediction task. Furthermore, we find that applying an additional similarity loss $\mathcal{L}_{s}$ brings some improvements in practice, where the similarity loss $\mathcal{L}_{s}$ penalizes experts that get too close to each other, which encourages experts to be more diverse. Specifically, we implement $\mathcal{L}_{s}$ on all ERA modules within our network, using a negative pairwise mean-squared loss among expert kernels in each Expert Bank, i.e., using one ERA module as an example, $\mathcal{L}_{s}=\sum_{p=1}^{d}\sum_{i=1}^{M}\sum_{j\neq i}||m_{i}^{p}-m_{j}^{p}||^{2}$. Overall, our loss $\mathcal{L}$ is then given by $\mathcal{L}=\mathcal{L}_{CE}-\gamma_{s}\mathcal{L}_{s}$, where $\gamma_{s}$ is a hyperparameter that weights the relative significance of losses.

Network Architecture. Following the previous works [30, 54], we use 2s-AGCN [45] and 3D ResNeXt-101 [13] as the backbone networks for skeletal and RGB datasets, respectively. As mentioned above, since the ERA module serves as a plug-and-play replacement for the conventional convolutional module, we uniformly replace $25\%$ of convolutional layers with our ERA module in the backbone networks. Network hyperparameters $N_{in},N_{out},b_{h},b_{w}$ at each layer follow the original settings in the backbone networks. Also, in each ERA module, $d=0.2N_{out}$, i.e. $80\%$ of the convolutional kernels are non-expert kernels and the other $20\%$ are expert kernels, and $M=5$.

For the mapping function $f^{p}$ in Eq. 1, we first conduct average pooling across the spatial and temporal dimensions of the feature map before a linear layer is used to downsample to the dimensionality $K$, where $K$ is set to $64$.

Training. We perform experiments on Nvidia RTX 3090 GPU. For skeletal datasets, NTU60, NTU120 and SYSU, we follow [45] and set the initial learning rate $\alpha$ as $0.1$, which then gradually decays to $0.001$. The batch size $B$ is 64. For RGB dataset UCF101, we follow the same experimental settings as [54]. Network parameters $\theta$ and expert learning rates $\beta$ are updated using $\mathcal{L}$ defined in Sec. 3.4. $\gamma_{s}$ is set to 0.1. Each $\beta_{i}^{p}$ is initialized to $\alpha$, and constrained to non-negative values. To allow end-to-end training of the retrieved experts in Eqn. 3, Gumbel-Softmax [19] gradients are computed during backpropagation for the Argmax operation, with temperature $\tau$ set to $1$.

NTU60 dataset [42] has been widely used for 3D action recognition and early action prediction. It is a large dataset that contains more than 56 thousand skeletal sequences from 60 activity classes. All human skeletons in the dataset contain 3D coordinates of 25 body joints. As noted in [30], this dataset is challenging for the 3D early action prediction task due to the presence of many classes with very similar starting sequences. We follow the evaluation protocol of [30].

We first compare the proposed ERA-Net with the state-of-the-art approaches on NTU60. The results over different observation ratios are shown in Table 1. Our full method is employed in the ERA-Net setting. In ERA-Net w/o ELRO, we use ERA modules but do not implement $\beta$ to train experts using the proposed ELRO algorithm, instead we train using a single backpropagation step that updates all model parameters at each iteration. We also provide the Baseline setting for comparison, where the backbone is used without ERA modules.

We report the prediction accuracy at each observation ratio. Furthermore, we use the Area Under Curve (AUC) metric in our experiments, following previous works [30, 55, 39]. The AUC measures the average precision over all observation ratios and broadly summarizes each model’s performance into a single metric. On NTU60, we achieve more than a 3 point improvement against existing state-of-the art method [30], suggesting that the ERA module effectively increases the discriminative capabilities on the early action prediction task.

One crucial observation is that ERA-Net outperforms existing methods more significantly when the observation ratio is low. For example, when the observation ratio is $20\%$, ERA-Net improves over state-of-the-art [30] by more than $11\%$, which further demonstrates that the ERA module is especially effective in picking up subtle cues to tackle hard samples (where samples are more similar at the earlier stages).