NMS Threshold matters for Ego4D Moment Queries

ActionFormer [12] takes as input a 1D sequence of clip-level video features and builds a feature pyramid using local self-attentions. This pyramid serves as a multi-scale representation of moment candidates. Each location on the pyramid defines the center of a moment, whose temporal scale is determined by the pyramid level (i.e., longer moments reside on higher levels of the pyramid). A classification head subsequently assigns a confidence score to each moment, whereas a regression head predicts the distances from the center of a moment to its onset and offset. These predictions are decoded into action segments and further combined using SoftNMS [2]. We refer readers to [12] for more details on ActionFormer.

ActionFormer follows a fixed set of rules collectively known as center sampling to convert ground-truth action segments into point-wise classification labels. Center sampling was first seen in the literature of single-stage object detection [11], in which it has lately been superseded by more powerful, dynamic label assignment strategies [4, 5] that evolve in tandem with training losses. In this work, we report a similar finding that training ActionFormer with SimOTA [5], an efficient dynamic label assignment technique, yields a small yet significant performance gain compared to using center sampling. We provide ablation results in Section 3 and refer readers to [5] for more details on SimOTA.

Our initial exploratory analysis revealed that $15\%$ of ground-truth moments in the Ego4D-MQ dataset are near-replicates (i.e., $\geq 90\%$ overlap with another moment). This presents a unique challenge to ActionFormer, which relies on aggressive non-maximum suppression (NMS) to reduce highly overlapping predictions, thereby harming precision at high recall in the presence of near-replicates. In this work, we propose to tune the standard deviation $\sigma$ of the Gaussian penalty function $f$ in SoftNMS [2] as a simple fix. Intuitively, a small $\sigma$ as recommended by [2] yields a peaky $f$ that incurs strong penalty on near-replicates, whereas $f$ flattens out as $\sigma$ increases, leaving near-replicates less affected. We empirically found that setting $\sigma$ to the unusually large value of 2 brought a significant improvement in mAP by 1.8 absolute percentage point compared to using the default value of 0.9 as in [10].

We follow the official train/val/test splits for evaluation. Our model is trained on the train split when results are reported on the val split, and is trained on the train and val splits combined when results are submitted for final evaluation on the test split. In line with the official guideline, we adopt average mAP as our main evaluation metric.

We use pre-extracted Omnivore [6], EgoVLP [7] and InternVideo [3] features from raw videos as input to ActionFormer, and set the embedding dimension throughout the model to 1152. Following the official code release, we train ActionFormer using the AdamW optimizer [8] for 15 epochs with a mini-batch size of 2, a learning rate of 0.0001 and a weight decay of 0.05.

Our results on the val and test splits are summarized in Table 1. Replacing the official SlowFast features with InternVideo features brings a notable 2.71 absolute-percentage-point improvement on average mAP on the val split. This highlights the strength of InternVideo as a video foundation model for representation learning. The introduction of dynamic label assignment further boosts the average mAP by a small yet significant 0.3 absolute percentage point. Finally, the largest performance gain ($>$1.3 absolute percentage points on both val and test splits) is attained by tuning the spread of Gaussian penalty function in SoftNMS to account for near-replicates. With everything combined, ActionFormer reaches an average mAP of 26.07$\%$ on the val split and 26.62$\%$ on the test split. Figure 2 provides visualizations of model predictions.

We present false negative analysis (Figure 3), false positive analysis (Figure 4) and sensitivity analysis (Figure 5) of our method. Our method demonstrates stronger performance on videos with high moment coverage and is more capable of identifying longer actions. It exhibits significantly higher error rate on videos with low moment coverage and was not able to accurately localize short actions. This is further manifested by the surprisingly high background error rate.

We thank Chen-Lin Zhang for fruitful discussions about ActionFormer.