Improving ProtoNet for Few-Shot Video Object Recognition: Winner of ORBIT Challenge 2022

Recently, Few-shot Learning has received increasing attention [19, 15, 7] as it allows models to recognize novel objects from only a few examples. This will enable computer vision systems adapt to dynamic real-world environments where users can provide a few training examples themselves. Few-shot concept is adapted and extended to various real-world settings, such as few-shot continual learning [4, 10, 17], test-time adaptation [5, 14, 8] and leveraging domain shift [21, 3].

Existing datasets in few-shot learning undergo a lack of high variation in both the number of examples per object and the quality of those examples. The resulting trained object recognizers are not robust to the noisy input data (e.g., video frames) streamed from real-world systems. To drive further innovation in few-shot learning, ORBIT dataset [11] captures the high variations inherent in real-world applications via the collection of thousands of videos recorded by vision-impaired people. ORBIT Few-Shot Object Recognition Challenge, newly introduced in 2022, invited teams to build a teachable object recognizer using the ORBIT dataset. Unlike a generic object recognizer, a user can ‘teach’ a teachable object recognizer to recognize their specific personal objects by providing just a few video sequences. Specifically, to register the object categories to be classified, a few clean video sequences with several user-centric objects are provided for the recognizer at the personalization stage. Then, the different video sequences from the same user are used to evaluate the recognizer at the recognition stage. Considering the fact that vision-impaired people cannot localize the target object accurately and quickly, video frames at the recognition stage are collected from cluttered scenes containing multiple objects.

[11] establishes the baseline on the ORBIT benchmark by extending 4 mainstream meta-learning based methods from image to videos [18, 12, 13, 7]. LITE [1] scales up the input image resolution with a general and memory-efficient episodic training scheme and thus achieves state-of-the-art accuracy. However, no algorithmic design for handling noisy examples and sampling video frames is considered in the existing methods. As observed in the real-world dataset, video frames are quite noisy, especially for those taken by vision-impaired people. In addition, the video frame sampling method used in the prior works is inefficient and ineffective. Since the video lengths are variable, random sampling may cause over-sampling or under-sampling. Both factors hamper the performance of the trained object recognizer.

We build our solution on top of the state-of-the-art method, ProtoNet [13] with LITE. In the ORBIT benchmark, ProtoNet firstly uses clean video clips as the support data to produce category-level prototypes at the personalization stage. Then, the clutter video clips as the query data can be classified by directly comparing their embeddings with the prototypes using a similarity metric at the recognition stage. To generate high-quality prototypes at the personalization stage, we add three techniques: First, inspired by [20], we incorporate one transformer encoder block to leverage the relationship among prototypes to enable the adaptation to the specific episode, and thus highlight their discriminative representation for a specific user. Also, we replace the random video clip sampler by the uniform sampler to enable higher temporal coverage during testing and to reduce down-sampling and over-sampling in long and short video sequences respectively. Last, we apply an edge detector on each sampled video frame and set an empirical threshold to determine and remove the frame that contains nothing. Our approach improve the accuracy significantly by 8% and was selected as the winner from 12 submissions in the ORBIT Few-Shot Object Recognition Challenge.

In addition to our innovation on algorithm design, we also refactor and re-implement the data pipeline in the original ORBIT codebase to encourage modularity, compatibility and performance improvement. Specifically, our implementation achieves more than 2.7x acceleration in data loading during both training and testing.

In this section, we present the details of our solution. We begin by introducing the entire pipeline of the baseline method using ProtoNet in Few-shot Video Object Recognition. Then, we introduce several improvements to each module, including a few-shot learner, video clip sampler, and frame edge detector, respectively.

To improve the code efficiency, we refactor and re-implement the data pipeline of the original official codebase to encourage modularity, compatibility and performance.

In this work, we proposed several improvements for the few-shot video object recognition task. The proposed components consist of embedding adaptation, uniform video sampling, and invalid frame detection. Our unified solution achieves the winner of the first challenge on the ORBIT dataset. Furthermore, we also contribute to refactoring and optimizing the original codebase to improve the productivity of other researchers. Future work includes the solution to tackle the domain shift between support and query videos which is quite common in real-world scenarios.