Dense Monocular Motion Segmentation Using Optical Flow and Pseudo Depth Map: A Zero-Shot Approach

Intensity based methods are traditional methods based on the image brightness constancy constraint, which presumes the pixel intensity of an image stays constant over a short period of time. Intensity based methods can be further categorized into indirect and direct methods. Indirect methods [8, 9, 10, 13] use pixel-wise correspondences as input and generate a pixel-wise segmentation mask that delineates different motion groups. Such pixel correspondences are usually obtained from optical flow, which is based on the brightness constancy constraint. In contrast, direct methods [32, 15, 33, 34] directly take a pair of images as input and combine the two processes of optimizing for the brightness constancy constraint and estimating the motion models together. Most recent works on intensity based methods use optical flow based indirect methods, possibly due to the fast advance in optical flow estimation [35, 36]. Intensity based methods typically use causal inference or iterative optimization techniques to estimate the motion regions and motion models simultaneously [8, 9, 10, 13].

Intensity-based methods work well on simple scenes where the object motion and scene structure are simple, but will fail on more complex objects or scenes. For example, the motion of a walking human may not be modeled by a single optical flow mask since it may contain multiple moving parts (legs, arms and torso). In this case, these methods can have problems like over-segmentation or under-segmentation, where they segment different human parts as different motions, or only segment part of the human as the moving object. In addition, if the scene exhibits significant depth variation, these methods often struggle to determine whether a part of the image is moving independently or is simply located at a different depth compared to its surroundings.

Most deep learning methods take a sequence of image frames and sometimes also the precomputed motion cues such as the optical flow field or the monocular depth maps of these images, and produce a dense segmentation map in an end-to-end manner. Earlier methods are often only able to perform binary motion segmentation [3, 17, 4, 20, 18], but more recent methods have made significant progress and are able to achieve promising results in multi-label motion segmentation [5, 2, 7, 6, 37, 21].

Many deep learning based methods adopt a fully supervised approach [3, 17, 5, 2]. These methods typically train a CNN-based encoder-decoder network to perform end-to-end learning, which is computation-intensive. Their network architecture usually have the following components: (1) a module to extract the motion information from consecutive image frames, (2) a module to extract appearance information from the same sequence of frames, (3) a module to fuse the appearance and motion information together, and (4) a decoder to generate the final segmentation. These methods perform very well on scenes similar to the datasets they are trained on, but cannot scale well to unseen environment where there are different motion patterns or object classes. Moreover, the data collection and training process are very time-consuming and computation intensive, which make them not an ideal method.

Aside from supervised methods, some methods also use semi-supervised, self-supervised and unsupervised approaches. In [19], the authors extended their previous work [13] by proposing an self-supervised approach to train a neural network to perform motion segmentation on synthetic angle fields, given that most optical flows can be reduced to rotation-compensated angle fields. In [38], the authors proposed an unsupervised learning method to solve multi-label motion segmentation problem by training a neural network to mimic the motion segmentation results from the Expectation-Maximization (EM) algorithm. However, these two methods purely rely on optical flow for motion information and thereby inheriting its limitations. To alleviate this problem, [37] proposes to train image segmentation and motion segmentation models together using both optical flow and raw video frames as inputs due to the fact that motion and appearance cues are usually highly related in practice. The unsupervised training is done in a very similar way as [38] using the EM-algorithm.

Unlike Intensity based methods or deep learning methods, sparse correspondence based methods cannot produce dense segmentation masks of different moving objects. Instead, they output clusters of predefined kepypoints corresponding to different motion groups instead of dense segmentation masks. These methods can be further categorized into two-frame based methods and multi-frame based methods. Two-frame based methods [39, 22, 23, 40] typically determine motion parameters using iterative optimization methods. This involves identifying a specific number of geometric models, such as homography, based on a set of corresponding keypoints, aiming to minimize an energy function representing the overall quality of the corresponding keypoints clustering. Unlike two-frame based methods, multi-frame based methods [28, 29, 30, 41, 42, 26, 43, 44, 45, 25, 27] usually establish point correspondences over multiple frames using an optical flow based point tracker. Noisy, occluded and unwanted points are often manually removed to produce a sparse set of completely noise-free point trajectories. Multi-frame based methods have proven to be superior than two-frame based methods due to their ability to analyze the motion data in a longer time window using various geometric models and spatio-temporal similarities. Moreover, unlike two-frame based methods which only rely on epipolar geometry to detect different motions, multi-frame based methods are able to combine different geometric models together and achieve impressive results in challenging scenes with complex motions.

To automatically identify all moving objects within a video, the first task is to recognize and detect each common object in the video and track their trajectories throughout the video. We leverage the recently proposed computer vision foundational models in object recognition (RAM)[46], detection (Grounding DINO)[47], and segmentation (SAM)[48], alongside a state-of-the-art object tracking model (DeAOT) [49]. This preprocessing pipeline is based on the Segment and Track Anything (SAMTrack) [50] framework, which integrates the aforementioned models into a unified object segmentation and tracking system. SAMTrack generates dense object tracking masks based on a user-defined textual prompt specifying the desired objects to be tracked. To automate our system and eliminate the need for manual text prompts, we incorporate RAM at the beginning of our pipeline to automatically identify common objects in the initial video frame.

In summary, our complete preprocessing pipeline involves using RAM to identify common objects in the first frame, using the output from RAM as a textual prompt for the Grounding DINO model to obtain object bounding boxes, and then using these bounding boxes with SAM to generate an instance segmentation mask of the first frame. Non-max suppression is applied to remove overlapping objects with an IoU score greater than 0.5 or with a mask area exceeding half the image size. Finally, The DeAOT tracker is used to follow each object’s mask throughout the video.

To account for potential new objects entering the scene midway through the video, we divide the video into multiple parts consisting of equal numbers of frames and apply the full preprocessing pipeline to each part individually. The specific length of the each individual part can vary, but it typically more beneficial to set smaller number in more dynamic videos where more objects enter the scene midway.

In order to determine if a set of objects have the same motion, we need to analyze object-specific motion cues for each object. More specifically, we calculate a dense optical flow mask and a monocular relative depth map for every video frame as the motion cues. The extraction of optical flow masks and relative depth maps are conducted using PWC-Net [51] and DINOv2 [52] respectively. Both models are the state-of-the-art models in their respective domains.

Relying solely on optical flow for motion segmentation is inadequate, as it cannot effectively distinguish between different motions and different depths. This limitation becomes evident when a camera moves, causing two stationary objects at different depths to appear as if they are moving differently due to motion parallax. To address this limitation, it is essential to integrate depth information with optical flow, enhancing its ability to accurately analyze motion. In the following section, we present a parametric model that combines optical flow and depth data. We will demonstrate how this model can be used to compute the three-dimensional screw motions of objects, thus enabling the differentiation of objects based on their motions. This parametric model offers a robust theoretical framework for understanding complex motions in dynamic scenes.

We propose a motion model fitting algorithm that uses a parametric model derived from optical flow and depth data to represent the motions of individual objects throughout the video. This parametric motion model incorporates a revised version of the model equations introduced by Longuet-Higgins and Pruzdny [53], which can be used to compute the instantaneous screw motion of rigid objects at arbitrary depths. The original Longuet-Higgins and Pruzdny model equations establish a relationship between the optical flow, the instantaneous screw motion of rigid objects and the depths of individual pixels as follows:

Here, $u$ and $v$ denote the optical flow vectors along the x and y axes respectively, $z$ denotes pixel depth, $f$ stands for the camera’s focal length, and $\tau_{1}$, $\tau_{2}$, $\tau_{3}$, $\omega_{1}$, $\omega_{2}$, $\omega_{3}$ symbolize the object’s translational and rotational movements. Nonetheless, the absolute depth of each pixel is often unknown in practice, making the complete utilization of this model for computing object motion unfeasible. To overcome this limitation, existing methods often use a parametric equation to infer object motion directly from the optical flow without knowing depth. For instance, [21] applies a segmented parametric formula with 12 parameters to precisely align with the optical flow field:

However, such a motion model lacks theoretical accuracy and fails to accommodate scenes with significant depth variations. Other research [13, 16] employs a simpler parametric motion equation to estimate the rotation-compensated optical flow angle field, albeit requiring known camera intrinsic parameters, which is impractical. To establish a motion model that is both theoretically robust and independent of camera intrinsics, we propose to linearize the Longuet-Higgins and Pruzdny equations using the monocular depth map produced from DINOv2. With the relative depth of each pixel known, we can reformulate the original equations into the following linear parametric form:

This set of linearized equations is more robust to motion parallax than (2) where the depth value is encoded in multiple unknown parameters and need to be approximated together with the screw motions. Using relative depth is sufficient for our goal, which is to distinguish different motions instead of computing the absolute values of the screw motion parameters.

Once all optical flow motion models are obtained, each object will have a parametric motion model for every pair of frames. By fitting every object’s optical flow vectors and depth map on its parametric motion model for the same frame pair, we can compute the residuals of each object to the motion models of all other objects. The residuals are computed with the mean squared error. Given N objects proposed for the scene in total, the motion residual vector for the i-th object at frame pair m are derived as follows:

where ${e}_{i,n}^{m}$ represents the mean residual calculated by fitting the motion model of object $i$ to the optical flow and depth of object $n$ for frames $m$ and $m+1$. A motion similarity matrix is then constructed to encode the motion similarity scores across all pairs of objects. This is achieved using the ordered residual kernel (ORK) [54]. To do so, the residual values in each residual vector are first sorted in ascending order. A threshold t is set to select the lowest t-th residuals as inliers. An binary inlier vector ${\boldsymbol{v}_{i}}=\{0,1\}^{N}$ is then obtained for each object, where $N$ is the total number of objects. The pairwise motion similarity between objects $i$ and $j$ can be calculated as ${\boldsymbol{d}_{ij}=\boldsymbol{v}_{i}^{\intercal}\boldsymbol{v}_{j}}$, representing the frequency at which these two objects occur as each other’s motion inliers. The ORK selects a certain number of inliers from each object’s motion vectors instead selecting inliers below a threshold, making it more robust to different scenes and motions. After the motion similarity matrix is constructed, each motion similarity score ${\boldsymbol{d}_{ij}}$ is normalized by dividing it with the number of frames that objects $i$ and $j$ have in common. This normalization step helps eliminate the weighting bias caused by incomplete trajectories.

We apply row normalization [55] to the constructed motion similarity matrix and use spectral clustering to cluster objects into different motion groups. Given a predefined number of motion groups, spectral clustering cluster objects with high motion similarity values into the same motion group. Spectral clustering is a widely adopted technique in sparse correspondence based motion segmentation methods. It has shown effectiveness in clustering different point trajectories into different motion groups [29, 56, 41]. Therefore, we use it to cluster different objects in the object proposal into different motion groups given the motion similarity matrix.

DAVIS-Moving and YTVOS-Moving [5] are the most recent benchmarks for generic instance motion segmentation. They are subsets of the DAVIS-17 dataset [57] and the YTVOS dataset [58] respectively. On the contrary to the original DAVIS and YTVOS datasets which focus on video object segmentation and only label moving objects in the foreground, the DAVIS-Moving and YTVOS-Moving datasets contain only videos where all moving objects are labeled. These datasets are particularly challenging due to occlusions, non-rigid motions and the diversity of object classes. We evaluate our method using the precision (Pu), recall (Ru), and F-score (Fu) metrics, as proposed by [5]. These metrics are designed to penalize false positives, with the F-score providing an overall performance score by combining precision and recall.

Tables I and II present quantitative comparisons of our method with the state-of-the-art unsupervised motion segmentation method (EM) [21] on DAVIS-Moving and YTVOS-Moving datasets. The comparisons are limited to binary motion segmentation due to the availability of model weights from [21]. However, these results are indicative of performance on multi-label segmentation tasks. Our method demonstrates superior performance on both datasets, producing higher F-scores than EM, showing more accurate segmentation results.

Figure 3 shows a qualitative comparison between our method and EM on the DAVIS-Moving and YTVOS-Moving datasets. The first row shows the original video frames, the second row shows the results from EM, the third row shows the results from out method, and the last row shows the ground truth. Both methods can detect moving objects in the scene, but our method produces much more coherent object masks and produces significantly less false-positive segments.

Table III shows quantitative comparison of our method and state-of-the-art supervised and semi-supervised methods on the two benchmarks. Both binary and multi-label motion segmentation scenes are used. Although we did not conduct any training, our method achieves competitive results and even outperforms one of the semi-supervised methods (RigidMask).

We also show qualitative comparison with these methods in Figure 4. MoSeg[5] produces the best results on both datasets. RigidMask[6] fails to produce coherent segmentation on most scenes, and also fails to detect the motions of the parrot and the train on the last two rows, whereas our method successfully detects and segments them coherently. Raptor[7] is able to detect most objects in the scene thanks to its powerful semantic backbone, but it still over-segments non-rigid objects like the parrot. Our method performs well in these cases, performing almost as well as the supervised method.

Our method has two main limitations: slow inference speed due to multiple deep learning models, and the need for a predefined number of motions in the scene. The latter can be addressed by incorporating an automatic model selection technique [59]. The inference speed, however, can only be improved by training an end-to-end network using our proposed motion residual functions.

We present an ablation study to compare the performances of the two different motion models with the baseline, which is obtained from the raw object proposal alone. Both qualitative (Figure. 5) and quantitative (Table IV) comparisons are shown between our motion segmentation result obtained by only using optical flow as the motion cue vs. using both optical flow and the depth map, as well as the baseline results.

To produce the motion segmentation results only from optical flow, We use the optical flow motion model of EM [21], which is the state-of-the-art unsupervised motion segmentation method using only optical flow as input. Their 12-parameter quadratic parametric motion model integrates the unknown depth information as unknown parameters (as shown in equation (2)). Results indicate that the motion model integrating both optical flow and depth (OC + Depth) significantly outperforms the model relying solely on optical flow (OC) across all metrics on DAVIS-Moving. However, on the YTVOS-Moving dataset, the performance improvement is small, suggesting that unknown depth information is not a critical limiting factor. This can be attributed to several reasons: First, many objects labeled as moving in YTVOS-Moving are static in most frames. Second, the dataset includes significant occlusions and uncommon objects such as camouflaged animals. These challenges likely have a greater impact on the accuracy of motion segmentation than the absence of depth information.