Track Any Peppers: Weakly Supervised Sweet Pepper Tracking Using VLMs

Given a sequence of frames $\mathcal{F}=\{f_{1},f_{2},\ldots,f_{T}\}$ from a video, the objective is to track small objects (e.g., sweet peppers) across consecutive frames. In each frame $f_{t}$, potential object detections $d_{i}^{t}$ are identified, where each detection $d_{i}^{t}$ is characterized by a bounding box $b(d_{i}^{t})=(x_{\min},y_{\min},x_{\max},y_{\max})$, a detection probability $p(d_{i}^{t})$, and a segmentation mask $m(d_{i}^{t})$. The goal is to associate these detections across frames to form consistent object trajectories $\mathcal{T}_{k}=\{d_{k}^{1},d_{k}^{2},\ldots,d_{k}^{N_{k}}\}$ for each object $k$, where $N_{k}$ denotes the number of frames in which object $k$ appears. This tracking task must address challenges such as occlusion, clutter, and varying lighting conditions to maintain accurate and continuous trajectories throughout the video sequence.

We utilize Grounding DINO, a vision-language model, to perform zero-shot object detection using textual queries like "bell_pepper", enabling us to detect objects without prior training on our dataset. Unlike conventional detection models, Grounding DINO uses this semantic information to identify target objects. For each image sequence $\mathcal{F}=\{f_{1},f_{2},\ldots,f_{T}\}$, where $f_{t}$ represents the frame at timestamp $t$, we define a sub-sampling interval $I$ to extract $N=\left\lfloor\frac{T}{I}\right\rfloor$ images, minimizing redundancy from visually similar frames. These sub-sampled images are input to Grounding DINO, which generates a set of pseudo-bounding boxes $\mathcal{P}=\{p_{1},p_{2},\ldots,p_{N}\}$, with each $p_{i}$ representing the coordinates of a detected object. To ensure quality, we consider only pseudo-labels with high confidence scores. The confidence score $c_{i}$ for each detection $p_{i}$ is computed as $c_{i}=\sigma(l_{i})$, where $\sigma(\cdot)$ is the sigmoid function and $l_{i}$ is the logit output associated with $p_{i}$. We select detections satisfying $c_{i}\geq\tau$, where $\tau$ is a predefined threshold, resulting in a refined set $\mathcal{P}^{\prime}=\{p_{i}\mid c_{i}\geq\tau\}$. Human experts then further refine these selected pseudo-labels to enhance their accuracy, particularly in the cluttered environment. Each image is manually inspected, and only those with misaligned boxes or significant detection errors are refined, typically requiring less than 2 minutes per image. The refined bounding boxes serve as prompts for the Segment Anything Model (SAM), which generates precise segmentation masks $\mathcal{M}=\{m_{1},m_{2},\ldots,m_{N}\}$ corresponding to the detected objects. These refined labels and masks are subsequently used in later stages for model training and detection.

We train a YOLOv8 [9] model using the refined bounding boxes $\mathcal{P}^{\prime}$ and segmentation masks $\mathcal{M}$ as inputs. While vision-language models like Grounding DINO exhibit strong zero-shot capabilities, we observed that YOLOv8 achieves superior performance due to its ability to generalize effectively with limited training data. This performance gain is attributed to YOLOv8’s robustness against overfitting and reduced susceptibility to over-parameterization, both common challenges when fine-tuning large foundational models on small datasets. The YOLOv8 model is trained using a combination of three loss functions: classification loss $\mathcal{L}_{\text{cls}}$, localization loss $\mathcal{L}_{\text{loc}}$, and segmentation loss $\mathcal{L}_{\text{seg}}$. The overall training objective is to minimize the total loss $\mathcal{L}_{\text{overall}}$, defined as:

where $\lambda_{\text{cls}}$, $\lambda_{\text{loc}}$, and $\lambda_{\text{seg}}$ are hyperparameters that weight the contribution of each loss component.

To enhance detection quality under varying lighting conditions, we implement an adaptive contrast and luminance adjustment scheme prior to inference. Our approach dynamically adjusts the luminance and contrast of each frame $f_{t}$ based on its overall luminance $\mathcal{L}(f_{t})$. The adjusted luminance $\mathcal{L}^{\prime}(f_{t})$ and contrast $\mathcal{C}^{\prime}(f_{t})$ are computed as:

where $\mathcal{C}(f_{t})$ represents the original image contrast. The scaling factors $\alpha(\mathcal{L}(f_{t}))$ and $\beta(\mathcal{L}(f_{t}))$ are functions of the frame’s luminance, designed to reduce luminance when it’s high and boost contrast when it’s low. By dynamically adjusting these factors, we enhance detection performance across a range of illumination levels by suppressing overexposed or underexposed areas and emphasizing object boundaries.

After training, we use the YOLOv8 model to detect sweet peppers in each video frame $f_{t}$. Once detections are obtained, we apply the segmentation masks $\mathcal{M}$, generated by YOLOv8, along with depth maps to further filter the detected objects. Let $d_{i}$ denote the depth of instance $i$, and let $\tau_{d}$ be a threshold distinguishing foreground from background objects. Each object is classified as:

For instances classified as foreground ($d_{i}<\tau_{d}$), we retain the predictions $\hat{y}_{i}=(\hat{b}_{i},\hat{p}_{i},\hat{m}_{i})$, where $\hat{b}_{i}$, $\hat{p}_{i}$, and $\hat{m}_{i}$ are the predicted bounding box, detection probability, and segmentation mask, respectively. Instances classified as background ($d_{i}\geq\tau_{d}$) are discarded ($\hat{y}_{i}=\varnothing$).

After detecting the bounding boxes in each frame of the video sequence, we perform object tracking using a hybrid ensemble approach. The first component employs the BoT-SORT [1] algorithm, a multi-object tracking method that associates detections across frames based on bounding box overlap and appearance features. Simultaneously, we utilize the detected bounding boxes as prompts for the Matching Anything by Segmenting Anything (MASA) [7] adapter, built on SAM. MASA enables instance-level correspondence tracking without requiring explicit video annotations by learning instance representations from dense, unlabeled data. This improves the model’s ability to track objects across frames by leveraging prompt-based supervision.