DyRoNet: Dynamic Routing and Low-Rank Adapters for Autonomous Driving Streaming Perception

The existing streaming perception methods fall into three main categories. (1) The initial methods focused on single-frame, with models like YOLOv5 [15] and YOLOX [6] achieving real-time performance. However, lacking motion trend capture, they struggle in dynamic scenarios. (2) The recent approaches incorporated current and historical frames, like StreamYOLO [35] building on YOLOX with dual-flow fusion. LongShortNet [19] used longer histories and diverse fusion. DAMO-StreamNet [10] added asymmetric distillation and deformable convolutions to improve large object perception. (3) Recognizing the limitations of single models, current methods explore dynamic multi-model systems. One approach [7] adapts models to environments via reinforcement learning. DaDe [14] extends StreamYOLO by calculating delays to determine frame steps. A later version [13] added multi-branch prediction heads. Beyond 2D detection, streaming perception expands into optical flow, tracking, and 3D detection, with innovations in metrics and benchmarks [32, 25, 30]. Distinct from these existing approaches, our proposed method, DyRoNet, introduces a low-rank enhanced dynamic routing mechanism specifically designed for streaming perception. DyRoNet stands out by integrating a suite of advanced branch networks, each fine-tuned for specific environmental conditions. Its key innovation lies in the speed router module, which not only routes input data efficiently but also dynamically adapts to the diverse and unpredictable nature of real-world driving scenarios.

Dynamic Neural Networks (DNNs) feature adaptive network selection, outperforming static models in efficiency and performance [9, 17, 36]. The existing research primarily focuses on structural design for core deep learning tasks like image classification [12, 31, 29]. DNNs follow two approaches: (1) Multi-branch models [1, 3, 26, 28, 24, 33, 18] rely on a lightweight router assessing inputs to direct them to appropriate branches, enabling tailored computation. (2) By generating new weights based on inputs [34, 5, 27, 39], these models dynamically alter computations to match diverse needs. DNN applications expand beyond conventional tasks. In object detection, DynamicDet [21] categorizes inputs and processes them through distinct branches. This illustrates DNNs’ broader applicability and potential for dynamic environments.

Autonomous driving faces variability from weather, scene complexity, and vehicle velocity. By strategically analyzing key factors and routing logic, this section details the rationale behind the proposed DyRoNet.

Mathematically, the core of DyRoNet lies the processing of a frame sequence, $\mathcal{S}=\{I_{t},\cdots,I_{t-N\delta t}\}$, where $N$ indicates the number of frames and $\delta t$ the interval between successive frames. The framework process is formalized as:

where $\mathcal{P}=\{P_{0},\cdots,P_{K-1}\}$ denotes a collection of streaming perception models, with each $P_{i}$ denoting an individual model within this suite. The architecture is further enhanced by incorporating a feature extractor $\mathcal{G}_{i}$ and a perception head $\mathcal{H}_{i}$ for each model. The Router Network, $\mathcal{R}$, is instrumental in selecting the most suitable streaming perception model for each specific scenario.

Correspondingly, the weights of DyRoNet are denoted by $\mathcal{W}=\{W^{d},W^{l},W^{r}\}$, where $W^{d}$ indicates the weights of the streaming perception model, $W^{l}$ relates to the Low-Rank Adaptation (LoRA) weights within each model, and $W^{r}$ pertains to the Router Network. The culmination of this process is the final output, $\mathcal{T}$, a compilation of feature maps. These maps can be further decoded through $Decode(\mathcal{T})$, revealing essential details like objects, categories, and locations.

Specifically, for frames at times $t$ and $t-1$, represented as $I_{t}$ and $I_{t-1}$ respectively, the frame difference is computed as $\Delta I_{t}=I_{t}-I_{t-1}$. The architecture of the Router Network, $\mathcal{R}$, is simple yet efficient. It consists of a single convolutional layer followed by a linear layer. The network’s output, denoted as $f^{r}\in\mathbb{R}^{K}$, captures the essence of the environmental dynamics. Based on this output, the index $\sigma$ of the optimal branch for processing the current input frame $I_{t}$ is determined through the following equation:

where $\sigma$ is the index of the branch deemed most suitable for the current environmental context. Once $\sigma$ is determined, the input frame $I_{t}$ is automatically routed to the corresponding branch by a dispatcher.

In particular, this strategy of using frame differences to gauge environmental speed is efficient. It offers a faster alternative to traditional methods such as optical flow fields. Moreover, it focuses on frame-level variations rather than the speed of individual objects, providing a more generalized representation of environmental dynamics. The sparsity of $\Delta I_{t}$ also contributes to the robustness of this method, reducing computational complexity and making the Router Network’s operations nearly negligible in the context of the overall model’s performance.

Note that the versatility of DyRoNet is further highlighted by its compatibility with a wide range of Streaming Perception models, even ones that rely solely on detectors [6]. To demonstrate the efficacy of DyRoNet, it has been evaluated using three contemporary streaming perception models: StreamYOLO [35], LongShortNet [19], and DAMO-StreamNet [10] (see Sec. 4.3). This Model Bank & Dispatcher strategy illustrates the adaptability and robustness of DyRoNet across different streaming perception scenarios.

To address these challenges, we have incorporated the LoRA [11] into DyRoNet. Within each model $P_{i}$, initially pre-trained on a dataset, the key components are the convolution kernel and bias matrices, symbolized as $W^{d}_{i}$. The rank of the LoRA module is defined as $r$, a value significantly smaller than the dimensionality of $W^{d}_{i}$, to ensure efficient adaptation. The update to the weight matrix adheres to a low-rank decomposition form, represented as $W_{d}^{i}+\delta W=W_{d}^{i}+BA$.²²2Here, $B$ is a matrix in $R^{d\times r}$, and $A$ is in $R^{r\times k}$, ensuring that the rank $r$ remains much smaller than $d$. This adaptation strategy allows for the original weights $W_{d}^{i}$ to remain fixed, while the low-rank components $BA$ are trained and adjusted. The adaptation process is executed through the following projection:

where $x$ represents the input image or feature map, and $\Delta W=W^{l}_{i}=BA$. The matrices $A$ and $B$ start from an initialized state and are fine-tuned during adaptation. This approach maintains the general applicability of the model by fixing $W_{d}^{i}$, while also enabling specialization within specific sub-domains, as determined by Router Network.

In DyRoNet, we employ $r=32$ for the LoRA module, though this can be adjusted based on specific requirements of the scenarios in question. This low-rank adaptation mechanism not only enhances the flexibility of the DyRoNet framework but also mitigates the risk of overfitting, ensuring that each branch remains efficient and effective in its designated role.

The training process of DyRoNet focuses on two primary goals: (1) Improving the performance of individual branches. The backpropagation only updates the chosen model’s weights in this step, enabling fine-tuning on segregated samples. (2) Achieving an optimal balance between accuracy and computational efficiency. This step only train the speed router while the remaining branches are frozen. This dual-objective framework is represented by the overall loss function:

where $\mathcal{L}^{sp}$ represents the streaming perception loss, and $\mathcal{L}^{E^{2}}$ denotes the effective and efficient (E²) loss, which supervises branch selection.

The speed router is inspired by Network Architecture Search (NAS). However, it uniquely addresses the challenge of streaming perception by transforming the search problem into an optimization of coded distances. The formulation of the loss function, $\mathcal{L}^{E^{2}}$, involves converting $v^{\text{time}}$ into a distribution via softmax, which is then combined with $v^{\text{sp}}$ to determine the optimal model through an one-hot vector $f^{E^{2}}$. This approach effectively simplifies the intricate problem of balancing accuracy and latency into a more tractable optimization task. Instead of employing NAS for loss search, our design is intricately linked to the specific needs of streaming perception, with KL divergence selected for its robustness in noisy situations[16]. This demonstrates the efficiency and innovation of our approach.

Overall, the process of training DyRoNet involves striking a meticulous balance between the SP loss, which ensures the efficacy of each branch, and the E² loss, which optimizes efficiency. The primary objective of this training is to develop a model that not only delivers high accuracy in perception tasks but also operates within acceptable latency constraints, which is a critical requirement for real-time applications. This balanced approach enables DyRoNet to adapt dynamically to varying computational resources and environmental conditions, thereby maintaining optimal performance in diverse streaming perception scenarios.

We tested three state-of-the-art streaming perception models: StreamYOLO[35], LongShortNet[19], and DAMO-StreamNet[10]. These models, integral to the DyRoNet architecture, come with pre-trained parameters across three distinct scales: small (S), medium (M), and large (L), catering to a variety of processing requirements. In constructing the model bank $\mathcal{P}$ for DyRoNet, we strategically selected different model configurations to evaluate performance across diverse scenarios. For instance, the notation DyRoNet ($\text{DAMO}_{\text{S + M}}$) represents a configuration where DyRoNet employs the small (S) and medium (M) scales of DAMO-StreamNet as its two branches.³³3Similar notations are used for other model combinations, allowing for a systematic exploration of the framework’s adaptability and performance under varying computational constraints. All experiments were conducted on a high-performance computing platform equipped with Nvidia 3090Ti GPUs (x4), ensuring robust and reliable computational power to handle the intensive processing demands of the streaming perception models. This setup provided a consistent and controlled environment for evaluating the efficacy of DyRoNet across different model configurations, contributing to the thoroughness and validity of our results. For more implementation details, please refer to Appendix C.

We compared DyRoNet with state-of-the-art methods to evaluate its performance. In this subsection, we directly copied the reported sAP performance from their original papers as their results, for fair comparison, we evaluated the latency of each real-time model on 1x RTX 3090. The performance comparison was conducted on the Argoverse-HD dataset [20]. An overview of the results reveals that our proposed DyRoNet with a model bank of DAMO-StreamNet series achieves 37.8% sAP in 37.61 ms latency, outperforming the current state-of-the-art methods in latency by a significant margin. This demonstrates the effectiveness of the systematic improvements in DyRoNet.

In this subsection, we conducted detailed experiments analyzing the trade-offs between DyRoNet’s inference time and performance under different model bank selection. It is notable that the latency of DyRoNet presented in Tab. 2 do not accurately reflect the real outperform. Due to the varying hardware platforms for measuring latency across methods, a fair comparison cannot be achieved. For instance, DAMO-StreamNet is tested on 1x V100. To address these differences, we conducted additional tests on a 1x RTX 3090, which highlight DyRoNet’s performance enhancements. Tab. 1 presents the latency comparison and highlights DyRoNet’s superior performance—maintaining competitive inference speed alongside accuracy gains versus the random and MoE approaches. Where the MoE predicts the weights of each branch via a gate module and then combines the output results accordingly. Specifically, DyRoNet achieves efficient speeds while preserving or enhancing performance. This balance enables meeting real-time needs without compromising perception quality, critical for autonomous driving where both factors are paramount. By validating effectiveness in inference time reductions and accuracy improvements, the results show the practicality and efficiency of DyRoNet’s dynamic model selection.

It shows that using $\Delta I_{t}$ as input exhibits better performance than other methods (35.0 sAP of $\text{sYOLO}_{\text{S + L}}$ and 34.6 sAP of $\text{sYOLO}_{\text{M+L}}$). This indicates that utilizing $\Delta I_{t}$ offers significant advantages in comprehending and characterizing environmental speed. Conversely, it also underscores that employing single frames as input or using multiple frames as input renders the lightweight model bank selection model ineffective. Furthermore, the proportion of sample splits across branches can also illustrate the discriminative power with respect to environmental factors. For instance, the $\mathbb{E}(\Delta I_{t})$ criterion resulting in a evenly spliting distribution (48.22% $\mathbb{E}(\Delta I_{t})>0$ over train set and 49.85% $\mathbb{E}(\Delta I_{t})>0$ over test set). Indicating the direct sample selection without router lacks estimation of environmental factors, thereby weakening its discriminative power.

In contrast, Tab.5 presents statistics indicating the router layer’s effectiveness in allocating samples to specific models and showcase its ability to strike a balance between latency and performance. This balance is crucial for streaming perception and underscores our contribution.

To validate the DyRoNet on other dataset, we converted the 3D streaming perception dataset nuScenes-H[30] into 2D format. The experiment details are provided in the Appendix D. As shown in Tab. 8, DyRoNet consistently achieves better results than other branch fusion methods on nuScenes-H 2D dataset. It demonstrates DyRoNet’s advantages in branch fusion selection and its versatility.