Vision Transformer Computation and Resilience
for Dynamic Inference ^††thanks: ¹Kavya Sreedhar was supported by an internship at NVIDIA and by a graduate fellowship award as a Knight-Hennessy Scholar at Stanford University.

RDD inference targets a computational platform with a finite amount of resources [47, 48]. Occasionally, there are not enough resources available for the full model execution, since there are other tasks that also need to be completed. In these scenarios, it is better to perform a degraded version of inference that requires less resources rather than to skip a frame and perform no inference. While statically selecting a model that matches the worst-case resource availability will ensure that no frames are missed, it does not allow for performing the full model execution and returning more accurate results when enough resources are available. Thus, the goal of RDD inference is to adjust the cost of inference to match the resources available for the computation, and thus maximize accuracy given input resource constraints.

We consider semantic segmentation [71] and object detection [72] since these tasks are widely used [73, 44, 42, 43] and more complicated than image classification. Semantic segmentation assigns class labels to each pixel in an input image, while object detection identifies objects by providing bounding boxes and class labels.

State-of-the-art models for these tasks have shifted to using transformer-based architectures, building on the success of using transformers for natural language processing [34, 35] and image classification [20, 21, 24]. Most segmentation models and some detection models use the transformer encoder as a backbone for feature extraction, pairing it with different decoders and task-specific heads to produce the final output [77, 18, 7, 31, 32, 33]. Other detection models models employ a transformer encoder-decoder structure, often with a CNN backbone to extract visual features [13, 14, 16, 15, 25, 31, 32, 33]. Since we analyze the first type of models with our segmentation model case studies, we focus on the second type of models for our detection model case studies.

Following from ViT, the first vision transformer, models reshape 2D images into a 1D sequence of flattened image patches, which are linearly projected into an embedding dimension [20]. Throughout the model, layers progressively increase the embedding dimension and reduce the spatial dimensions to capture details at various image resolutions for higher accuracy. ViT is convolution-free since the original transformer layers consist of multi-head self-attention layers and multi-layer perceptrons (MLPs) [34]. However, many modern vision transformers have incorporated convolutions with the transformer architecture for better accuracy results [78, 7, 17, 18, 13, 14, 16, 15, 25, 22].

We use NVIDIA’s SegFormer [17] and Microsoft’s Swin Transformer [18] as case studies for semantic segmentation since they achieve state-of-the-art accuracy and are commonly used. Figure 2 shows these model architectures. These models build from the encoder in ViT, and further optimize attention and integrate convolutions to preserve local continuity information. Both models have four encoder stages followed by a decoder. SegFormer B2 has three, four, six, and three encoder blocks in stages zero through three, respectively. Swin Tiny has six encoder blocks in stage two and two encoder blocks in all the other encoder stages. Individual layers inside these blocks are shown in Figure 2, with breakout boxes showing the individual layers in the MLP and attention components.

Prior work has focused on improving encoder backbones [77, 18] that can be used with different decoders and task-specific heads for various vision tasks. In these backbones, attention dominates FLOPs [18, 17, 20, 21, 24], so prior work has heavily focused on reducing the cost of attention [79, 80, 81, 65]. SegFormer contributes a simpler attention-free decoder, while Swin directly uses the UPerNet decoder head [65]. Both of these decoders use convolutions to extract and fuse features from different spatial resolutions and reduce channel dimensions, which results in higher accuracy.

For object detection, many models build from the DETR architecture [13]. DETR uses a CNN backbone to extract image features, a conventional transformer, and a feed forward network to return the detection prediction. We examine DETR and three recent models based on DETR: DAB DETR [14], Anchor DETR [15], and Conditional DETR [16]. These models improve aspects of the transformer in DETR. We consider the base variants of these models, as included in detrex, an open-source toolbox for detection models [82]. The base versions of these models use ResNet-50 [9] as the CNN backbone.

Prior work has also explored the resiliency of ResNet-50 [64, 61]. One example is OFA [64], which trains the model once to produce many sets of weights for different model subsets, resulting in models that significantly reduce FLOPs with modest decreases in accuracy. In Section V-C, we leverage switching between OFA ResNet-50 models to enable RDD inference. We denote the most accurate OFA ResNet-50 model as OFA-ResNet-50.

Finally, we use standard accuracy metrics in our analysis: mean intersection over union (mIoU) for semantic segmentation and average precision (AP) for object detection. IoU is the area of overlap between the predicted segmentation and the ground truth divided by their total area. Thus, mIoU is the average of the IoU for every class. AP measures the precision-recall curve across different IoU confidence thresholds, which specify the minimum IoU to consider a positive match. For the COCO dataset [76], AP is the average precision for IoU from 0.5 to 0.95, in increments of 0.05. Both mIoU and AP range from zero to one, and higher values indicate better accuracy.

We show the distribution of FLOPs across model layers for SegFormer B2 and Swin-Tiny with a 512 by 512 image size in Figure 3. We find that 68% and 89% of the total FLOPs are in convolution layers in SegFormer B2 and Swin-Tiny, respectively, in stark contrast to the zero convolutions in ViT [20] and BERT [35]. Relatively large convolutional layers are labeled in the FLOPs distributions in Figure 3. While encoders use convolutions to overlap image patches to incorporate local continuity information and reduce the size of inputs for attention, only 5% of convolution FLOPs are in the SegFormer encoder and 1% are in Swin encoder. Instead, as shown in Figure 2, decoders replace attention with large convolutions to fuse information at different image resolutions from the output of each encoder stage. We denote these convolutions as Conv2DFuse in SegFormer and fpn_bottleneck_conv in Swin.

Looking at SegFormer’s decoder, where nearly 70% of the FLOPs are used, the Conv2DFuse, Conv2DPred, and Decoder Linear connected to output of encoder stage zero (denoted as DecodeLinear0) layers individually comprise $62\%$, $3\%$, and $1.3\%$ of the total number of FLOPs for a 512 by 512 image, respectively, as shown in Figure 3. In particular, Conv2D Fuse alone constitutes the majority of FLOPs, considerably more so than any other layer in the model. The Conv2D Fuse layer has 3072 input channels, 768 output channels, and a 1 by 1 stencil.

When considering the Swin Tiny model for segmentation, we find that 89% of FLOPs are in the decoder, and 99% of convolutions are in this decoder. The fpn_bottleneck_Conv2D layer alone comprises 65% of FLOPs. This layer has 2048 input channels, 512 output channels, and a 3 by 3 stencil.

From these results, it follows that these segmentation models have high operational intensity, at 130+ operations/byte, and that they are not memory bound like traditional transformers for language [35] and classification [20, 21] tasks. We also see that the task-specific decoder heads are computationally dominant for transformer models for segmentation, and critical to include in profiling analyses for this task.

Figure 1 shows the FLOPs distribution between convolution and non-convolution layers as well as the FLOPs distribution between the ResNet-50 backbone and transformer for DETR-based models across various image sizes. Note that the percentage of FLOPs in convolutions and in ResNet-50 are very similar, indicating that the convolutions in DETR-based models are predominantly in the ResNet-50 backbone (and that the contribution of non-convolutional layers to the ResNet-50 backbone FLOPs is negligible as expected). Solely focusing on smaller image sizes below 64K pixels would lead one to conclude that improving the transformer is important for overall model efficiency. However, for object detection, we typically need to consider much larger image sizes. The importance of the ResNet-50 backbone compared to the transformer mostly increases with larger image sizes: for more than 128K image pixels, the ResNet-50 backbone comprises at least half of the total FLOPs in all of these models, and for more than 1M pixels, which corresponds to standard image sizes for detection datasets [76, 75], this backbone requires 80+% of FLOPs.

We observe that the number of FLOPs in all of these models does not directly map to GPU execution time, as shown in Figure 4 for the segmentation models and Figure 1 for the detection models. For 512 by 512 image sizes, convolutions comprise 28% and 42% of the total execution time for SegFormer B2 and Swin Tiny, respectively, despite requiring 68% and 89% of the FLOPs. With the DETR-based models, 30% to 40% of the total execution time for larger image sizes is in convolutions, which make up 80+% of the total FLOPs. For all models, the importance of convolutions generally increases with larger image sizes, but the proportion of GPU runtime spent in convolutions is much lower than the relative FLOP counts. These results show that the GPU implementation is well-designed for reusing convolution weights and exploiting convolution parallelism. Thus, integrating convolutions results in better model accuracy and performance.

Figure 4 shows that convolutions comprise a smaller portion of FLOPs and GPU performance in larger Swin models compared to Swin Tiny. This result makes sense because all models have a similar decoder, which majorly comprises of convolutions, but Swin Small and Swin Base have two times the number of encoder blocks compared to Swin Tiny. We also observe that matrix multiplications comprise about an equal proportion of execution time compared to convolutions on the GPU for SegFormer and Swin for larger image sizes commonly used for segmentation.

Thus, while many prior papers use FLOPs to estimate the runtime savings for inference [60, 49, 50, 51, 52, 53, 54], we find that FLOPs alone is not a good predictor of runtime on a GPU. GPU runtime is instead a function of FLOPs, layer parallelism, and layer operational intensity. Convolutions enable spatial parallelism since each output pixel depends on a relatively small receptive field in the input image. As a result, these layers can be efficiently executed on the highly parallel architecture in a GPU. Attention can be computed in parallel for several image patches, enabling temporal parallelism, and matrix multiplication can be easily parallelized. However, attention mechanisms also involve sequential operations where these patches attend to each other, making the overall operation less parallelizable. In addition, attention layers in these vision transformers have lower operational intensity compared to convolutions, resulting in more memory accesses. Thus, it is important for inference engines for vision transformers to efficiently execute CNNs and attention.

We use MAGNet [69], a CNN accelerator framework, which has been previously extended to execute attention-dominated transformers for language and classification tasks as well [70]. As a result, its parameterizable accelerator template provides a reasonable architecture for profiling our applications on a hardware accelerator. Most other accelerators for transformers focus on attention and custom detection components [79, 80, 81, 84], often without support for convolutions, and are thus not well-optimized for more recent vision transformers.

Figure 5 shows a block diagram of the MAGNet accelerator and PE architecture templates. The accelerator has three levels of compute with vector multiply-accumulate (MAC) units, processing elements (PEs), and the PE array. All levels allow for parallel execution in order to exploit data parallelism present in model layers: each vector MAC unit can execute $C0$ multiplications in parallel, every PE has $K0$ parallel vector MAC units, and the PE array has $NumPE$ PEs. Each PE also contains a post-processing unit to fuse activation and pooling layers with the preceding convolutions.

The memory hierarchy has four levels: small register files within each vector MAC unit (labeled as “RegFile” in Figure 5), local buffers within each PE, a global buffer at the PE array level, and an off-chip DRAM. Inputs are shared across the vector MAC units with one input buffer per PE. Each vector MAC unit has its own weight buffer, with $K0$ total weight buffers in every PE. We can temporally tile the weights (split by output channels) and activations (split by image height and width). The paper on the MAGNet framework provides further detail on how convolutions and matrix multiplications can be mapped to this accelerator template [69].

MAGNet uses Mentor Graphics Catapult HLS to generate RTL from a synthesizable SystemC and C++ architecture description. We report results in a 5nm technology, using Synopsys Design Compiler for synthesis for execution time and area results and Primetime-PX with Gaussian synthetic data for power results.

Since prior work has shown that dataflow choice minimally affects energy efficiency [85], we directly use an output-stationary local-weight-stationary dataflow that was previously shown to work well for CNNs [69] and attention layers [70]. Like these prior works, we also use 8-bit precision for data.

To examine throughput, area, and energy tradeoffs for vision transformer applications, we compare different accelerator template parameters, as shown in Table II. Accelerators C through M can compute the same number of MACs in parallel, while accelerators A and B have twice the compute capability. To fairly compare the execution of this class of models on these parallel, throughput-focused compute engines, we compute the energy per operation, which to first order is independent of parallelism, and throughput/mm², to normalize throughput by the silicon cost of its hardware. As a result, accelerator B has twice the available compute compared to accelerator D and is twice as fast, but it looks identical to accelerator D with our metrics since it is also twice as big.

Figure 6 explores the tradeoff between these metrics for SegFormer ADE B2, where lower energy per operation and higher throughput per area signify more efficient inference. We also checked Swin Tiny and OFA-ResNet-50, which have similar results compared to SegFormer. All of these models can significantly reuse data across the channel dimension. As a result, accelerators with reduced vectorization ($K0=C0=16$) result in lower efficiency (1.4$\times$ more energy per FLOP and 2.8$\times$ more area per FLOP) compared to those with $K0=C0=32$. Since the accelerators with $K0=C0=16$ are not competitive with either metric, they are not shown in Figure 6.

Swin Tiny has significantly more model layers with an odd number of input and/or output channels (such as 49), due to default layer parameters or how matrix multiplications are mapped to the hardware. These channel counts are not divisible by $K0$ and $C0$ whether they are equal to 16 or 32, resulting in similar underutilization in the PEs for all the accelerator parameterizations, and similar performance across accelerators. In contrast, SegFormer and OFA-ResNet-50 have more evenly divisible channel values, resulting in slightly higher utilization and 10% faster execution with accelerators with $K0=C0=16$ compared to those with $K0=C0=32$.

For all of these models, accelerators D, E, and G are Pareto-optimal with our metrics. We expect this trend to continue for larger models, including CNN backbones such as ResNet-101 for DETR-based models. Note that accelerators E and G are the two smallest accelerators in Table II as well. Not surprisingly, these accelerators are similar to accelerator D, which was the design previously used for transformer models dominated by attention [70].

In Section III, we saw that these vision transformers have higher operational intensity than attention-dominated transformers, due to the increased locality of the convolution operation. Thus, it makes sense that the smaller memories in accelerators E and G result in higher throughput per area compared to accelerator D. We find that 64 to 128 B/MAC and 32 to 64 B/MAC are sweet spots for the weight and input buffers, respectively, to balance throughput and area tradeoffs for these applications.

To balance energy and area constraints, we profile our model case studies on accelerator E, which is between accelerators D and G on the Pareto curve in Figure 6. The synthesized clock frequency for accelerator E is 1.25 GHz.

The SegFormer ADE B2 model runs in 3.6ms on accelerator E. Figure 7 shows the execution time and energy breakdowns across model layers for inference on accelerator E. Convolutions constitute 74% of the total execution time and energy. The largest layer, Conv2DFuse, alone requires over half of the total execution time and energy on accelerator E.

Five convolution layers in the encoder have higher energy per FLOP compared to other layers, as indicated in Figure 8. Fortunately, these layers comprise only 14% of the total energy. These layers have a small number of input channels: the Overlap Patch Embedding Conv2D layer has three input channels for an input image to the model, and the MLP DW Conv layers each have one input channel due to how we exploit parallelism in mappings for depthwise convolutions. This results in hardware underutilization, since all 32 parallel multiplications in a vector MAC unit are not utilized as often as in other layers. Thus, convolutions comprise a slightly greater portion of accelerator time and energy (74%) compared to model FLOPs (68%). Note that Conv2DFuse, with 3072 input channels, more fully utilizes available parallelism in the vector MAC units and thus has relatively low energy per FLOP.

The Swin Tiny model runs in 12ms on accelerator E. The Swin Tiny execution time and energy distributions on accelerator E, shown in Figure 9, closely match the model FLOPs distribution: 87% of the execution time and energy on E are in convolutions, compared to the 89% of FLOPs in convolutions. The distribution of time and energy on accelerator E across individual layers also match the distribution of FLOPs across model layers. For example, the largest layer, fpn_bottleneck_conv, comprises 63% of the total execution time and energy on accelerator E, and 65% of model FLOPs.

Swin Tiny layers with relatively higher energy per FLOP are either convolutions in the decoder with a low number of input channels, due to how we map this computation to maximize the available parallelism, or matrix multiplications in attention blocks, due to odd numbers of input channels that do not fully utilize available parallelism, leading to lower utilization.

With OFA-ResNet-50, the execution time and energy distributions for inference on accelerator E are mostly evenly split among all the convolutions in the CNN. Two layers have significantly higher energy per FLOP compared to the remaining layers. These layers are the first layer (the input layer) and the last layer (the classifier) in the model. The first layer has three input channels for an input image to the model, while the last layer has one input channel for a linear layer. Thus, as with the other models, these layers with a small number of input channels have lower utilization in the vector MAC units.

We examine the resilience of SegFormer B2 with different image sizes, using models trained and validated with 512 by 512 images in the ADE20K dataset [74] (denoted “SegFormer ADE B2”) and with 1024 by 1024 images in the Cityscapes dataset [75] (denoted “SegFormer City B2”). Since the majority of FLOPs are in the decoder for both models, we find that the model accuracy is not resilient to reducing scaling factors and input channels in individual layers in the encoder, making these modified execution paths ill-suited for RDD inference.

Next, we examine the impact of reducing the number of input and output channels for the top three layers in the FLOPs distribution in Figure 3: Conv2DFuse, Conv2DPred, and DecodeLinear0. Reducing the number of input channels to DecodeLinear0 does not enable skipping computation in earlier layers since the full output of encoder stage zero must still be computed as input for encoder stage one. Similarly, since the interpolation of the output of the Linear layers after the encoder stages is reversed and then concatenated, reducing input channels from 3072 to Conv2DFuse only allows for skipped computation to propagate backwards if the model execution path uses less than or equal to 768 input channels (corresponding to the contribution from encoder stage three, which is not input to another encoder stage and only input to Conv2DFuse). Since the decoder layer outputs have only one following destination, skipped computation associated with input channels to Conv2DPred can be propagated backwards through preceding decoder layers (such as ReLU) to further reduce FLOPs.

Given a constant number of bypassed channels $x$, we found that which channels that were removed (e.g., the first $x$ versus the last $x$ versus the smallest $x$) did not substantially change the resulting accuracy, so we bypass the last $x$ in our experiments. As more channels are removed from the inputs to Conv2DFuse, Conv2DPred, and DecodeLinear0, the marginal accuracy drop increases. However, if we additionally bypass full encoder blocks, we can shift these trade-off curves more to the left than down (saving relatively more execution time compared to the relative accuracy drop), and produce new Pareto-optimal execution paths. Thus, Figure 10 shows the piecewise concatenation of the Pareto-optimal points on these shifted curves for SegFormer ADE B2 and SegFormer City B2 on the A5000 GPU.

We find that combinations of varying the number of encoder blocks in each stage and input channels to Conv2DFuse and Conv2DPred result in Pareto-optimal points, as shown with the “ADE Time (Pretrained B2)” and “City Time (Pretrained B2)” lines. We generate these two lines by pruning the pretrained SegFormer B2 model weights, with no additional training. Figure 10 denotes the model parameters that two example Pareto-optimal configurations $a$ and $b$ correspond to, along with the original B2 model configuration. Surprisingly, model configuration $a$ achieves slightly better mIoU (0.4655 without any retraining and 0.4698 after training) than the original SegFormer B2 ADE model (0.4651). This point has 32 fewer input channels to Conv2DPred compared to the B2 model and is 6% faster on the GPU, so we can start the ADE Pareto Curve from $a$ instead of the full SegFormer ADE B2 model square.

On the GPU, we can save 11% of execution time with a 1.9% accuracy loss for SegFormer ADE B2, without any retraining. The pretrained SegFormer City B2 model is more resilient: we can save 11% of execution time with only a 0.9% accuracy loss, without any retraining. SegFormer City B2 is trained and executed on larger image sizes (1024 by 1024 vs 512 by 512), so it achieves 1.74$\times$ higher mIoU than SegFormer ADE B2 to begin with. Thus, there is likely more redundancy in these model weights, enabling a flatter tradeoff compared to SegFormer ADE B2.

We plot the original SegFormer B0, B1, and B2 models, labeled as “ADE Time (Retrained)” and “City Time (Retrained)” in Figure 10, when executed on the GPU. We can switch between SegFormer models with retrained weights to save 51% of execution time with a 4.3% accuracy loss for the ADE B2 model and 45% of execution time with a 2.5% accuracy loss for the City B2 model. As expected, switching between pruned models that have been retrained offers a better tradeoff, but the pretrained models are surprisingly resilient: pruning without retraining is competitive up until reducing 15% of the execution time for the B2 models on the GPU.

Figure 11 shows the execution time, energy, and accuracy tradeoff for the various model configurations listed in Table III, when executed on accelerator E. These points are a subset of the Pareto-optimal configurations found from the GPU experiments. With specialized hardware, we can save 18% of execution time and 28% of energy with a 1.4% accuracy loss for SegFormer ADE B2 without any additional training.

We also plot the original SegFormer B1 and B2 models, labeled as “ADE Time (Retrained)” and “ADE Energy (Retrained)” in Figure 11, and see that with retraining, we can save 55% of execution time and energy for a 4.3% accuracy loss. On accelerator E, pruning without retraining reduces execution time and energy by 45% with the same accuracy loss. Thus, switching between retrained models offers an additional 10% of execution time and energy savings at the same accuracy for the SegFormer ADE model. One of these approaches can be chosen over the other depending on the range of dynamic resource constraints expected in a real-time system.

In Swin-Tiny, almost 90% of the FLOPs are in convolutions in the decoder. As a result, we vary the input and output channels in these layers along with bypassing encoder blocks, in a similar approach to our SegFormer experiments. Skipping even a few encoder blocks in Swin Tiny leads to a higher drop in model accuracy compared to execution time on the GPU and accelerator E, as shown in Figure 12, and the marginal accuracy loss quickly outpaces the marginal savings in execution time and energy.

Swin Small and Swin Base, with $1.35\times$ and $2\times$ as many parameters as Swin Tiny, are slightly more resilient to these optimizations. While all three models have the same number of input channels to the fpn_bottleneck_Conv2D layer, Swin Small and Base have 18 blocks in encoder stage two compared to only six in Swin Tiny. Thus, we can bypass stage two encoder blocks to achieve a better tradeoff. However, the resilience to pruning is small even for these larger models.

While the tradeoff curve is flatter with accelerator E than with the GPU, we can only save 8% of execution time and energy with a 2% accuracy loss for Swin Tiny inference on the accelerator, without any retraining. Thus, unlike with SegFormer, we recommend directly switching between retrained Swin model weights for RDD inference. On the GPU, this approach saves 36% of execution time for a 3.6% accuracy drop when switching from Swin Base to Swin Tiny.

While the Swin models require 3.8$\times$ to 4.7$\times$ more FLOPs than SegFormer ADE B2, these larger pretrained models are not more resilient to pruning. The result is because most of the extra FLOPs in Swin are from additional convolutions in the decoder and Swin Tiny has 25% fewer encoder blocks compared to SegFormer. Thus, while we could bypass encoder blocks to generate a flatter tradeoff curve with SegFormer, the Swin encoder has less redundant information, and the Swin model accuracy is more sensitive to skipping attention layers in the encoder.

Since the ResNet-50 backbone dominates FLOPs for the DETR-based models, we need to scale the work done in this CNN for RDD inference. Fortunately, this is a well-studied area [64, 61, 47], and we leverage switching between OFA ResNet-50 models, which scale model parameters such as the number of layers and convolution kernel sizes. The advantage of the OFA approach is that many model weight subsets are generated within a single training [64].

We execute various OFA models for ResNet-50 on accelerator E. Figure 13 shows the effectiveness of switching between these models for dynamic inference on accelerator E: we can save 58% of the execution time and 53% of energy with a 3.3% loss in accuracy. Switching between these models produces a better tradeoff compared to switching between retrained segmentation models on accelerator E, which have comparable execution time and energy savings with a 9% accuracy loss. This result illustrates that CNNs are more resilient to pruning compared to these segmentation models with the transformer architecture.

Our results in Sections V-A to V-C highlight general principles to help focus the search for competitive alternative execution paths in vision transformer models. First, we identify that these model executions greatly depend on convolutions, and that we should target convolutional paths in these models instead of pruning attention heads and attention layers as in prior work. Furthermore, our experiments in this section show that convolution weights are less critical to model accuracy compared to attention weights. Second, for modern encoder-decoder architectures, it is advantageous to prune computation in the decoder instead of the encoder. In these attention-free decoders, layers sequentially feed into each other and have one destination, while most encoder layer outputs have two destinations (to the following encoder layers and the decoder). Thus, skipped computation in the decoder can often be propagated backwards through decoder layers to further reduce FLOPs as exemplified with SegFormer in Section V-A, while this can rarely be done for skipped computation in the encoder. Third, the distribution of FLOPs between the encoder and decoder guides which alternative execution paths will be competitive. When more FLOPs are in the decoder, there is less redundancy in encoder weights, and bypassing encoder layers will lead to higher accuracy losses more quickly, as seen in Section V-B with pruning Swin Transformer, where 89% of the FLOPs are in the decoder.

In this section, we have identified alternative model execution paths that bypass computation to meet specific resource targets for our model case studies. During runtime, the current resource availability is input to the inference engine, which can select the model execution path that matches the resource constraints for that particular inference. Note that this selection is independent of the input image. Each inference task can use a different model configuration that is looked up in real time, as opposed to a pre-selection of a static model to always execute. When enough resources are available, the original model can be executed without any modifications. This means that the average accuracy loss will usually be lower than the percentages reported for particular model configurations in this section, since RDD inference is usually using the full model accuracy.

For more resilient models like SegFormer, we only need to train the original model once and store one set of model weights. During inference, we can use the subset of the weights required for model execution. For other models, we can switch between multiple sets of model weights stored in off-chip DRAM for execution on an accelerator. The OFA approach still requires only one training for this approach with ResNet-50, while switching between the Swin models requires multiple trainings for each model configuration.

Note that analyzing the resilience of pretrained models provides a floor on the accuracy attainable given a model architecture and a model configuration, while switching between retrained models provides a ceiling. Training techniques such as OFA can be used to improve model accuracy from the floor with less training overheads for RDD inference.

Finally, since we bypass some computation in critical layers for RDD inference, the relative importance of these layers decreases for model configurations with less computation. For example, configuration $B2\_f$ in Figure 11 requires 60% fewer FLOPs than the full SegFormer ADE B2 model, and Conv2DFuse requires less than 25% of the total FLOPs for this configuration. While the set of critical layers change across model configurations, these layers are still convolutions. For example, 55% of the total execution time and energy on accelerator E is spent in convolutions for the smaller $B2\_f$ configuration. Thus, we find that accelerators D, E, G are still the best at balancing area, energy, and throughput tradeoffs across all these dynamic model configurations for RDD inference.