Ekya: Continuous Learning of Video Analytics Models on Edge Compute Servers

Data drift: As specialized edge DNNs have fewer weights and shallower architectures than general DNNs, they can only memorize limited amount of object appearances, object classes, and scenes. As a result, specialized edge DNNs are particularly vulnerable to data drift (datadrift-7, ; datadrift-8, ; datadrift-a, ; datadrift-b, ), where live video data diverges significantly from the initial training data. For example, variations in the angles of objects, scene density (e.g. rush hours), and lighting (e.g., sunny vs. rainy days) over time make it difficult for traffic cameras to accurately identify the objects of interest (cars, bicycles, road signs). Cameras in modern cars observe vastly varying scenes (e.g., building colors, crowd sizes) as they move through different neighborhoods and cities. Further, the distribution of the objects change over time, which in turn, reduces the edge model’s accuracy (distribution-20, ; distribution-21, ). Owing to their ability to memorize limited amount of object variations, to maintain high accuracy, edge DNNs have to be continuously updated with the recent data and to the changing object distributions.

Continuous training: The preferred approach, that has gained significant attention, is for edge DNNs to continuously learn as they incrementally observe new samples over time (incremental-13, ; icarl-14, ; incremental-15, ). The high temporal locality of videos allows the edge DNNs to focus their learning on the most recent object appearances and object classes (DBLP:conf/cvpr/ShenHPK17, ; mullapudi2019, ). In Ekya, we use a modified version of iCaRL (icarl-14, ) though our techniques are generally applicable. Our learning algorithm on-boards new classes, as well as adapts to the changing characteristics of the existing classes. Since manual labeling is not feasible for continuous training systems on the edge, we obtain the labels using a “golden model” that is highly accurate but is far more expensive because it uses a deeper architecture with large number of weights. The golden model cannot keep up with inference on the live videos and we use it to label only a small fraction of the videos in the retraining window that we use for retraining. Our approach is essentially that of supervising a low-cost “student” model with a high-cost “teacher” model (or knowledge distillation (44873, )), and this has been broadly applied in computer vision literature (incremental-13, ; mullapudi2019, ; incremental-15, ; distribution-20, ).

Tradeoffs in retraining configurations. The hyperparameters for retraining, or “retraining configurations”, influence the resource demands and accuracy. Retraining fewer layers of the DNN (or, “freezing” more layers) consumes lesser GPU resources, as does training on fewer data samples, but they also produce a model with lower accuracy; Figure 3(a).

Tradeoffs in inference configurations. Inference pipelines also allow for flexibility in their resource demands at the cost of accuracy through configurations to downsize and sample frames (rocket-github, ). Prior work has made dramatic advancements in developing profilers to efficiently obtain the resource-accuracy relationship for inference configurations (chameleon, ). We use these efficient inference profilers in our joint solution for retraining and inference, and also to ensure that the inference pipelines continue to keep up with analyzing the live video streams with their currently allocated resources.

Uniform scheduling: Building upon prior work in cluster schedulers (fair-1, ; fair-2, ) and video analytics systems (videostorm, ), a baseline solution for resource allocation evenly splits the GPUs between video streams, and each stream evenly partitions its allocated GPUs for retraining and inference tasks; see Figure 4(a). Just like model training systems (vizier, ; hyperband, ; pbt, ), the baseline always picks the configuration for retraining that results in the highest accuracy (Cfg1A, Cfg1B for both windows).

Accuracy-optimized scheduling: Figures 4(b) and 4(d) illustrate an accuracy-optimized scheduler, which by taking a holistic view on the multi-dimensional tradeoffs, provides an an average inference accuracy of $73\%$. In fact, to match the accuracies, the above uniform scheduler would require nearly twice the GPUs (i.e., $6$ GPUs instead of $3$ GPUs). ³³3The techniques in our scheduler apply to other optimization metrics too, like max-min of accuracy. Evaluating other metrics is left to future work.

Decisions. For each video $v\in\mathcal{V}$ in a window $T$, we decide: (1) the retraining configuration $\gamma\in\Gamma$ ($\gamma=\emptyset$ means no retraining); (2) the inference configuration $\lambda\in\Lambda$; and (3) how many GPUs (in multiples of $\delta$) to allocate for retraining ($\mathcal{R}$) and inference ($\mathcal{I}$). We use binary variables $\phi_{v\gamma\lambda\mathcal{R}\mathcal{I}}\in\{0,1\}$ to denote these decisions (see Table 2 for the definition). These decisions require $C_{T}(v,\gamma,\lambda)$ GPU-time and yields overall accuracy of $A_{T}(v,\gamma,\lambda,\mathcal{R},\mathcal{I})$. As we saw in §3.2, $A_{T}(v,\gamma,\lambda,\mathcal{R},\mathcal{I})$ is averaged across the window $T$, and the inference accuracy at each point in time is determined by the above decisions.

Optimization Problem. The optimization problem maximizes inference accuracy averaged across all videos in a retraining window within the GPU resource limit.

Complexity Analysis. Assuming all the $A_{T}(v,\gamma,\lambda,\mathcal{R},\mathcal{I})$ values are known, the above optimization problem can be reduced to a multi-dimensional binary knapsack problem, a NP-hard problem (DBLP:journals/mor/MagazineC84, ). Specifically, the optimization problem is to pick binary options ($\phi_{v\gamma\lambda\mathcal{R}\mathcal{I}}$) to maximize overall accuracy while satisfying two capacity constraints (the first and second constraints in Eq 1). In practice, however, getting all the $A_{T}(v,\gamma,\lambda,\mathcal{R},\mathcal{I})$ is infeasible because this requires training the edge DNN using all retraining configurations and running inference using all the retrained DNNs with all possible GPU allocations and inference configurations.

Design rationale:  We call out the key aspects that makes the scheduler’s decision efficient by pruning the search space.

• $\bullet$

  Coarse allocations: The thief scheduler allocates GPU resources in quantums of $\Delta$. We empirically pick the value of $\Delta$ that is coarse and yet accurate enough for scheduling decisions, while also being mindful of the granularity achievable in modern GPUs (nvidia-mps, ). We analyze the sensitivity of $\Delta$ in §6.3. Resource stealing always ensures that the total allocation is within the limit (second constraint in Eq 1).

• $\bullet$

  Reallocating resources only when a retraining job completes: Although one can reallocate GPU resource among jobs at finer temporal granularity (e.g., whenever a retraining job has reached a high accuracy), we empirically find that the gains from such complexity is marginal. That said, Ekya periodically checkpoints the model (§5) so that inference can get the up-to-date accuracy from retraining.

• $\bullet$

  Pruned configuration list: Our micro-profiler (described next) speeds up the thief scheduler by giving it only the more promising configurations. Thus, the list $\Gamma$ used in Algorithm 1 is significantly smaller than the exhaustive set.

Profiling in Ekya vs. hyperparameter tuning:  While Ekya’s profiling may look similar to hyperparameter tuning (e.g., (DBLP:conf/nips/SnoekLA12, ; DBLP:journals/jmlr/LiJDRT17, )) at first blush, there are two key differences. First, Ekya needs the performance estimates of a broad set of candidate configurations for the thief scheduler, not just of the single best configuration, because the best configuration is jointly decided across the many retraining and inference jobs. Second, in contrast to hyperparameter tuning which runs separately of the eventual inference/training, Ekya’s profiling must share compute resource with all retraining and inference.

Opportunities:  Ekya leverages three empirical observations for efficient profiling of the retraining configurations. $(i)$ Resource demands of the configurations are deterministic. Hence, we measure the GPU-time taken to retrain for each epoch in the current retraining window when $100\%$ of the GPU is allocated to the retraining. This allows us to scale the time for varying number of epochs, GPU allocations, and training data sizes in Algorithm 1. $(ii)$ Post-retraining accuracy can be roughly estimated by training on a small subset of training data for a handful of epochs. $(iii)$ The thief scheduler’s decisions are not impacted by small errors in the estimations.

Micro-profiling design:  The above insights inspired our approach, called micro-profiling, where for each video, we test various retraining configurations, but on a small subset of the retraining data and only for a small number of epochs (well before models converge). Our micro-profiler is nearly $100\times$ more efficient than exhaustive profiling (of all configurations on the entire training data), while predicting accuracies with an error of $5.8\%$, which is low enough in practice to mostly ensure that the thief scheduler makes the same decisions as it would with a fully accurate prediction. We explain the techniques that make Ekya’s micro-profiling efficient.

1) Training data sampling: Ekya’s micro-profiling works on only a small fraction (say, $5\%-10\%$) of the training data in the retraining window (which is already a subset of all the videos accumulated in the retraining window). While we considered weighted sampling techniques for the micro-profiling, we find that uniform random sampling is the most indicative of the configuration’s performance on the full training data, since it preserves all the data distributions and variations.

2) Early termination: Similar to data sampling, Ekya’s micro-profiling only tests each configuration for a small number (say, 5) of training epochs. Compared to a full fledged profiling that needs few tens of epochs to converge, such early termination greatly speeds up the micro-profiling process.

3) Pruning out bad configurations: Finally, Ekya’s micro-profiling also prunes out those configurations for micro-profiling (and hence, for retraining) that have historically not been useful. These are configurations that are usually significantly distant from the configurations on the Pareto curve of the resource-accuracy profile (see Figure 3(b)), and thus unlikely to be picked by the thief scheduler. Avoiding these configurations improves the efficiency of the micro-profiling.

Annotating training data:  For both the micro-profiling as well as the retraining, Ekya acquires labels using a “golden model” (§2.2). This is a high-cost but high-accuracy model trained on a large dataset. As explained in §2, the golden model cannot keep up with inference on the live videos and we use it to label only a small subset of the videos for retraining.

Datasets:  We use both on-road videos captured by dashboard cameras as well as urban videos captured by mounted cameras. The dashboard camera videos are from cars driving through cities in the US and Europe, Waymo Open (waymo, ) (1000 video segments with in total 200K frames) and Cityscapes (cityscapes, ) (5K frames captured by 27 cameras) videos. The urban videos are from stationary cameras mounted in a building (“Urban Building”) as well as from five traffic intersections (“Urban Traffic”), both collected over 24-hour durations. We use a retraining window of 200 seconds in our experiments, and split each of the videos into 200 second segments. Since the Waymo and Cityscapes dataset do not contain continuous timestamps, we create retraining windows by concatenating images from the same camera in chronological order to form a long video stream and split it into 200 second segments.

DNN models.:  We use the ResNet18 object classifier model as our edge DNN. As explained in §2.2, we use an expensive golden model (ResNeXt 101 (wang2019elastic, )) to get ground truth labels for training and testing. On a subset of data that have human annotations, we confirm that the labels produced by the golden model are very similar to human-annotated labels.

Testbed and trace-driven simulator:  We run Ekya’s implementation (§5) on AWS EC2 p3.2xlarge instances for 1 GPU experiments and p3.8xlarge instances for 2 GPU experiments. Each instance has Nvidia V100 GPUs with NVLink interconnects and Intel Skylake Xeon processors.

Retraining configurations:  As listed in §3.1, our retraining configurations are obtained by combining these hyperparameters: number of epochs to train, batch size, number of neurons in the last layer, number of layers to retrain, and the fraction of data between retraining windows to use for retraining.

Baselines:  Our baseline, called uniform scheduler, uses $(a)$ a fixed retraining configuration, and $(b)$ a static retraining/inference resource allocation (these are adopted by prior schedulers (fair-1, ; fair-2, ; videostorm, )). For each dataset, we test all retraining configurations on a hold-out dataset ⁴⁴4 The same hold-out dataset is used to customize the off-the-shelf DNN inference model. This is a common strategy in prior work (e.g., (noscope, )). (i.e., two video streams that were never used in later tests) to produce the Pareto frontier of the accuracy-resource tradeoffs (e.g., Figure 3). The uniform scheduler then picks two points on the Pareto frontier as the fixed retraining configurations to represent “high” (Config 1) and “low” (Config 2) resource usage, and uses one of them for all retraining windows in a test.

Accuracy vs. Number of concurrent video streams:  Figure 6 shows that the accuracy of Ekya and the uniform baselines when analyzing a growing number of concurrent video streams under a fixed number of provisioned GPUs for Waymo and Cityscapes datasets. The uniform baselines use different combinations of pre-determined retraining configurations and resource partitionings. As the number of video streams increases, Ekya enjoys a growing advantage (upto 29% under 1 GPU and 23% under 2 GPU) in accuracy over the uniform baselines. This is because Ekya gradually shifts more resource from retraining to inference and uses cheaper retraining configurations. In contrast, increasing the number of streams forces the uniform baseline to allocate less GPU cycles to each inference job, while retraining jobs, which use fixed configurations, slow down and take the bulk of each window. This trend persists with different GPUs.

Number of video streams vs. provisioned resource:  We compare Ekya’s capacity (defined by the maximum number of concurrent video streams subject to an accuracy threshold) with that of uniform baseline, as more GPUs are available. Setting an accuracy threshold is common in practice, since applications usually require accuracy to be above a threshold for the inference to be usable. Table 3 uses the Cityscapes results (Figure 6) to derive the scaling factor of capacity vs. the number of provisioned GPUs and shows that with more provisioned GPUs, Ekya scales faster than uniform baselines.

Accuracy vs. provisioned resource:  Finally, Figure 7 stress-tests Ekya and the uniform baselines to process 10 concurrent video streams and shows their average inference accuracy under different number of GPUs. To scale to more GPUs, we use the simulator (§6.1), which uses profiles recorded from real tests and we verified that it produced similar results as the implementation at small-scale. As we increase the number of provisioned GPUs, we see that Ekya consistently outperforms the best of the two baselines by a considerable margin and more importantly, with 4 GPUs Ekya achieves higher accuracy (marked with the dotted horizontal line) than the baselines at 16 GPUs (i.e., 4$\times$ resource saving).

Summary:  The results highlight two advantages of Ekya. First, it allocates resources to retraining only when the accuracy gain from the retraining outweighs the temporary inference accuracy drop due to frame subsampling. Second, when it allocates resource to retraining, it retrains the model with a configuration that can finish in time for the inference to leverage the higher accuracy from the retrained model.

Component-wise contribution:  Figure 8 understands the contributions of resource allocation and configuration selection (on 10 video streams with 4 GPUs provisioned). We construct two variants from Ekya: Ekya-FixedRes, which removes the smart resource allocation in Ekya (i.e., using the inference/training resource partition of the uniform baseline), and Ekya-FixedConfig removes the microprofiling-based configuration selection in Ekya (i.e., using the fixed configuration of the uniform baseline). Figure 8 shows that both adaptive resource allocation and configuration selection has a substantial contribution to Ekya’s gains in accuracy, especially when the system is under stress (i.e., fewer resources are provisioned).

Resource allocation across streams:  Figure 9 shows Ekya’s resource allocation across two example video streams over several retraining windows. In contrast to the uniform baselines that use the same retraining configuration and allocate equal resource to retraining and inference (when retraining takes place), Ekya retrains the model only when it benefits and allocates different amounts of GPUs to the retraining jobs of video streams, depending on how much accuracy gain is expected from retraining on each stream. In this case, more resource is diverted to video stream #1 (#1 can benefit more from retraining than #2) and both video streams achieve much higher accuracies (0.82 and 0.83) than the uniform baseline.

Impact of scheduling granularity:  A key parameter in Ekya’s scheduling algorithm (§4.2) is the allocation quantum $\Delta$: it controls the runtime of the scheduling algorithm and the granularity of resource allocation. Figure 10 plots this tradeoff with the same setting as Figure 7 (10 video streams). While increasing $\Delta$ from $1.0$ (coarse-grained; one full GPU) to $0.1$ (fine-grained; fraction of a GPU), we see the accuracy increases substantially $\sim 8\%$. Though the runtime also increases to 9.5 seconds, it is still a tiny fraction ($4.7\%$) of the retraining window ($200$s), and we use $\Delta=0.1$ in our experiments.

Errors of microprofiled accuracy estimates:  Ekya’s micro-profiler estimates the accuracy of each configuration (§4.3) by training it on a subset of the data for a small number of epochs. To evaluate the micro-profiler’s estimates, we run it on all configurations for 5 epochs and on 10% of the retraining data from all streams of the Cityscapes dataset, and calculate the estimation error with respect to the retrained accuracies when trained on 100% of the data for 5, 15 and 30 epochs. Figure 11(a) plots the distribution of the errors in accuracy estimation and and show that the micro-profiled estimates are largely unbiased with an median absolute error of 5.8%.

Sensitivity to microprofiling estimation errors:  Finally, we test the impact of accuracy estimation errors (§4.3) on Ekya’s inference accuracy. We add a controlled Gaussian noise on top of the real retraining accuracy as the predictions when the microprofiler is queried. Figure 11(b) shows that Ekya is quite robust to accuracy estimate errors: with upto 20% errors (in which all errors in Figure 11(a) lie) in the profiler prediction, the maximum accuracy drop is merely 3%.

Ekya vs. Cloud-based retraining:  One may upload a sub-sampled video stream to the cloud, retrain the model in the cloud, and download the model to the edge server (khani2020real, ). While this solution compromises the privacy of the videos and is not an option for many deployments due to legal stipulations (sweden-data, ; azure-data, ), we nonetheless evaluate this option as it lets the edge servers focus on inference. We conclude that the cloud-based solution results in lower accuracy due to significant network delays on the constrained networks typical of edges (getmobile, ).

Ekya vs. Re-using pretrained models:  Another alternative to Ekya’s continuous retraining is to cache retrained models and reuse them, e.g., pick the model that was trained on a similar class distribution. To test this baseline, we pre-train and cache a few tens of DNNs from earlier retraining windows from the Cityscapes dataset. In each retraining window of our experiment with 8 GPUs and 10 video streams, we pick the cached DNN whose class distribution (vector of object class frequencies) of its training data has the closest Euclidean distance with the current window’s data. GPU cycles are evenly shared by the inference jobs (since there is no retraining). The resulting average inference accuracy is 0.72 is lower than the Ekya’s accuracy of 0.78 (see Figure 7(a)). This is because even though the class distributions may be similar, the models cannot be directly reused from any window as the appearances of objects may still differ considerably (Figure 2).