Automatic Configuration for Optimal Communication Scheduling in DNN Training

For a deep learning task, a DNN model can be trained through an iterative way. The iterations are repeated over the large training dataset until the loss function is minimized and the model converges to an acceptable prediction accuracy [11].

In the parameter update step, devices need to share their gradients, which involves network communication issues. Two kinds of communication architectures are widely used. They are Parameters Server [33] and all-reduce [38].

The layer-wise structure of deep learning training makes it convenient to parallel the communication tasks and computing tasks. The communication scheduling strategies mainly targets at minimizing the network communication time [32, 36, 50, 26, 19, 39, 28, 29, 46, 47, 48]. Poseidon [50] supports overlapping communication process with backward propagation, reducing bursty network communication. P3 [26] attempts to overlap communication with forward propagation by layer partitioning and scheduling on MXNet PS architecture. TicTac [19] proposes a similar idea that reduces the iteration time by calculating the order of transmission. It guarantees near-optimal overlap of communication and computation. It is only implemented on TensorFlow PS. ByteScheduler [39] provides a generic communication scheduler for distributed DNN training. It supports TensorFlow, PyTorch, and MXNet on both Parameter Server and all-reduce architectures. PipeDream [36] also exploits the scheduling on the model parallelism. it proposes to aggressively pipeline mini-batch processing to maximize the utilization of computing resources. However, its coarse-grained partitioning potentially misses optimization opportunities and may fail to scale extremely large layers. TensorExpress [29] schedules tensor packets in-network using P4 to mitigate network contention and reduce training time.

To sum up, there are two main techniques for communication scheduling, namely, tensor partition and priority-based scheduling [26, 39], which have remarkable effects and thus been widely used.

There are two tunable parameters which are essential to the training performance: the partition size and the credit size [39].

However, there is a gap between theoretical upper bound and the real world performance. The cost of tensor partition is not small enough to be ignored. This partition overhead will bring about additional time consumption, making infinitesimal partition size hard to satisfy in practice. There is usually an inherent partition size which can achieve optimal training speed [39].

As shown in Figure 2. When the credit size equals to 1X in Figure 2(a), the priority scheduling policy is stop-and-wait since it allows one tensor in the sending buffer. Such a credit size is far from optimal in a 10Gbps network because of low bandwidth utilization. When the credit size equals to 2X as shown in Figure 2(b), the training time is reduced since a larger credit size leads to better bandwidth utilization. It increases the training speed from 1.37$\times$ to 1.52$\times$. However, a larger credit size will undermine the priority-scheduling. In Figure 2(c), when the credit size equals to 5X, the P0 tensors can not finish transmission before P1 tensors, thus F0 can not start beforehand. A larger credit size performs worse than a smaller credit size (the speedup drops to 1.12$\times$). Actually, credit-size is a critical parameter to balance the trade-off between the network utilization and priority scheduling.

These two important parameters, partition size and credit size, affect training performance. We use experiments to showcase this. We train four typical models, ResNet [20], VGG [41], AlexNet [30] and DenseNet [25] with MXNet [10] PS [33] and PyTorch [37] Allreduce [38] (NCCL [27]). In the motivation experiments, we vary partition size from 1M to 8M and credit size from 1X to 4X. For the dynamic bandwidth experiment, we vary the bandwidth from 3Gbps to 20Gbps. The results are shown in Figure 3.

The results show two points. (1) Different ML jobs result in different optimal pair of $\langle S_{p},S_{c}\rangle$. Different network environments do the same. As shown in Figure 3(a), the optimal $\langle S_{p},S_{c}\rangle$ is $\langle 1,4\rangle$ for ResNet, $\langle 4,1\rangle$ for VGG, $\langle 1,1\rangle$ for AlexNet and $\langle 8,2\rangle$ for DenseNet. Different ML models have respective proportion of communication to computation, thus their inherent parallel ratios are different. This reflects to different optimal configuration of $\langle S_{p},S_{c}\rangle$.

Besides, in Figure 3(b) different bandwidth will have influence on the choice of optimal $\langle S_{p},S_{c}\rangle$. The optimal $\langle S_{p},S_{c}\rangle$ is $\langle 1,4\rangle$ for 20G bandwidth, $\langle 1,2\rangle$ for 8G bandwidth and $\langle 4,2\rangle$ for 5G and 3G bandwidth. This is because in the real-world, scheduling and tensor partitioning have networking-related overhead. And credit-based priority scheduling is designed to fully utilize the network. Thus there exists an optimal pair of $\langle S_{p},S_{c}\rangle$ in particular network setting.

(2) Optimal pair of $\langle S_{p},S_{c}\rangle$ can greatly improve the training speed. As shown in Figure 3(a), the improvement of adjusting configuration ranges from $5.4\%-99.0\%$ for different ML jobs and $21.8\%-45.4\%$ for different network setting.

Actually, current communication scheduling methods [26, 39] have used different methods to tune these two knobs $S_{p}$ and $S_{c}$. Table I provides an overview and categorization of these relevant methods. 1)Default method. P3  [26] uses a default partition size of 160KB and credit size of 1X. This method has no additional overhead. But this partition size is far from optimal in 10Gbps network. This stable method may lead to performance degradation in some particular network condition and ml models. 2) The Grid Search method. It is a simple way to search the best credit size and partition size. It enumerates all possible combinations of $\langle S_{p},S_{c}\rangle$. But the cost of enumeration is too high. It is also ineffective where background traffic exists. 3) Bayesian Optimization (BO). ByteScheduler  [39] adopts BO to tune $\langle S_{p},S_{c}\rangle$ together. BO can reach optimal configuration with much less cost and give much more stable performance. However, BO only works before the beginning of training. Once the train starts, the $\langle S_{p},S_{c}\rangle$ stay constant. It can not deal with the dynamic condition like bandwidth competition and computing power change. 4) AutoByte. Our proposed method should not only find the optimal pair for particular ML model and network, but also keep the optimal performance when sudden change happens.

The optimal pair are likely to vary with many factors as listed in table I. The physical network bandwidth, gradient synchronization method, DNN model types all have effects on these parameters. Sudden change of network bandwidth and computation power also have effects. Besides, the cost of reconfiguration should be within an acceptable range.

Since the network condition and computation power are actually ever-changing during the training process, we should choose appropriate metrics to represent the dynamic runtime environment. We measure local computation time and communication time at the end of every mini-batch. Some background traffic may preempt the bandwidth. Thus we need an index to show the real-time bandwidth. So we choose the network upload speed for all workers $B_{u}$ and network download speed $B_{d}$ to show the real-time network transmission speed. Since the bandwidth metrics may fluctuate, we apply a n-dimension vector. On the other hand, the computation power also needs to be monitored to prevent sudden machine breakdown. Here we use a group of parameters to represent the real-time computation power, namely the layer-wise backward computation time of all workers $T=[T_{1},T_{2},....T_{n}]$. With these runtime metrics, the dynamic changes of allocated computation power from GPU clusters and the network bandwidth can be detected.

We denote the $V$ as the training speed, the objective of the meta network is to learn the mapping

under the different settings of $S_{p}$ and $S_{c}$. Then we can choose the best pair of $\langle S_{p},S_{c}\rangle$ to optimize the training speed.

As the objective of the meta-network is to predict precisely the speed up vector $V$. we define the loss function of the meta-network as a typical L2-loss function, i.e.,

Here, the $\overline{V}$ denote the observed average training speed. We set 10 iterations as a group. We calculate the average training speed for this group to eliminate the error and jitters.

One possible solution is to perform online training. we can online train and update the meta-network on the target distributed deep learning task. However, this also introduces system overhead as we are required to try exhaustive parameter settings recurrently to meet with various of system conditions. This method violates our original intention, i.e.,maximizing the training speed.

To overcome the issue, we proposed an offline training, online adapting approach, whose key idea is use transfer learning to quickly adapt the meta-network to the current environment with low system overhead. This method has the following three advantages. 1)It does not bring about much additional system overhead. 2) Transfer learning helps the meta network optimizer quickly adapt to the current condition. 3) This method provides higher accuracy of speed prediction than offline trained version.

The workflow of the AutoByte’s online framework is shown in Figure 4(b). AutoByte contains three important components, Meta Network Optimizer, Optimization Trigger and Execution. The Meta Network Optimizer estimates the optimal configuration for the running ML job using runtime metrics. Following the decision of Optimization Trigger, the Execution applies the necessary changes dynamically to partition size and credit size without stopping the running job.

The Optimization Trigger will also monitor the difference between the predict $V$ and the true speedup. When the difference is larger than a certain threshold $10\%$. It will trigger the online adapting scheme. The Meta Network Optimizer opportunistically tries to adapt the model to the current environment. It will use transfer learning to update the offline trained model and also update the dataset. Thus the optimizer can adjust to find an optimal configuration including the new online dataset.

To clearly show how AutoByte’s Meta Network Optimizer works, especially how it optimizes the parameter configuration when network resource changes and how much overhead it incurrs. We first compare the Meta Network Optimizer with other methods, i.e., the Grid Search and Bayesian Optimization (BO). We use the code of BO method in ByteScheduler. We compare these three optimizers on the speed and the cost of reconfiguration. We measure how much iterations they need to find the optimal pair of $\langle S_{p},S_{c}\rangle$, and how much extra overhead they bring.

We use training speed (samples/sec) as the main metric. We measure the average training speed over the first 50 iterations after a warm-up of 10 iterations.

We conduct the evaluation of AutoByte and illustrate the result in the following four sections: (1) In $\S$ IV-C, we validate that AutoByte can timely search for the optimal hyper-parameters during the training. We firstly introduce the method we use to find the optimal configuration. Then we show that AutoByte can quickly find a nearly optimal hyper parameters. (2) In $\S$ IV-D, we evaluate AutoByte when available bandwidth and computing resources are changing, which are common in a cloud environment. We emulate the resources changing by artificially adding a new training job. The experiment shows that AutoByte outperformances ByteScheduler with BO by $18.2\%\sim 39.3\%$. (3) In $\S$ IV-E, we show the speedup of AutoByte in different environments. We compare AutoByte with the baseline, ByteScheduler (with BO), on three models under different network bandwidth, different architectures (PS and Ring all-reduce). The result shows that with the help of Meta Network Optimizer, AutoByte can improve the training speed with up to 94.4$\%$ and 33.2$\%$, compared with baseline and ByteScheduler. (4) In $\S$ IV-F, we investigate the overhead introduced by AutoByte. We also compare the reconfiguration speed of AutoByte with other searching methods, namely, BO and grid search.

In this section, we evaluate AutoByte to show that: 1) The output of AutoByte can be very close to the optimal. 2) The hyper-parameters chosen by AutoByte can fast converge to the optimal. 3) AutoByte can bring considerable speedup to the training.

The above experiments are under a static environment. Actually, AutoByte can further speedup the training with a dynamic environment. AutoByte can keep track of dynamic network and computation resources in the cluster and timely updates the parameter configuration.

In the first experiment, we train a ResNet50 model. We simulate throughput variation through artificially changing the NIC bandwidth. We switch the bandwidth between 3Gbps and 10Gbps every 20 iterations. At the beginning, the bandwidth is 10Gbps. We compare the training speed of AutoByte with ByteScheduler (BO). BO chooses the best credit size and partition size for the initial environment. Once chosen, it assumes the value stay constant throughout the training process. Figure 6 shows the average speed of training for every 5 iterations. At the $20^{\mathrm{th}}$ iteration, AutoByte’s configuration moves toward $\langle S_{p}\colon 4,S_{c}\colon 2\rangle$. This configuration better fits the low bandwidth setting, achieving higher speedup than ByteScheduler. At the $40^{\mathrm{th}}$ iteration, AutoByte’s configuration returns to the previous choice $\langle S_{p}\colon 8,S_{c}\colon 3\rangle$. Thus the AutoByte can achieve 6.4$\%$-23$\%$ improvement in the training speedup.

In the second experiment, we simulate the changing of available bandwidth and computation resources by adding new training jobs. We add a new training job at $5^{\mathrm{th}}$ and $25^{\mathrm{th}}$ iterations. Since the job initialization takes 15 iteration’s time, in the first $20^{\mathrm{th}}$ iterations, the measured job occupy the whole bandwidth (20Gbps) and the whole GPU. In the next 20 iterations, two jobs share the resources. In the last 20 iterations, three jobs share resources. As shown in Figure 7, we measure the average training speed of every 5 iterations. At the $20^{\mathrm{th}}$ iteration, AutoByte’s configuration moves toward $\langle S_{p}\colon 4,S_{c}\colon 2\rangle$. This configuration better fit the low bandwidth setting. Thus achieve higher speedup than ByteScheduler with BO. At the $40^{\mathrm{th}}$ iteration, AutoByte’s configuration turns to the $\langle S_{p}\colon 2,S_{c}\colon 1\rangle$. and AutoByte achieve $8.8\%\sim 19.2\%$ higher performance.

Figure 8 and 9 compare the training speed of baseline and ByteScheduler with AutoByte on three models under different network bandwidth ranging from 500Mbps to 25 Gbps (500Mbps, 1Gbps, 5Gbps, 10Gbps, 25Gbps) under two different architectures (PS $\&$ All-reduce), which are implemented by MXNet and PyTorch respectively.

From Figure 8 (a)-(c) We observe that AutoByte can outperform baseline / ByteScheduler with BO by 16.7$\%$ to 60.1$\%$ / 3.1$\%$ to 17.5$\%$ for ResNet50, 30.2$\%$ to 94.4 $\%$ / 15.5$\%$ to 33.2$\%$ for VGG16, 40.7$\%$ to 79.7$\%$ / 10.6$\%$ to 31.1$\%$ for AlexNet. AutoByte’s meta network takes into consideration of the model’s layer-wise information and dynamic change of available resources. This is the reason AutoByte outperform baseline and ByteScheduler (BO) . We could observe the same trend for Ring all-reduce in PyTorch in Figure 9 (a)-(c), 15.1$\%$ to 144.4$\%$ improvements for baseline, and 2.3$\%$ to 17.4 $\%$ improvements for ByteScheduler. This shows that AutoByte is not only adapt to different run-time environments, but also generic to different frameworks.

To make AutoByte ready deployable, we need to ensure a low overhead and fast execution speed. To measure the overhead, we compare AutoByte with a constant hyper-parameter setting. We find that their cpu utilization is almost the same (difference is less than 1%). To measure the execution speed, we compare AutoByte with grid search and BO, as shown in Figure 10, AutoByte reduce the execution time by $28\%\sim 81\%$ compared to the grid search and the performance is close to BO.