Netra

netra

Apparate: Early-Exit Models for ML Latency and Throughput Optimization - Conclusion, References

Authors:
(1) Yinwei Dai, Princeton University (Equal contributions);
(2) Rui Pan, Princeton University (Equal contributions);
(3) Anand Iyer, Georgia Institute of Technology;
(4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.3 Challenges 3 Design 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 4 Implementation 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 6 Additional Related Work 7 Conclusion, References, Appendix 6 ADDITIONAL RELATED WORK A number of model-serving systems have been proposed [4, 5,7,17,22,39,44,49] where the focus is on serving large volumes of inference requests within a pre-defined SLO. Existing systems favor maximizing the system throughput while adhering to the latency constraints (2) by the use of intelligent placement [22,49], batching [4] and routing [44]. To the best of our knowledge, no existing serving proposals focus on alleviating the latency-throughput tension. The ML community has been actively working on early-exit networks, with several proposals focusing on the EE’s ramp architecture and exit strategy [28,36,46,53,57,58,64]. The architecture of the ramp depends on the domain, but it typically consists of one or more layers that provide information necessary to make an exit decision by emulating the original model. Replicating the last (few) layers is the common [57, 58], Apparate builds on this approach and prefers shallow ramps in its workflow (3). Once a ramp architecture is chosen, the exit strategy could be based on confidence of the labels [36] or entropy of the prediction [57]. More sophisticated approaches exist, e.g., instead of considering the ramps as fully independent, [64] uses counter-based exiting. Apparate’s focus is on leveraging EEs to resolve the latencythroughput tension in serving systems with a design that generalizes to a large class of EE architectures. Optimizing model serving objectives based on workload characteristics has been discussed in recent works [15, 16, 21, 44, 49, 62, 63]. Inferline [16] optimizes cost in serving pipelines of models while adhering to strict latency constraints using intelligent provisioning and management. Shepherd [63] maximizes goodput and resource utilization in highly unpredictable workloads. Despite their impressive results, these works still optimize their metric of choice at the expense of latency and do not resolve the latency-throughput tension, which is the focus of our (complementary) work. A recent line of work has focused on creating variants of an ML model to optimize serving performance. Some of these works look at execution graph level optimizations such as quantization and fusing to reduce inference latency [1, 3], while others replace the model with an equivalent one that meets the provided constraints. Solutions like Mystify [23] and INFaaS [44] generates and chooses model variants based on their intent and constraints (including performance). As shown in §5.2, Apparate’s wins persist even on compressed models, and is thus complementary to these works in that it can operate on their outputs. Finally, optimizing the execution of dynamic neural networks that alter NN execution (e.g., EEs, mixture of experts) was proposed in [18, 61]. These are low-level optimizations (e.g., at the GPU) which can benefit Apparate and improve its performance. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Authors: Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.2 Early-Exit Models 2.3 Challenges 2.3 Challenges 3 Design 3 Design 3.1 Preparing Models with Early Exits 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 3.3 Latency-Focused Ramp Adjustments 4 Implementation 4 Implementation 5 Evaluation and 5.1 Methodology 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 5.4 Microbenchmarks 6 Additional Related Work 6 Additional Related Work 7 Conclusion, References, Appendix 7 Conclusion, References, Appendix 6 ADDITIONAL RELATED WORK A number of model-serving systems have been proposed [4, 5,7,17,22,39,44,49] where the focus is on serving large volumes of inference requests within a pre-defined SLO. Existing systems favor maximizing the system throughput while adhering to the latency constraints (2) by the use of intelligent placement [22,49], batching [4] and routing [44]. To the best of our knowledge, no existing serving proposals focus on alleviating the latency-throughput tension. model-serving systems The ML community has been actively working on early-exit networks , with several proposals focusing on the EE’s ramp architecture and exit strategy [28,36,46,53,57,58,64]. The architecture of the ramp depends on the domain, but it typically consists of one or more layers that provide information necessary to make an exit decision by emulating the original model. Replicating the last (few) layers is the common [57, 58], Apparate builds on this approach and prefers shallow ramps in its workflow (3). Once a ramp architecture is chosen, the exit strategy could be based on confidence of the labels [36] or entropy of the prediction [57]. More sophisticated approaches exist, e.g., instead of considering the ramps as fully independent, [64] uses counter-based exiting. Apparate’s focus is on leveraging EEs to resolve the latencythroughput tension in serving systems with a design that generalizes to a large class of EE architectures. early-exit networks Optimizing model serving objectives based on workload characteristics has been discussed in recent works [15, 16, 21, 44, 49, 62, 63]. Inferline [16] optimizes cost in serving pipelines of models while adhering to strict latency constraints using intelligent provisioning and management. Shepherd [63] maximizes goodput and resource utilization in highly unpredictable workloads. Despite their impressive results, these works still optimize their metric of choice at the expense of latency and do not resolve the latency-throughput tension, which is the focus of our (complementary) work. A recent line of work has focused on creating variants of an ML model to optimize serving performance. Some of these works look at execution graph level optimizations such as quantization and fusing to reduce inference latency [1, 3], while others replace the model with an equivalent one that meets the provided constraints. Solutions like Mystify [23] and INFaaS [44] generates and chooses model variants based on their intent and constraints (including performance). As shown in §5.2, Apparate’s wins persist even on compressed models, and is thus complementary to these works in that it can operate on their outputs. Finally, optimizing the execution of dynamic neural networks that alter NN execution (e.g., EEs, mixture of experts) was proposed in [18, 61]. These are low-level optimizations (e.g., at the GPU) which can benefit Apparate and improve its performance. dynamic neural networks This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Apparate: Early-Exit Models for ML Latency and Throughput Optimization - Additional Related Work

Authors:
(1) Yinwei Dai, Princeton University (Equal contributions);
(2) Rui Pan, Princeton University (Equal contributions);
(3) Anand Iyer, Georgia Institute of Technology;
(4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.3 Challenges 3 Design 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 4 Implementation 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 6 Additional Related Work 7 Conclusion, References, Appendix 5.4 Microbenchmarks To ease presentation, results in this section use representative CV and NLP models (ResNet50, GPT2-medium) running on a random corpus video and Amazon reviews, respectively. All reported trends hold for all considered workloads. Parameter sensitivity. Recall that Apparate ingests values for two key parameters: ramp aggression (i.e., a ramp budget) and accuracy constraint (i.e., acceptable accuracy loss). Fig. 18 and Tab. 3 studies the effect that these parameters have on Apparate’s latency wins. The findings are intuitive: Apparate’s latency savings over vanilla models decrease as ramp budgets shrink or accuracy constraints tighten. Both trends are a result of Apparate being granted less flexibility for adaptation, either via smaller acceptable threshold ranges (that meet accuracy targets), or active ramp capacity (and thus potential ramp configurations). Importantly, accuracy constraint has a larger impact on Apparate’s wins. The reason is that inter-ramp dependencies result in overlap in the set of inputs that can exit at any ramp when run in isolation; thus, wins from using more ramps eventually hits diminishing returns. Indeed, Apparate begins by using the full budget, only to quickly disable many ramps that have a net negative effect on serving latencies. Ramp architectures. Although Apparate opts for using many lightweight ramps, it’s adaptation algorithms can support any ramp architecture. To illustrate this, we ran Apparate with DeeBERT’s more expensive ramps (described above). Overall, on the Amazon Reviews dataset, we find that these costlier ramps dampen Apparate’s latency savings by 4% since they constrain Apparate’s runtime adaptation in terms of feasible configurations, i.e., fewer active ramps at any time. Crucially, we note that accuracy constraints were still entirely met due to Apparate’s frequent threshold tuning. Impact of serving platform. Apparate runs atop existing serving platforms, responding to overall serving (and exiting patterns) rather than altering platform decisions, e.g., for queuing. Table 4 shows that, despite the discrepancies in platform scheduling strategies and knobs, Apparate’s performance wins are largely insensitive to the underlying platform when CV or NLP workloads are configured with the same SLO goal. For example, Apparate’s median latency savings for the Amazon workload and GPT-2 are within 2.9% when using Clockwork or TensorFlow-Serving. Profiling Apparate. Figure 11 in §3.2 analyzes the runtime and optimality of Apparate’s threshold tuning algorithm. Beyond that, Apparate includes two other overheads while running: ramp adjustment and coordination between its CPU controller and serving GPUs. Ramp adjustment rounds take an average of 0.5 ms. Coordination overheads are also low because of Apparate’s small ramp sizes (definitions and weights consume only ∼10KB) and profiling data (simply a top-predicted result with an error score, collectively consuming around ∼1KB). Thus, CPU-GPU coordination delays take an average of 0.5ms per communication, 0.4ms of which comes from fixed PCIe latencies in our setup. Importance of Apparate’s techniques. Apparate’s runtime adaptation considers frequent (accuracy-guided) threshold tuning, with periodic ramp adjustments. Table 2 highlights the importance of threshold tuning on average accuracies. Here, we evaluate the importance of ramp adjustment on Apparate’s latency improvements by comparing versions with and without this optimization. Overall, disabling ramp adjustment results in 20.8-33.4% lower median latency wins, though worst-case latency (and throughput) and accuracy constraints remain continually met. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Authors: Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.2 Early-Exit Models 2.3 Challenges 2.3 Challenges 3 Design 3 Design 3.1 Preparing Models with Early Exits 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 3.3 Latency-Focused Ramp Adjustments 4 Implementation 4 Implementation 5 Evaluation and 5.1 Methodology 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 5.4 Microbenchmarks 6 Additional Related Work 6 Additional Related Work 7 Conclusion, References, Appendix 7 Conclusion, References, Appendix 5.4 Microbenchmarks To ease presentation, results in this section use representative CV and NLP models (ResNet50, GPT2-medium) running on a random corpus video and Amazon reviews, respectively. All reported trends hold for all considered workloads. Parameter sensitivity. Recall that Apparate ingests values for two key parameters: ramp aggression (i.e., a ramp budget) and accuracy constraint (i.e., acceptable accuracy loss). Fig. 18 and Tab. 3 studies the effect that these parameters have on Apparate’s latency wins. The findings are intuitive: Apparate’s latency savings over vanilla models decrease as ramp budgets shrink or accuracy constraints tighten. Both trends are a result of Apparate being granted less flexibility for adaptation, either via smaller acceptable threshold ranges (that meet accuracy targets), or active ramp capacity (and thus potential ramp configurations). Importantly, accuracy constraint has a larger impact on Apparate’s wins. The reason is that inter-ramp dependencies result in overlap in the set of inputs that can exit at any ramp when run in isolation; thus, wins from using more ramps eventually hits diminishing returns. Indeed, Apparate begins by using the full budget, only to quickly disable many ramps that have a net negative effect on serving latencies. Parameter sensitivity. Ramp architectures. Although Apparate opts for using many lightweight ramps, it’s adaptation algorithms can support any ramp architecture. To illustrate this, we ran Apparate with DeeBERT’s more expensive ramps (described above). Overall, on the Amazon Reviews dataset, we find that these costlier ramps dampen Apparate’s latency savings by 4% since they constrain Apparate’s runtime adaptation in terms of feasible configurations, i.e., fewer active ramps at any time. Crucially, we note that accuracy constraints were still entirely met due to Apparate’s frequent threshold tuning. Ramp architectures. Impact of serving platform. Apparate runs atop existing serving platforms, responding to overall serving (and exiting patterns) rather than altering platform decisions, e.g., for queuing. Table 4 shows that, despite the discrepancies in platform scheduling strategies and knobs, Apparate’s performance wins are largely insensitive to the underlying platform when CV or NLP workloads are configured with the same SLO goal. For example, Apparate’s median latency savings for the Amazon workload and GPT-2 are within 2.9% when using Clockwork or TensorFlow-Serving. Impact of serving platform. Profiling Apparate. Figure 11 in §3.2 analyzes the runtime and optimality of Apparate’s threshold tuning algorithm. Beyond that, Apparate includes two other overheads while running: ramp adjustment and coordination between its CPU controller and serving GPUs. Ramp adjustment rounds take an average of 0.5 ms. Coordination overheads are also low because of Apparate’s small ramp sizes (definitions and weights consume only ∼10KB) and profiling data (simply a top-predicted result with an error score, collectively consuming around ∼1KB). Thus, CPU-GPU coordination delays take an average of 0.5ms per communication, 0.4ms of which comes from fixed PCIe latencies in our setup. Profiling Apparate. Importance of Apparate’s techniques. Apparate’s runtime adaptation considers frequent (accuracy-guided) threshold tuning, with periodic ramp adjustments. Table 2 highlights the importance of threshold tuning on average accuracies. Here, we evaluate the importance of ramp adjustment on Apparate’s latency improvements by comparing versions with and without this optimization. Overall, disabling ramp adjustment results in 20.8-33.4% lower median latency wins, though worst-case latency (and throughput) and accuracy constraints remain continually met. Importance of Apparate’s techniques. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Apparate: Early-Exit Models for ML Latency and Throughput Optimization - Microbenchmarks

Authors:
(1) Yinwei Dai, Princeton University (Equal contributions);
(2) Rui Pan, Princeton University (Equal contributions);
(3) Anand Iyer, Georgia Institute of Technology;
(4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.3 Challenges 3 Design 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 4 Implementation 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 6 Additional Related Work 7 Conclusion, References, Appendix 5.3 Comparison with Existing EE Strategies We compare Apparate with two off-the-shelf EE models: BranchyNet [53] and DeeBERT [57]. BranchyNet extends ResNet models with ramps of the same style as Apparate, while DeeBERT extends BERT-base with deeper ramps (using the entire BERT pooler, as described in §3.1). For each, we follow their prescribed architectures, with ramps after every layer that are always active. We perform one-time tuning of thresholds as recommended by both works, and consider two variants: the default recommendation where all ramps must use the same threshold, and a more flexible version that removes this restriction (+). For both, threshold tuning is done optimally (via grid search), and is based on uniformly sampled data across the workload. For fair comparison, Apparate’s ramp budget is configured to support ramps at all layers (though it never does so). Table 2 presents our results. The main takeaway is that existing EE approaches, even when favorably tuned, yield unacceptable drops in average accuracy up to 23.9% and 17.8% for CV and NLP. In contrast, Apparate consistently meets the imposed accuracy constraint (1% in this experiment) for both workloads. Further, even with such accuracy violations, tail latencies are 0.9-9.4% lower with Apparate than with these systems. The reason is again lack of adaptation: all ramps are always active despite their current efficacy which vary dramatically over time (§2.3), yielding undue overheads for large numbers of non-exiting inputs. In contrast, throughout these experiments, despite having a full ramp budget, Apparate maintained only 9.1-27.2% of all possible ramps. For fair median latency comparison, we consider an optimally-tuned (opt) version of existing EE models that perform one-time tuning on the actual test dataset, picking the best (latency-wise) thresholds that ensure <1% accuracy drop. As shown, due to its regular and less-constrained adaptation, Apparate outperforms even this oracle version of existing EEs with up to 14.1% higher median latency savings. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Authors: Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.2 Early-Exit Models 2.3 Challenges 2.3 Challenges 3 Design 3 Design 3.1 Preparing Models with Early Exits 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 3.3 Latency-Focused Ramp Adjustments 4 Implementation 4 Implementation 5 Evaluation and 5.1 Methodology 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 5.4 Microbenchmarks 6 Additional Related Work 6 Additional Related Work 7 Conclusion, References, Appendix 7 Conclusion, References, Appendix 5.3 Comparison with Existing EE Strategies We compare Apparate with two off-the-shelf EE models: BranchyNet [53] and DeeBERT [57]. BranchyNet extends ResNet models with ramps of the same style as Apparate, while DeeBERT extends BERT-base with deeper ramps (using the entire BERT pooler, as described in §3.1). For each, we follow their prescribed architectures, with ramps after every layer that are always active. We perform one-time tuning of thresholds as recommended by both works, and consider two variants: the default recommendation where all ramps must use the same threshold, and a more flexible version that removes this restriction (+). For both, threshold tuning is done optimally (via grid search), and is based on uniformly sampled data across the workload. For fair comparison, Apparate’s ramp budget is configured to support ramps at all layers (though it never does so). Table 2 presents our results. The main takeaway is that existing EE approaches, even when favorably tuned, yield unacceptable drops in average accuracy up to 23.9% and 17.8% for CV and NLP. In contrast, Apparate consistently meets the imposed accuracy constraint (1% in this experiment) for both workloads. Further, even with such accuracy violations, tail latencies are 0.9-9.4% lower with Apparate than with these systems. The reason is again lack of adaptation: all ramps are always active despite their current efficacy which vary dramatically over time (§2.3), yielding undue overheads for large numbers of non-exiting inputs. In contrast, throughout these experiments, despite having a full ramp budget, Apparate maintained only 9.1-27.2% of all possible ramps. For fair median latency comparison, we consider an optimally-tuned (opt) version of existing EE models that perform one-time tuning on the actual test dataset, picking the best (latency-wise) thresholds that ensure <1% accuracy drop. As shown, due to its regular and less-constrained adaptation, Apparate outperforms even this oracle version of existing EEs with up to 14.1% higher median latency savings. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Apparate: Early-Exit Models for ML Latency and Throughput Optimization - Comparisons

Authors:
(1) Yinwei Dai, Princeton University (Equal contributions);
(2) Rui Pan, Princeton University (Equal contributions);
(3) Anand Iyer, Georgia Institute of Technology;
(4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.3 Challenges 3 Design 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 4 Implementation 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 6 Additional Related Work 7 Conclusion, References, Appendix 5.2 Overall Results Figures 13- 16 compare Apparate with vanilla model serving and optimal exiting across our workloads. Overall, Apparate significantly lowers latencies compared to serving vanilla models, while always adhering to the imposed 1% accuracy constraint. For instance, median speedups range from 40.5-91.5% (2.7-30.5 ms) for CV workloads, and jump to 70.2-94.2% (5.2-31.4 ms) at the 25th percentiles. NLP workloads follow a similar pattern, with median and 25th percentile savings ranging from 10.0-24.2% (3.9-25.3 ms) and 16.0-37.3% (4.8-53.2 ms), respectively. Importantly, across all workloads, Apparate’s tail latency (and thus impact on throughput) always falls within its granted budget (2% here) and is most often negligible (Figure 14). Beyond this, there are several important trends to note. First, Apparate’s raw latency savings grow with increased model sizes, e.g., 25th percentile wins of 53.2ms, 28.4ms, 14.3ms, 5.5ms for GPT-2, BERT-large, BERT-base, and Distilbert-base on the Amazon dataset. This is because only the results exit models with Apparate, with inputs running to completion; thus, latency savings pertain entirely to serving times (not queuing delays), with exit impact being higher as model size (and thus, runtime) grows. Relative (%) latency savings follow the same pattern for CV workloads, e.g., Apparate’s median wins grow by 13.8% and 5.3% moving from the smallest to the biggest models in the ResNet and VGG families. However, relative wins remain relatively stable in NLP models, e.g., 15.8% and 13.7% for GPT2 and BERTlarge on Amazon Reviews. The reason for this difference is in the effectiveness of the models in each domain. Results and task performance are largely similar across the considered CV models, enabling Apparate to inject ramps early in (even larger) models. In contrast, results and task efficacy are far better with the larger models in NLP; thus, Apparate’s ramps fall in similar (relative) positions across the models. Second, Apparate’s wins are larger for CV workloads than NLP workloads for two reasons. As previously noted, CV workloads use lighter models and lower request rates (bound by video fps), and thus incur far lower queuing delays. More importantly, in contrast to CV where spatiotemporal similarities across frames (and thus, requests) are high due to physical constraints of object motion in a scene, NLP requests exhibit less continuity, e.g., back-to-back reviews are not constrained in semantic similarity. The effects on Apparate’s adaptation of EE configurations are that (1) past data is less representative of future data, and (2) the duration until subsequent adaptation is shorter. Lastly, Apparate’s wins for NLP are consistently higher with the Amazon dataset versus the IMDB dataset. The reason is that the IMDB dataset exhibits ‘harder’ inputs that require later-ramp exits. For instance, for GPT-2, average exit locations for the offline optimal exit strategy are at layers 2.2 and 2.7 for Amazon and IMDB, respectively; these numbers are 2.1 and 2.5 for BERT-base. Comparisons with optimal. As shown in Figure 13, latency savings with Apparate for CV workloads largely mirror those of the optimal that tunes exiting decisions based on perfect knowledge of the upcoming workload, e.g., median savings are within 20.5% of the optimal. In contrast, building on the discussion above, the limited continuity across inputs in NLP workloads leads to a wider gap of 73.1-83.2% at the median (Figure 16). To further characterize Apparate’s performance on these workloads, we also consider a more realistic online optimal algorithm that relaxes the following elements. First, rather than per-sample adaptation of thresholds and ramps, ramp adjustments are set to operate only as fast as model definitions in the GPU can be updated. Second, rather than using perfect knowledge of upcoming inputs, decisions are made using only recent (historical) data; for each threshold or ramp adaptation decision, we tune based on the past {20, 40, 80} batches of inputs and select the decision that performs best on the upcoming data. As shown in Figure 16, Apparate’s median latency savings are within 16.9-52.1% of this (more) realistic optimal exiting strategy Varying SLOs. SLOs affect serving system batching and queuing decisions, and thus Apparate’s benefits. To study this effect, we considered SLOs for each model that were 2× and 4× those in our default experiments (Table 5). Generally, higher SLOs induce larger serving batch sizes and higher per-request queuing delays (as inputs wait for the previous, slower inference batch to complete); this dampens Apparate’s relative latency savings which target model runtimes, but not queuing delays. Figure 17 illustrates this effect. For instance, median latency savings for GPT-2 drop from 16.3% to 6.8% as SLO grows by 4×. Note that, to illustrate this trend for CV workloads, we upsampled each video to 120 fps. The reason is that our serving platforms are workconserving and, at 30 fps, they are able to consistently schedule jobs with batch size 1 and low queuing delays given the low runtimes of the considered CV models. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Authors: Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.2 Early-Exit Models 2.3 Challenges 2.3 Challenges 3 Design 3 Design 3.1 Preparing Models with Early Exits 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 3.3 Latency-Focused Ramp Adjustments 4 Implementation 4 Implementation 5 Evaluation and 5.1 Methodology 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 5.4 Microbenchmarks 6 Additional Related Work 6 Additional Related Work 7 Conclusion, References, Appendix 7 Conclusion, References, Appendix 5.2 Overall Results Figures 13- 16 compare Apparate with vanilla model serving and optimal exiting across our workloads. Overall, Apparate significantly lowers latencies compared to serving vanilla models, while always adhering to the imposed 1% accuracy constraint. For instance, median speedups range from 40.5-91.5% (2.7-30.5 ms) for CV workloads, and jump to 70.2-94.2% (5.2-31.4 ms) at the 25th percentiles. NLP workloads follow a similar pattern, with median and 25th percentile savings ranging from 10.0-24.2% (3.9-25.3 ms) and 16.0-37.3% (4.8-53.2 ms), respectively. Importantly, across all workloads, Apparate’s tail latency (and thus impact on throughput) always falls within its granted budget (2% here) and is most often negligible (Figure 14). Beyond this, there are several important trends to note. First, Apparate’s raw latency savings grow with increased model sizes, e.g., 25th percentile wins of 53.2ms, 28.4ms, 14.3ms, 5.5ms for GPT-2, BERT-large, BERT-base, and Distilbert-base on the Amazon dataset. This is because only the results exit models with Apparate, with inputs running to completion; thus, latency savings pertain entirely to serving times (not queuing delays), with exit impact being higher as model size (and thus, runtime) grows. Relative (%) latency savings follow the same pattern for CV workloads, e.g., Apparate’s median wins grow by 13.8% and 5.3% moving from the smallest to the biggest models in the ResNet and VGG families. However, relative wins remain relatively stable in NLP models, e.g., 15.8% and 13.7% for GPT2 and BERTlarge on Amazon Reviews. The reason for this difference is in the effectiveness of the models in each domain. Results and task performance are largely similar across the considered CV models, enabling Apparate to inject ramps early in (even larger) models. In contrast, results and task efficacy are far better with the larger models in NLP; thus, Apparate’s ramps fall in similar (relative) positions across the models. Second, Apparate’s wins are larger for CV workloads than NLP workloads for two reasons. As previously noted, CV workloads use lighter models and lower request rates (bound by video fps), and thus incur far lower queuing delays. More importantly, in contrast to CV where spatiotemporal similarities across frames (and thus, requests) are high due to physical constraints of object motion in a scene, NLP requests exhibit less continuity, e.g., back-to-back reviews are not constrained in semantic similarity. The effects on Apparate’s adaptation of EE configurations are that (1) past data is less representative of future data, and (2) the duration until subsequent adaptation is shorter. Lastly, Apparate’s wins for NLP are consistently higher with the Amazon dataset versus the IMDB dataset. The reason is that the IMDB dataset exhibits ‘harder’ inputs that require later-ramp exits. For instance, for GPT-2, average exit locations for the offline optimal exit strategy are at layers 2.2 and 2.7 for Amazon and IMDB, respectively; these numbers are 2.1 and 2.5 for BERT-base. Comparisons with optimal. As shown in Figure 13, latency savings with Apparate for CV workloads largely mirror those of the optimal that tunes exiting decisions based on perfect knowledge of the upcoming workload, e.g., median savings are within 20.5% of the optimal. In contrast, building on the discussion above, the limited continuity across inputs in NLP workloads leads to a wider gap of 73.1-83.2% at the median (Figure 16). To further characterize Apparate’s performance on these workloads, we also consider a more realistic online optimal algorithm that relaxes the following elements. First, rather than per-sample adaptation of thresholds and ramps, ramp adjustments are set to operate only as fast as model definitions in the GPU can be updated. Second, rather than using perfect knowledge of upcoming inputs, decisions are made using only recent (historical) data; for each threshold or ramp adaptation decision, we tune based on the past {20, 40, 80} batches of inputs and select the decision that performs best on the upcoming data. As shown in Figure 16, Apparate’s median latency savings are within 16.9-52.1% of this (more) realistic optimal exiting strategy Comparisons with optimal. Varying SLOs. SLOs affect serving system batching and queuing decisions, and thus Apparate’s benefits. To study this effect, we considered SLOs for each model that were 2× and 4× those in our default experiments (Table 5). Generally, higher SLOs induce larger serving batch sizes and higher per-request queuing delays (as inputs wait for the previous, slower inference batch to complete); this dampens Apparate’s relative latency savings which target model runtimes, but not queuing delays. Figure 17 illustrates this effect. For instance, median latency savings for GPT-2 drop from 16.3% to 6.8% as SLO grows by 4×. Note that, to illustrate this trend for CV workloads, we upsampled each video to 120 fps. The reason is that our serving platforms are workconserving and, at 30 fps, they are able to consistently schedule jobs with batch size 1 and low queuing delays given the low runtimes of the considered CV models. Varying SLOs. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Apparate: Early-Exit Models for ML Latency and Throughput Optimization - Overall Results

Authors:
(1) Yinwei Dai, Princeton University (Equal contributions);
(2) Rui Pan, Princeton University (Equal contributions);
(3) Anand Iyer, Georgia Institute of Technology;
(4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.3 Challenges 3 Design 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 4 Implementation 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 6 Additional Related Work 7 Conclusion, References, Appendix 5 EVALUATION We evaluated Apparate across a wide range of NLP and CV workloads and serving platforms. Our key findings are: • Apparate lowers 25th percentile and median latencies by 40.5-91.5% and 70.2-94.2% for CV, and 16.0-37.3% and 10.0-24.2% for NLP workloads, compared to original (non-EE) models. • Unlike existing EE models that unacceptably worsen accuracies and tail latencies by up to 23.9% and 11.0%, Apparate consistently meets specified accuracy and tail latency constraints. • Apparate automatically generalizes to different model architectures (e.g., compressed) and EE configurations (e.g., ramp style), and its wins gracefully shrink as accuracy or tail-latency constraints grow. 5.1 Methodology Models. We consider 10 models (across 4 families) that cover popular architectures and a variety of model sizes in both CV and NLP. For CV, we use the ResNet{18, 50, 101} residual models, as well VGG{11, 13, 16} models that follow a chained (linear) design. All of these models are pretrained on ImageNet and from the PyTorch Model Zoo [43]; we further fine-tune the models to our video scenes using a random sampling of 10% of frames across our dataset. For NLP, we consider 3 encoder-only transformers from the BERT family – BERT-base, BERT-large, and Distilbert [45] (a variant of BERT-base that was compressed via distillation) – as well as a decoder-only large language model: GPT2-medium. These models span 66-345 million parameters, were collected from HuggingFace [30], and were pretrained (without more fine-tuning) on Yelp reviews [8] that are separate from any evaluation dataset we use. Workloads. CV workloads comprise real-time object classification (people, cars) on 8 one-hour videos used in recent video analytics literature [11, 27]. The videos were sampled at 30 frames per second, and collectively span day/night from diverse urban scenes; for each, we perform classification. NLP workloads focus on sentiment analysis using two datasets: Amazon product reviews [10] and IMDB movie reviews [41]. To the best of our knowledge, there do not exist public streaming workloads for NLP classification, so we convert these datasets into streaming workloads as follows. For Amazon, we follow the order of product categories in the original dataset, but within each category, we keep reviews only from frequent users (i.e., those with >1k reviews) and order streaming by user (250k requests in total). For IMDB, we follow the order of reviews in the original dataset, but stream each in sentence by sentence (180k requests in total). We then define arrival patterns for these ordered requests by using the Microsoft Azure Functions (MAF) as in prior serving work [22]. Specifically, to ensure meaningful and realistic workloads despite the high degree of variation in runtime across our models, we paired each model with a randomly selected trace snippet from the set that met the following criteria: (1) number of requests match that in our largest dataset, and (2) queries per second should not result in >20% dropped requests with vanilla serving for the given model and selected SLO (described next). Parameter configurations. Given our focus on interactivity, we cope with heterogeneity in model runtimes by setting SLOs to be 2× each model’s inference time with batch size 1 in our main experiments. This results in SLOs between 13-204 ms, which match the ranges considered in prior work [22, 40, 49]; Table 5 in §A lists the specific SLO values per model, and we study the effect of SLO on Apparate in §5.4. Unless otherwise noted, results use 1% for Apparate’s accuracy constraint (in line with industry reports [42]) and a ramp budget of 2% impact on worst-case latency; we consider other values for both parameters in §5.4. Setup. Experiments were conducted on a dedicated server with one NVIDIA RTX A6000 housing 48GB of memory, one AMD EPYC 7543P 32-Core CPU, and 256GB DDR4 RAM. We run experiments with two serving platforms: TensorFlow-Serving [39] and Clockwork [22]. We primarily present results with Clockwork due to space constraints, but note that report trends hold for both platforms; we compare cross-platform results in §5.4. Metrics and baselines. Our main metrics of evaluation are classification accuracy and per-request latencies. Accuracy is defined as the percentage of inputs that are assigned the correct label as defined by each original (i.e., non-EE) model. Per-request latency is measured as the time between when a request arrives at a serving platform, and when its response is released by the platform. We mainly compare Apparate with two baselines: (1) original models without EEs (vanilla) running in serving platforms, and (2) optimal EEs as defined in §2.2, i.e., assuming all inputs exit at their earliest possible ramps with non-exiting inputs incurring no ramp overheads. We compare Apparate to existing EE strategies in §5.3. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Authors: Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.2 Early-Exit Models 2.3 Challenges 2.3 Challenges 3 Design 3 Design 3.1 Preparing Models with Early Exits 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 3.3 Latency-Focused Ramp Adjustments 4 Implementation 4 Implementation 5 Evaluation and 5.1 Methodology 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 5.4 Microbenchmarks 6 Additional Related Work 6 Additional Related Work 7 Conclusion, References, Appendix 7 Conclusion, References, Appendix 5 EVALUATION We evaluated Apparate across a wide range of NLP and CV workloads and serving platforms. Our key findings are: • Apparate lowers 25th percentile and median latencies by 40.5-91.5% and 70.2-94.2% for CV, and 16.0-37.3% and 10.0-24.2% for NLP workloads, compared to original (non-EE) models. • Unlike existing EE models that unacceptably worsen accuracies and tail latencies by up to 23.9% and 11.0%, Apparate consistently meets specified accuracy and tail latency constraints. • Apparate automatically generalizes to different model architectures (e.g., compressed) and EE configurations (e.g., ramp style), and its wins gracefully shrink as accuracy or tail-latency constraints grow. 5.1 Methodology Models . We consider 10 models (across 4 families) that cover popular architectures and a variety of model sizes in both CV and NLP. For CV, we use the ResNet{18, 50, 101} residual models, as well VGG{11, 13, 16} models that follow a chained (linear) design. All of these models are pretrained on ImageNet and from the PyTorch Model Zoo [43]; we further fine-tune the models to our video scenes using a random sampling of 10% of frames across our dataset. For NLP, we consider 3 encoder-only transformers from the BERT family – BERT-base, BERT-large, and Distilbert [45] (a variant of BERT-base that was compressed via distillation) – as well as a decoder-only large language model: GPT2-medium. These models span 66-345 million parameters, were collected from HuggingFace [30], and were pretrained (without more fine-tuning) on Yelp reviews [8] that are separate from any evaluation dataset we use. Models Workloads . CV workloads comprise real-time object classification (people, cars) on 8 one-hour videos used in recent video analytics literature [11, 27]. The videos were sampled at 30 frames per second, and collectively span day/night from diverse urban scenes; for each, we perform classification. Workloads NLP workloads focus on sentiment analysis using two datasets: Amazon product reviews [10] and IMDB movie reviews [41]. To the best of our knowledge, there do not exist public streaming workloads for NLP classification, so we convert these datasets into streaming workloads as follows. For Amazon, we follow the order of product categories in the original dataset, but within each category, we keep reviews only from frequent users (i.e., those with >1k reviews) and order streaming by user (250k requests in total). For IMDB, we follow the order of reviews in the original dataset, but stream each in sentence by sentence (180k requests in total). We then define arrival patterns for these ordered requests by using the Microsoft Azure Functions (MAF) as in prior serving work [22]. Specifically, to ensure meaningful and realistic workloads despite the high degree of variation in runtime across our models, we paired each model with a randomly selected trace snippet from the set that met the following criteria: (1) number of requests match that in our largest dataset, and (2) queries per second should not result in >20% dropped requests with vanilla serving for the given model and selected SLO (described next). Parameter configurations. Given our focus on interactivity, we cope with heterogeneity in model runtimes by setting SLOs to be 2× each model’s inference time with batch size 1 in our main experiments. This results in SLOs between 13-204 ms, which match the ranges considered in prior work [22, 40, 49]; Table 5 in §A lists the specific SLO values per model, and we study the effect of SLO on Apparate in §5.4. Unless otherwise noted, results use 1% for Apparate’s accuracy constraint (in line with industry reports [42]) and a ramp budget of 2% impact on worst-case latency; we consider other values for both parameters in §5.4. Parameter configurations. Setup . Experiments were conducted on a dedicated server with one NVIDIA RTX A6000 housing 48GB of memory, one AMD EPYC 7543P 32-Core CPU, and 256GB DDR4 RAM. We run experiments with two serving platforms: TensorFlow-Serving [39] and Clockwork [22]. We primarily present results with Clockwork due to space constraints, but note that report trends hold for both platforms; we compare cross-platform results in §5.4. Setup Metrics and baselines. Our main metrics of evaluation are classification accuracy and per-request latencies. Accuracy is defined as the percentage of inputs that are assigned the correct label as defined by each original (i.e., non-EE) model. Per-request latency is measured as the time between when a request arrives at a serving platform, and when its response is released by the platform. We mainly compare Apparate with two baselines: (1) original models without EEs (vanilla) running in serving platforms, and (2) optimal EEs as defined in §2.2, i.e., assuming all inputs exit at their earliest possible ramps with non-exiting inputs incurring no ramp overheads. We compare Apparate to existing EE strategies in §5.3. Metrics and baselines. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Apparate: Early-Exit Models for ML Latency and Throughput Optimization - Evaluation and Methodology

Authors:
(1) Yinwei Dai, Princeton University (Equal contributions);
(2) Rui Pan, Princeton University (Equal contributions);
(3) Anand Iyer, Georgia Institute of Technology;
(4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.3 Challenges 3 Design 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 4 Implementation 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 6 Additional Related Work 7 Conclusion, References, Appendix 4 IMPLEMENTATION Apparate is implemented as a layer atop TensorFlowServing [39] and Clockwork [22] (using PyTorch [7]) and includes the components described in 3 written as Python modules in ∼7500 lines of code. Although we chose these platforms for our current implementation, we note that Apparate is not limited to them and its techniques can be implemented in any inference platform. Importantly, Apparate entirely leverages the scheduling and queuing mechanisms of the underlying framework. Original models are ingested in the ONNX format [6] and compiled for performance. Ramp training (during bootstrapping) uses the first 10% of each dataset following a 1:9 split for training and validation; the remaining 90% of each dataset is used for evaluation. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Authors: Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.2 Early-Exit Models 2.3 Challenges 2.3 Challenges 3 Design 3 Design 3.1 Preparing Models with Early Exits 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 3.3 Latency-Focused Ramp Adjustments 4 Implementation 4 Implementation 5 Evaluation and 5.1 Methodology 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 5.4 Microbenchmarks 6 Additional Related Work 6 Additional Related Work 7 Conclusion, References, Appendix 7 Conclusion, References, Appendix 4 IMPLEMENTATION Apparate is implemented as a layer atop TensorFlowServing [39] and Clockwork [22] (using PyTorch [7]) and includes the components described in 3 written as Python modules in ∼7500 lines of code. Although we chose these platforms for our current implementation, we note that Apparate is not limited to them and its techniques can be implemented in any inference platform. Importantly, Apparate entirely leverages the scheduling and queuing mechanisms of the underlying framework. Original models are ingested in the ONNX format [6] and compiled for performance. Ramp training (during bootstrapping) uses the first 10% of each dataset following a 1:9 split for training and validation; the remaining 90% of each dataset is used for evaluation. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Apparate: Early-Exit Models for ML Latency and Throughput Optimization - Implementation

Authors:
(1) Yinwei Dai, Princeton University (Equal contributions);
(2) Rui Pan, Princeton University (Equal contributions);
(3) Anand Iyer, Georgia Institute of Technology;
(4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.3 Challenges 3 Design 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 4 Implementation 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 6 Additional Related Work 7 Conclusion, References, Appendix 3.3 Latency-Focused Ramp Adjustments The set of active ramps ultimately dictates where inputs can exit, and thus provides bounds on potential latency savings. Unlike threshold tuning which runs reactively (since accuracy is a constraint) and uses only recent profiling data to evaluate new configurations, ramp adjustment is strictly an optimization (for latency savings), and requires deployment to evaluate the impact of any new ramp. Thus, Apparate’s ramp tuning runs periodically (every 128 samples by default) and conservatively alters the set of active ramps to incrementally converge on high-performing configurations. Evaluating active ramps. In each round, Apparate’s controller starts by computing a utility score for each active ramp that evaluates its overall impact on workload latency. To do so, Apparate couples per-ramp exit rates (based on profiling data from threshold tuning in §3.2) with two additional inputs that are collected once per model during bootstrapping: (1) the latency overhead per ramp, and (2) a layerwise breakdown of time spent during model inference (for different batch sizes [22]). The latter is necessary since different models can exhibit wildly different latency characteristics that govern the impact of any exits. For instance, latency of CV models is often heavily skewed towards early layers given the high dimensionality of the input data [40], while NLP transformers exhibit more consistent latency values across coding blocks. Note that, in the case of distributed serving, latency breakdowns are updated to account for network delays between serving nodes. Using these inputs, Apparate defines the utility of ramp R as savings - overheads, where savings represents the sum of raw latency that exiting inputs avoided by using ramp R, and overheads is the sum of raw latency that R added to inputs that it was unable to exit. A negative (positive) utility value means a ramp is causing more (less) harm than benefit to overall latencies for the current workload. Adding new ramps. If any negative utility values exist, Apparate applies a fast threshold tuning round to see if ramp utilities become entirely positive without harming overall latency savings. If not, Apparate immediately deactivates all negative-utility ramps. From there, the key question to address is what ramps (if any) should be added to make use of the freed ramp budget. The main difficulty is in predicting the utility of each potential addition. Indeed, while per-exit latency savings for each potential ramp are known (using the latency breakdown from above), exit rates are not. To cope with this uncertainty, our guiding intuition is that, subject to the same accuracy constraint, later ramps almost always exhibit higher exit rates than earlier ones. [2] The reason is that late ramps have the luxury of leveraging more of an original model’s computations when making a prediction. Importantly, this implies that a candidate ramp’s exit rate is bound by the exit rate of the closest downstream ramp. Building on this, Apparate’s controller computes an upper bound on the utility of candidate ramps as follows. To avoid inter-ramp dependencies harming ramps that are already performing well, we only consider additions after the latest positive ramp P in the model. In particular, Apparate divides the range following P into intervals separated by any negative ramps deactivated in this round. The first round of candidate ramps are those in the middle of each interval. For each candidate ramp, we compute its upper-bound exit rate as the sum of profiled exit rates for the following deactivated ramp and any earlier deactivations (Figure 12); the idea is that inputs from earlier deactivations would have reached the following deactivated ramp and might have exited there. Utility scores are then computed as above, and the ramp with the highest positive utility score is selected for trial. If all ramps have negative projected utilities, Apparate repeats this process for later candidate ramps in each interval. Once a ramp is selected for trial, Apparate adds it to the deployed model definition, while removing deactivated ramps. Trialed ramps start with threshold=0 to prevent inaccurate exiting, but are soon updated in the next round of threshold tuning. Until now, we have only discussed how Apparate handles scenarios with at least one negative ramp utility. In the event that all ramps exhibit positive utilities, Apparate enters a lowrisk probing phase to determine if latency savings can grow by injecting (or shifting to) earlier ramps. There are two scenarios for this. If ramp budget remains, we add a ramp immediately before the existing ramp with highest utility (while keeping that ramp to preserve its exiting wins). In contrast, if no ramp budget remains, we shift the ramp with the lowest utility score one position earlier, leaving the most positive ramp untouched. Additions are incorporated into the configuration using the same process as above. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. [2] For a single input, being able to exit at an early ramp does not guarantee exit capabilities at later ramps [34]. However, later-ramp exit rates were always higher than earlier ones for our workloads, and we note that Apparate only uses this property for search efficiency, not correctness. Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Authors: Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.2 Early-Exit Models 2.3 Challenges 2.3 Challenges 3 Design 3 Design 3.1 Preparing Models with Early Exits 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 3.3 Latency-Focused Ramp Adjustments 4 Implementation 4 Implementation 5 Evaluation and 5.1 Methodology 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 5.4 Microbenchmarks 6 Additional Related Work 6 Additional Related Work 7 Conclusion, References, Appendix 7 Conclusion, References, Appendix 3.3 Latency-Focused Ramp Adjustments The set of active ramps ultimately dictates where inputs can exit, and thus provides bounds on potential latency savings. Unlike threshold tuning which runs reactively (since accuracy is a constraint) and uses only recent profiling data to evaluate new configurations, ramp adjustment is strictly an optimization (for latency savings), and requires deployment to evaluate the impact of any new ramp. Thus, Apparate’s ramp tuning runs periodically (every 128 samples by default) and conservatively alters the set of active ramps to incrementally converge on high-performing configurations. Evaluating active ramps. In each round, Apparate’s controller starts by computing a utility score for each active ramp that evaluates its overall impact on workload latency. To do so, Apparate couples per-ramp exit rates (based on profiling data from threshold tuning in §3.2) with two additional inputs that are collected once per model during bootstrapping: (1) the latency overhead per ramp, and (2) a layerwise breakdown of time spent during model inference (for different batch sizes [22]). The latter is necessary since different models can exhibit wildly different latency characteristics that govern the impact of any exits. For instance, latency of CV models is often heavily skewed towards early layers given the high dimensionality of the input data [40], while NLP transformers exhibit more consistent latency values across coding blocks. Note that, in the case of distributed serving, latency breakdowns are updated to account for network delays between serving nodes. Evaluating active ramps. Using these inputs, Apparate defines the utility of ramp R as savings - overheads, where savings represents the sum of raw latency that exiting inputs avoided by using ramp R, and overheads is the sum of raw latency that R added to inputs that it was unable to exit. A negative (positive) utility value means a ramp is causing more (less) harm than benefit to overall latencies for the current workload. Adding new ramps. If any negative utility values exist, Apparate applies a fast threshold tuning round to see if ramp utilities become entirely positive without harming overall latency savings. If not, Apparate immediately deactivates all negative-utility ramps. From there, the key question to address is what ramps (if any) should be added to make use of the freed ramp budget. The main difficulty is in predicting the utility of each potential addition. Indeed, while per-exit latency savings for each potential ramp are known (using the latency breakdown from above), exit rates are not. To cope with this uncertainty, our guiding intuition is that, subject to the same accuracy constraint, later ramps almost always exhibit higher exit rates than earlier ones. [2] The reason is that late ramps have the luxury of leveraging more of an original model’s computations when making a prediction. Importantly, this implies that a candidate ramp’s exit rate is bound by the exit rate of the closest downstream ramp. Building on this, Apparate’s controller computes an upper bound on the utility of candidate ramps as follows. To avoid inter-ramp dependencies harming ramps that are already performing well, we only consider additions after the latest positive ramp P in the model. In particular, Apparate divides the range following P into intervals separated by any negative ramps deactivated in this round. The first round of candidate ramps are those in the middle of each interval. P P For each candidate ramp, we compute its upper-bound exit rate as the sum of profiled exit rates for the following deactivated ramp and any earlier deactivations (Figure 12); the idea is that inputs from earlier deactivations would have reached the following deactivated ramp and might have exited there. Utility scores are then computed as above, and the ramp with the highest positive utility score is selected for trial. If all ramps have negative projected utilities, Apparate repeats this process for later candidate ramps in each interval. Once a ramp is selected for trial, Apparate adds it to the deployed model definition, while removing deactivated ramps. Trialed ramps start with threshold=0 to prevent inaccurate exiting, but are soon updated in the next round of threshold tuning. Until now, we have only discussed how Apparate handles scenarios with at least one negative ramp utility. In the event that all ramps exhibit positive utilities, Apparate enters a lowrisk probing phase to determine if latency savings can grow by injecting (or shifting to) earlier ramps. There are two scenarios for this. If ramp budget remains, we add a ramp immediately before the existing ramp with highest utility (while keeping that ramp to preserve its exiting wins). In contrast, if no ramp budget remains, we shift the ramp with the lowest utility score one position earlier, leaving the most positive ramp untouched. Additions are incorporated into the configuration using the same process as above. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv [2] For a single input, being able to exit at an early ramp does not guarantee exit capabilities at later ramps [34]. However, later-ramp exit rates were always higher than earlier ones for our workloads, and we note that Apparate only uses this property for search efficiency, not correctness.

Apparate: Early-Exit Models for ML Latency and Throughput Optimization - Latency-Focused Adjustments

Authors:
(1) Yinwei Dai, Princeton University (Equal contributions);
(2) Rui Pan, Princeton University (Equal contributions);
(3) Anand Iyer, Georgia Institute of Technology;
(4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.3 Challenges 3 Design 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 4 Implementation 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 6 Additional Related Work 7 Conclusion, References, Appendix 3.2 Accuracy-Aware Threshold Tuning To avoid accuracy drops as workload characteristics change over time, Apparate’s controller employs frequent and fast tuning of thresholds for already-enabled ramps. The reason is that threshold tuning for any set of ramps is sufficient to ensure that user-specified accuracy constraints are not violated – at the extreme, all thresholds could be set to zero, which precludes any early exiting. Altering only the set of active ramps fails to provide this property. To enable threshold tuning, as requests pass through a model, Apparate’s controller continually records exiting information at each active ramp, as well as the final result that the original model predicts. More precisely, for each ramp, Apparate records the result (e.g., classification label) with the lowest error rate, i.e., the highest-confidence result for the ramp, even if the error exceeds the ramp’s threshold (precluding exiting). Importantly, since inputs always pass fully through models with Apparate, this information is recorded for all inputs at each active ramp, irrespective of upstream exiting decisions. This is paramount since the information serves not only as a signal for when to tune thresholds, but also provides guidance for how to do so. Triggering tuning. Apparate maintains an average achieved accuracy over the past 16 samples by comparing exiting results with the deployed configuration to results of the original model. Threshold tuning is triggered any time a window’s accuracy falls below the user-specified accuracy constraint. The threshold tuning process (described below) runs asynchronously on a CPU, without any disruptions to ongoing jobs. This is possible since thresholds are anyway enforced only by Apparate’s controller; GPUs are agnostic to threshold values, and instead simply stream ramp results to the Apparate controller which determines exiting decisions. Evaluating threshold configurations. Threshold tuning requires insight into how any alterations to active ramp thresholds would affect overall model exiting behavior (and accuracies). The aforementioned per-request, per-ramp monitoring information grants this visibility, enabling Apparate to rapidly evaluate any threshold values across active ramps without requiring additional inference, and while accounting for inter-ramp dependencies. In particular, to evaluate new threshold values, Apparate simply identifies the earliest ramp whose top prediction now has an error rate below its threshold. Comparing these results with those of the original model indicates the achieved accuracy for the new configuration; latency wins for these exit patterns are computed using the one-time profiling data described in §3.3. Instead, Apparate employs a greedy heuristic (Algorithm 1 in §A) that leverages a fundamental property of EEs when evaluated against an original model: higher thresholds result in monotonic decreases in accuracy and monotonic increases in latency savings. This prunes the space of threshold values to consider by providing a clear boundary in the Rdimensional space that separates configurations that are sufficiently accurate from those that are not. Additionally, for accurate configurations, maximum latency savings must fall on that boundary. Figure 10 illustrates this. These properties inform Apparate’s hill climbing strategy [47] for threshold tuning. Starting with threshold values of 0 for each active ramp, and a step size of 0.1, threshold tuning runs in a series of (incremental) exploration rounds. In each round, we increase the threshold of each ramp in isolation (leaving the others unchanged), and evaluate the achieved accuracy and latency savings as described above. Apparate then chooses the single ramp threshold change that delivered the largest additional latency savings per unit of additional accuracy loss. This process repeats until no ramp’s threshold can be increased without an accuracy violation. To enhance this process, Apparate follows a multiplicative increase, multiplicative decrease policy on step sizes to balance search speed and granularity. Specifically, each time a step increase results in an accuracy violation, Apparate halves that ramp’s step size for subsequent rounds to hone in on the boundary at fine granularity; step sizes are lower-bounded at 0.01. Conversely, selection of a ramp for threshold alteration suggests a potentially promising path of exploration; in this case, for a speedup, Apparate doubles that ramp’s step size for the following round. Overall, as shown in Figure 11, Apparate’s threshold tuning algorithm runs up to 3 orders of magnitude faster than a pure grid search (11.9ms vs. 3.0s on average). Note that these results maximally parallelize grid search across a 30-core machine. Further, selected threshold values achieve within 0-3.8% of the latency savings of the optimal configurations. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Authors: Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.2 Early-Exit Models 2.3 Challenges 2.3 Challenges 3 Design 3 Design 3.1 Preparing Models with Early Exits 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 3.3 Latency-Focused Ramp Adjustments 4 Implementation 4 Implementation 5 Evaluation and 5.1 Methodology 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 5.4 Microbenchmarks 6 Additional Related Work 6 Additional Related Work 7 Conclusion, References, Appendix 7 Conclusion, References, Appendix 3.2 Accuracy-Aware Threshold Tuning To avoid accuracy drops as workload characteristics change over time, Apparate’s controller employs frequent and fast tuning of thresholds for already-enabled ramps. The reason is that threshold tuning for any set of ramps is sufficient to ensure that user-specified accuracy constraints are not violated – at the extreme, all thresholds could be set to zero, which precludes any early exiting. Altering only the set of active ramps fails to provide this property. To enable threshold tuning, as requests pass through a model, Apparate’s controller continually records exiting information at each active ramp, as well as the final result that the original model predicts. More precisely, for each ramp, Apparate records the result (e.g., classification label) with the lowest error rate, i.e., the highest-confidence result for the ramp, even if the error exceeds the ramp’s threshold (precluding exiting). Importantly, since inputs always pass fully through models with Apparate, this information is recorded for all inputs at each active ramp, irrespective of upstream exiting decisions. This is paramount since the information serves not only as a signal for when to tune thresholds, but also provides guidance for how to do so. Triggering tuning . Apparate maintains an average achieved accuracy over the past 16 samples by comparing exiting results with the deployed configuration to results of the original model. Threshold tuning is triggered any time a window’s accuracy falls below the user-specified accuracy constraint. The threshold tuning process (described below) runs asynchronously on a CPU, without any disruptions to ongoing jobs. This is possible since thresholds are anyway enforced only by Apparate’s controller; GPUs are agnostic to threshold values, and instead simply stream ramp results to the Apparate controller which determines exiting decisions. Triggering tuning Evaluating threshold configurations. Threshold tuning requires insight into how any alterations to active ramp thresholds would affect overall model exiting behavior (and accuracies). The aforementioned per-request, per-ramp monitoring information grants this visibility, enabling Apparate to rapidly evaluate any threshold values across active ramps without requiring additional inference, and while accounting for inter-ramp dependencies. In particular, to evaluate new threshold values, Apparate simply identifies the earliest ramp whose top prediction now has an error rate below its threshold. Comparing these results with those of the original model indicates the achieved accuracy for the new configuration; latency wins for these exit patterns are computed using the one-time profiling data described in §3.3. Evaluating threshold configurations. Instead, Apparate employs a greedy heuristic (Algorithm 1 in §A) that leverages a fundamental property of EEs when evaluated against an original model: higher thresholds result in monotonic decreases in accuracy and monotonic increases in latency savings. This prunes the space of threshold values to consider by providing a clear boundary in the Rdimensional space that separates configurations that are sufficiently accurate from those that are not. Additionally, for accurate configurations, maximum latency savings must fall on that boundary. Figure 10 illustrates this. These properties inform Apparate’s hill climbing strategy [47] for threshold tuning. Starting with threshold values of 0 for each active ramp, and a step size of 0.1, threshold tuning runs in a series of (incremental) exploration rounds. In each round, we increase the threshold of each ramp in isolation (leaving the others unchanged), and evaluate the achieved accuracy and latency savings as described above. Apparate then chooses the single ramp threshold change that delivered the largest additional latency savings per unit of additional accuracy loss. This process repeats until no ramp’s threshold can be increased without an accuracy violation. To enhance this process, Apparate follows a multiplicative increase, multiplicative decrease policy on step sizes to balance search speed and granularity. Specifically, each time a step increase results in an accuracy violation, Apparate halves that ramp’s step size for subsequent rounds to hone in on the boundary at fine granularity; step sizes are lower-bounded at 0.01. Conversely, selection of a ramp for threshold alteration suggests a potentially promising path of exploration; in this case, for a speedup, Apparate doubles that ramp’s step size for the following round. Overall, as shown in Figure 11, Apparate’s threshold tuning algorithm runs up to 3 orders of magnitude faster than a pure grid search (11.9ms vs. 3.0s on average). Note that these results maximally parallelize grid search across a 30-core machine. Further, selected threshold values achieve within 0-3.8% of the latency savings of the optimal configurations. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Apparate: Early-Exit Models for ML Latency and Throughput Optimization - Accurate Threshold Tuning

Authors:
(1) Yinwei Dai, Princeton University (Equal contributions);
(2) Rui Pan, Princeton University (Equal contributions);
(3) Anand Iyer, Georgia Institute of Technology;
(4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.3 Challenges 3 Design 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 4 Implementation 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 6 Additional Related Work 7 Conclusion, References, Appendix 3.1 Preparing Models with Early Exits Upon job registration with a generic DNN, Apparate’s initial task is to automatically prepare that model to leverage EEs, without requiring any developer effort. This phase repeats any time the submitted model changes, e.g., continual retraining to cope with data drift [13, 35, 51]. Ramp locations. Apparate accepts a model in the ONNX format, a widely used IR that represents the computation as a directed acyclic graph [6]. Once ingested, Apparate must first identify candidate layers for ramp addition. The goal is to maximize ramp coverage across the model (to provide more configuration options for Apparate’s runtime management), while avoiding ramps that are unlikely to be fruitful (but add complexity adaptation decisions). To balance these aspects for diverse models, Apparate marks feasible ramp locations as those where operators are cut vertices, i.e., a vertex whose removal would disconnect a graph into two or more disjoint sub-graphs. In other words, no edge can start before a ramp and re-enter the model’s computation after the ramp. The idea is that such ramps take advantage of all available data outputs from the original model’s processing to that point, boosting their chance at accurate predictions. As an example, consider families like ResNet or BERT which enable deep models by stitching together series of residual blocks, i.e., ResNet blocks for convolutions, or BERT encoders that each embed multi-head attention and feed-forward network residual blocks. To avoid performance degradations late in the model, the output of each block is ultimately a combination of its processing results and its input. In such scenarios, Apparate injects ramps between blocks, but not within each block to avoid ramps making decisions on partial data, i.e., ignoring block inputs. In contrast, for VGG models, ramps are feasible at all layers since their intermediates represent the full extent of data flow throughout the model. Figure 8 depicts these examples. Overall, this strategy results in 9.2-68.4% of layers having ramps for the models in our corpus, which we empirically observe is sufficient to adapt to dynamic workloads (§5.2). However, we note that Apparate can directly support any other ramp configuration strategy, and offers a simple API for developers to express ramp policies or restrictions. Moreover, for each feasible ramp, Apparate’s runtime monitoring (§3.2-3.3) ultimately monitors its accuracy and utility (and thus, if it is ever activated) for the workload at hand. Ramp architectures. For each feasible ramp location, Apparate must determine the style of ramp computations to use. Recall from §2 that ramps can ultimately be composed of arbitrary layers and computations, with the only prerequisite being that the final layer sufficiently mimics that of the original model to ensure that response formats match. Determining the appropriate ramp complexity in this large space presents a tradeoff: additional computation can improve the exit capabilities of a ramp, but comes at the expense of (1) increased ramp latency, and (2) coarser flexibility and coverage at runtime since ramps become illogical if their computation exceeds that in the original model up until the next ramp. Apparate opts for the shallowest ramps that can transform the intermediates at any layer into a final model prediction. Specifically, ramps comprise the model’s final fullyconnected (fc) layer, prepended with a lightweight pooling operation that reduces the dimensionality of intermediates to ensure compatability with the fc layer. This manifests differently for various model types. For instance, for vision models like ResNet, pooling is simply the model’s penultimate layer. In contrast, for BERT, only the basic operator is drawn from the BERT pooler module, i.e., extracting the hidden state corresponding to the first token [19]. Note that, for all models, the input width of the fc layer is modified to match that of the intermediates at each ramp location; the output remains unchanged to preserve result formats. Figure 9 evaluates this methodology by comparing Apparate’s ramps with two, more expensive alternatives. With ResNet, to mimic model operations following each ramp, we add 1-2 convolution layers prior to pooling. For BERT, we consider two approaches: (1) add two fc layers after pooling, each with reduced width to shrink inputs to the final fc, and (2) inspired by DeeBERT [57], replacing the simple pooling operator with the entire BERT pooler block (which also includes dense linear and activation layers) and adding a dropout after pooling as in the original model. In all cases, the number of ramps is subject to the same ramp budget (i.e., Apparate’s default uses the most ramps), ramps are uniformly spaced across feasible positions in each model, and thresholds are optimally selected as in §2.2. As shown, and in line with reflections from prior handdesigned EE models [53], we observe that the additional computation has minimal effect on ramp effectiveness. For example, median latencies are 1.3-1.8× and 1.9-5.4× larger with Apparate’s default ramps than the more complex alternatives for CV and NLP. Nonetheless, to showcase Apparate’s generality, we consider other ramp styles in §5.4. Training ramps and deploying models. To determine the appropriate weights for each ramp, Apparate relies on an initial dataset that can either be user-provided or auto-generated by running the original model on historical data. Regardless, during training, Apparate freezes the original model weights to ensure that non-EE behavior and feedback for tuning EEs is unchanged from the user’s original intentions. In addition, Apparate enforces that all inputs are used to train all ramps, i.e., exiting is prohibited during training. This ensures that ramps are trained independently of the presence (or behavior) of any upstream ramps, which is crucial since the set of active ramps can vary at runtime. Further, such independence and the model freezing enable loss calculations to be backwards propagated in parallel across ramps, rapidly speeding up training despite Apparate’s use of many lightweight ramps. §4 details the training process. For initial deployment, Apparate evenly spaces the max number of allowable ramps (based on the budget and GPU resources) across the model. To avoid accuracy dips due to discrepancies between training data and the current workload, each ramp begins with a threshold of 0, i.e., no exiting. The updated model definition (with enabled ramps) is passed to the serving platform which operates as normal, e.g., profiling expected model runtimes for different batch sizes [22]. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Authors: Authors: (1) Yinwei Dai, Princeton University (Equal contributions); (2) Rui Pan, Princeton University (Equal contributions); (3) Anand Iyer, Georgia Institute of Technology; (4) Ravi Netravali, Georgia Institute of Technology. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background and Motivation and 2.1 Model Serving Platforms 2 Background and Motivation and 2.1 Model Serving Platforms 2.2 Early-Exit Models 2.2 Early-Exit Models 2.3 Challenges 2.3 Challenges 3 Design 3 Design 3.1 Preparing Models with Early Exits 3.1 Preparing Models with Early Exits 3.2 Accuracy-Aware Threshold Tuning 3.2 Accuracy-Aware Threshold Tuning 3.3 Latency-Focused Ramp Adjustments 3.3 Latency-Focused Ramp Adjustments 4 Implementation 4 Implementation 5 Evaluation and 5.1 Methodology 5 Evaluation and 5.1 Methodology 5.2 Overall Results 5.2 Overall Results 5.3 Comparison with Existing EE Strategies 5.3 Comparison with Existing EE Strategies 5.4 Microbenchmarks 5.4 Microbenchmarks 6 Additional Related Work 6 Additional Related Work 7 Conclusion, References, Appendix 7 Conclusion, References, Appendix 3.1 Preparing Models with Early Exits Upon job registration with a generic DNN, Apparate’s initial task is to automatically prepare that model to leverage EEs, without requiring any developer effort. This phase repeats any time the submitted model changes, e.g., continual retraining to cope with data drift [13, 35, 51]. Ramp locations. Apparate accepts a model in the ONNX format, a widely used IR that represents the computation as a directed acyclic graph [6]. Once ingested, Apparate must first identify candidate layers for ramp addition. The goal is to maximize ramp coverage across the model (to provide more configuration options for Apparate’s runtime management), while avoiding ramps that are unlikely to be fruitful (but add complexity adaptation decisions). To balance these aspects for diverse models, Apparate marks feasible ramp locations as those where operators are cut vertices, i.e., a vertex whose removal would disconnect a graph into two or more disjoint sub-graphs. In other words, no edge can start before a ramp and re-enter the model’s computation after the ramp. Ramp locations. The idea is that such ramps take advantage of all available data outputs from the original model’s processing to that point, boosting their chance at accurate predictions. As an example, consider families like ResNet or BERT which enable deep models by stitching together series of residual blocks, i.e., ResNet blocks for convolutions, or BERT encoders that each embed multi-head attention and feed-forward network residual blocks. To avoid performance degradations late in the model, the output of each block is ultimately a combination of its processing results and its input. In such scenarios, Apparate injects ramps between blocks, but not within each block to avoid ramps making decisions on partial data, i.e., ignoring block inputs. In contrast, for VGG models, ramps are feasible at all layers since their intermediates represent the full extent of data flow throughout the model. Figure 8 depicts these examples. Overall, this strategy results in 9.2-68.4% of layers having ramps for the models in our corpus, which we empirically observe is sufficient to adapt to dynamic workloads (§5.2). However, we note that Apparate can directly support any other ramp configuration strategy, and offers a simple API for developers to express ramp policies or restrictions. Moreover, for each feasible ramp, Apparate’s runtime monitoring (§3.2-3.3) ultimately monitors its accuracy and utility (and thus, if it is ever activated) for the workload at hand. Ramp architectures. For each feasible ramp location, Apparate must determine the style of ramp computations to use. Recall from §2 that ramps can ultimately be composed of arbitrary layers and computations, with the only prerequisite being that the final layer sufficiently mimics that of the original model to ensure that response formats match. Determining the appropriate ramp complexity in this large space presents a tradeoff: additional computation can improve the exit capabilities of a ramp, but comes at the expense of (1) increased ramp latency, and (2) coarser flexibility and coverage at runtime since ramps become illogical if their computation exceeds that in the original model up until the next ramp. Ramp architectures. Apparate opts for the shallowest ramps that can transform the intermediates at any layer into a final model prediction. Specifically, ramps comprise the model’s final fullyconnected (fc) layer, prepended with a lightweight pooling operation that reduces the dimensionality of intermediates to ensure compatability with the fc layer. This manifests differently for various model types. For instance, for vision models like ResNet, pooling is simply the model’s penultimate layer. In contrast, for BERT, only the basic operator is drawn from the BERT pooler module, i.e., extracting the hidden state corresponding to the first token [19]. Note that, for all models, the input width of the fc layer is modified to match that of the intermediates at each ramp location; the output remains unchanged to preserve result formats. Figure 9 evaluates this methodology by comparing Apparate’s ramps with two, more expensive alternatives. With ResNet, to mimic model operations following each ramp, we add 1-2 convolution layers prior to pooling. For BERT, we consider two approaches: (1) add two fc layers after pooling, each with reduced width to shrink inputs to the final fc, and (2) inspired by DeeBERT [57], replacing the simple pooling operator with the entire BERT pooler block (which also includes dense linear and activation layers) and adding a dropout after pooling as in the original model. In all cases, the number of ramps is subject to the same ramp budget (i.e., Apparate’s default uses the most ramps), ramps are uniformly spaced across feasible positions in each model, and thresholds are optimally selected as in §2.2. As shown, and in line with reflections from prior handdesigned EE models [53], we observe that the additional computation has minimal effect on ramp effectiveness. For example, median latencies are 1.3-1.8× and 1.9-5.4× larger with Apparate’s default ramps than the more complex alternatives for CV and NLP. Nonetheless, to showcase Apparate’s generality, we consider other ramp styles in §5.4. Training ramps and deploying models. To determine the appropriate weights for each ramp, Apparate relies on an initial dataset that can either be user-provided or auto-generated by running the original model on historical data. Regardless, during training, Apparate freezes the original model weights to ensure that non-EE behavior and feedback for tuning EEs is unchanged from the user’s original intentions. In addition, Apparate enforces that all inputs are used to train all ramps, i.e., exiting is prohibited during training. This ensures that ramps are trained independently of the presence (or behavior) of any upstream ramps, which is crucial since the set of active ramps can vary at runtime. Further, such independence and the model freezing enable loss calculations to be backwards propagated in parallel across ramps, rapidly speeding up training despite Apparate’s use of many lightweight ramps. §4 details the training process. Training ramps and deploying models. For initial deployment, Apparate evenly spaces the max number of allowable ramps (based on the budget and GPU resources) across the model. To avoid accuracy dips due to discrepancies between training data and the current workload, each ramp begins with a threshold of 0, i.e., no exiting. The updated model definition (with enabled ramps) is passed to the serving platform which operates as normal, e.g., profiling expected model runtimes for different batch sizes [22]. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

Stories, News & Mentions about NETRA

Netra’s Stories on HackerNoon