Southeast Texas citizens are no strangers to inclement weather and lengthy power outages in the aftermath. However, with these upgrades that may be changing soon.

Entergy completes $5M project to upgrade Port Arthur's power infrastructure

Get the most recent info and news about Entergy on HackerNoon, where 10k+ technologists publish stories for 4M+ monthly readers.

Entergy Corporation, together with its subsidiaries, engages in the production and retail distribution of electricity in the United States. The company operates in two segments, Utility and Entergy Wholesale Commodities. The Utility segment generates, transmits, distributes, and sells electric power in portions of Arkansas, Louisiana, Mississippi, and Texas, including the City of New Orleans; and distributes natural gas. The Entergy Wholesale Commodities segment engages in the ownership, operation, and decommissioning of nuclear power plants; and ownership of interests in non-nuclear power plants that sell electric power to wholesale customers, as well as provides services to other nuclear power plant owners. It generates electricity through gas, nuclear, coal, hydro, and solar power sources. The company sells energy to retail power providers, utilities, electric power co-operatives, power trading organizations, and other power generation companies. The company's power plants have approximately 24,000 megawatts (MW) of electric generating capacity, which include 5,000 MW of nuclear power. It delivers electricity to 3 million utility customers in Arkansas, Louisiana, Mississippi, and Texas. The company was founded in 1913 and is headquartered in New Orleans, Louisiana.

Entergy

Prestidigitation! President Biden’s Energy Boom, the Sop to Cerberus, and Stein's Law

প্রেস্টিডিটেশন ! প্রেসিডেন্ট বিডেনের এনার্জি বুম, দ্য সোপ টু সার্বেরাস এবং স্টেইনের আইন

Taschenspielereien! Präsident Bidens Energieboom, die Beschwichtigung von Cerberus und Steins Gesetz

¡Prestidigitación! El auge energético del presidente Biden, la concesión a Cerberus y la ley de Stein

Prestidigitation! Le boom énergétique du président Biden, la solution à Cerberus et la loi de Stein

प्रेस्टीजिटेशन! राष्ट्रपति बिडेन की ऊर्जा उछाल, सेर्बेरस को रियायत, और स्टीन का नियम

手品！バイデン大統領のエネルギーブーム、ケルベロスへのご褒美、そしてスタインの法則

요술! 바이든 대통령의 에너지 붐, 케르베로스의 Sop, 그리고 스타인의 법칙

Prestidigitação! O boom energético do presidente Biden, o golpe para Cerberus e a lei de Stein

Престидигитация! Энергетический бум президента Байдена, подачка Церберу и закон Штейна

Prestijitasyon! Başkan Biden'ın Enerji Patlaması, Cerberus'a Sop ve Stein Yasası

Uy tín kỹ thuật số! Sự bùng nổ năng lượng của Tổng thống Biden, Sop to Cerberus và Định luật Stein

魔术！拜登总统的能源繁荣、塞伯鲁斯的贿赂和斯坦因定律

The Noonification: The Energy-Inefficient AI Era Is Already Here: The Cost of AI (4/5/2024)

Artificial intelligence is exciting; the fresh technological breakthrough that took over our lives in less than a year hasn't stopped since. Personally, I have some issues with AI tools and how they are being used at the moment. With that, I cannot deny that they are a mighty instrument when given to the right person to do the right job - but as we all know, with great power comes great responsibility. Amidst the excitement and semi-fulfilled promises - did we ever stop to think about the cost of running 1,2,3 billion parameter models for hours on end, scaled across multiple instances, for millions of users, all at once? I was always an advocate of the "minimal web" concept (I will write an article about the concept in the future; there is a lot to say). We live in an era where on one side, everyone seems extremely concerned about the health of the planet (and for good reason). Yet on the other hand, no one is stopping to think how much the internet in its current state is costing us in terms of raw energy. Is it efficient? Is it green? Far from it. Traffic and Energy A while ago, I wrote a tiny TUI tool using Perl ( ), that monitors my internet traffic and aggregates the byte-length of packets that are sent and received while the tool is running. webmeter Just by opening Reddit's front page, my tool logged a whopping 5.85 megabytes of data transferred between my interface and Reddit's servers. Reddit as a website is a combination of text, low-to-mid-quality images, and the occasional video. It seems extremely unreasonable that just by opening Reddit's front page I transferred so much data across the planet (auto-playing videos were off during testing, this result excludes video content). Reddit's front page, a page that could have concluded with 10 kilobytes of content, javascript, and CSS (perhaps a few more kilobytes for additional images), needed 300 times the bandwidth just to load up. In comparison (considering the moon landing did not take place in a Hollywood basement), the software written for the Apollo 11 mission was only 72 kilobytes in size. Just for the kick of it, imagine billions of people, opening Reddit's front page, all together, at every moment, all the time. Try and think: what is the direct cost of transferring the data? What is the secondary cost? How much energy are we wasting as internet-consuming beings? This concept has been dwelling in my mind for the past year or so which was the main reason I hacked together my Perl tool in the first place. One "minimal web" thought led to another, and I realized that we are so consumed by the hype (or hate) that we are ignoring the cost of our artificial intelligence endeavors. How Can We Calculate the Energy Cost of AI As a Service (or Stable Diffusion for Example)? Accurately? We can't. Not without knowing how these services operate: which CPUs or GPUs (or dedicated chips) they are using. How many instances are they running, what is their load, what is the exact power consumption of the critical hardware, etc? But we can come to a pretty realistic (although not as accurate) conclusion if we put our minds to it. I chose for this example which is the most straightforward (and I believe popular) stable diffusion GUI out there. Let's start simple: Fooocus Using Fooocus, it took my old-timer GPU (a Quadro P2000) about 10 minutes to generate an image. According to the performance table on Fooocus's repository, we know that using an Nvidia RTX 4XXX GPU would give us the fastest result. My Quadro is somewhat comparable to an Nvidia GTX 1060, therefore, we know that using an Nvidia GTX 1060 GPU, we can generate an image in approximately 10 minutes. After some searching online, I concluded that the most commonly used (enterprise) GPU for AI/ML is the Nvidia A100 (up to 400W under load). The Nvidia A100 is a massively more powerful GPU than the Nvidia GTX 1060; it is also built specifically for AI and ML. Therefore, let's assume that generating an image using an Nvidia A100 will take 5 seconds (if anyone reading this has accurate data about the A100 and its performance, please share). Using the formula E = P * T, an Nvidia A100 GPU power consumption under load for 5 seconds costs around 0.5 watt-hours. Adding on to this example: Midjourney (or other AI image generation services) generates 4 images per prompt. Let's assume the service is using an array of Nvidia A100s to generate the images which means that the service is probably wasting around 2 watt-hours of energy for each prompt, for each user. As of November 2023, Midjourney has 2.5 million DAUs. 2Wh x 2,500,000 Users x 24 Hours = 120,000,000Wh (per day, considering every DAU is executing a prompt per hour, in reality, it is much more than that).
120,000,000Wh = 120,000MWh 120,000 Mega watt hours could potentially power over 100,000 households for a whole month. Modestly, services such as Midjourney, Dall-E, or even self-hosted solutions such as Fooocus, waste at the very least 50 million watt-hours per day, each (considering their GPUs are state-of-the-art and each DAU uses the service for a few hours every day). This amount of energy could potentially power over 100,000 average-sized households for a month, per day. Keep in mind that we only talked about image generation services; let's not forget about all other services, such as ChatGPT, Gemini, Bard, and all of their versions and flavors. The list, and waste, goes on. What Should We Do? How should we, as developers, come together to address this emerging issue? How can we take measures to educate others to use energy responsibly? I would love to hear your thoughts about this topic and its possible solutions. Also published here

The Energy-Inefficient AI Era Is Already Here: The Cost of AI

শক্তি-অদক্ষ এআই যুগ ইতিমধ্যেই এখানে: এআই-এর খরচ

Das Zeitalter der energieineffizienten KI ist bereits angebrochen: Die Kosten der KI

La era de la IA energéticamente ineficiente ya está aquí: el coste de la IA

L’ère de l’IA économe en énergie est déjà là : le coût de l’IA

ऊर्जा-अकुशल एआई युग पहले ही आ चुका है: एआई की लागत

エネルギー効率の悪い AI 時代はすでに到来: AI のコスト

에너지 비효율적인 AI 시대는 이미 도래했습니다: AI의 비용

A era da IA com ineficiência energética já chegou: o custo da IA

Эра энергоэффективного искусственного интеллекта уже наступила: стоимость искусственного интеллекта

Enerji Açısından Verimsiz Yapay Zeka Çağı Zaten Geldi: Yapay Zekanın Maliyeti

Kỷ nguyên AI sử dụng năng lượng kém đã đến: Chi phí của AI

能源效率低下的人工智能时代已经到来：人工智能的成本

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Minghao Yan, University of Wisconsin-Madison; (2) Hongyi Wang, Carnegie Mellon University; (3) Shivaram Venkataraman, myan@cs.wisc.edu. Table of Links Abstract & Introduction Motivation Opportunities Architecture Overview Proble Formulation: Two-Phase Tuning Modeling Workload Interference Experiments Conclusion & References A. Hardware Details B. Experimental Results C. Arithmetic Intensity D. Predictor Analysis B EXPERIMENTAL RESULTS In this section, we further demonstrate the tradeoff between memory frequency and maximum GPU frequency by presenting an array of results. These results underline the interesting observation that the energy consumption patterns vary for the same model operating on different devices. Furthermore, even for the same model device pairing, the optimization landscape can be significantly influenced by the batch size. This underlines the complexities of energy optimization and the need for an adaptive framework that can take these factors into account. Figures 6 − 12 show the energy consumption patterns of EfficientNet and Bert on Jetson TX2 and Orin under various batch sizes. Table 7 shows the optimal CPU frequency and corresponding energy consumption reduction in image preprocessing.

PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices: Experimental Results

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Minghao Yan, University of Wisconsin-Madison; (2) Hongyi Wang, Carnegie Mellon University; (3) Shivaram Venkataraman, myan@cs.wisc.edu. Table of Links Abstract & Introduction Motivation Opportunities Architecture Overview Proble Formulation: Two-Phase Tuning Modeling Workload Interference Experiments Conclusion & References A. Hardware Details B. Experimental Results C. Arithmetic Intensity D. Predictor Analysis 3 OPPORTUNITIES In this section, we perform empirical experiments to uncover new opportunities for optimizing energy use in NN inference. As discussed in Section 2, prior work did not study how memory frequency, minimum GPU frequency, and CPU frequency play a role in energy consumption. This is partially limited by hardware constraints. Specialized power rails need to be built into the device during manufacturing to enable accurate measurement of energy consumption associated with each component. We leverage two Jetson developer kits, TX2 and Orin, which offer native support for component-wise energy consumption measurement and frequency tuning, to study how these frequencies impact inference latency and energy consumption in modern deep learning workloads. We find that the default frequencies are much larger than optimal and throttling all these frequency knobs offer energy consumption reduction with minimal impact on inference latency. Figure 3 illustrates the energy optimization landscape when varying GPU and memory frequencies, without imposing any constraints on latency SLO. The plot reveals that, without any other constraints, the energy optimization landscape generally exhibits a bowl shape. However, this shape varies depending on the models, devices, and other hyperparameters, such as batch sizes (See Appendix B for more results). Next, we dive into how each hardware component affects inference energy consumption. : Experiments in this section are performed with 16-bit floating point number precision, as it has been demonstrated to have minimal impact on model accuracy in practice. We use Bert and EfficientNet models and vary the EfficientNet model size between B0, B4, B7 (Table 4). Setup For each model, we fix the GPU frequency at the optimal frequency determined by grid-search of all possible frequency configurations. We then examine the tradeoff between inference latency and energy consumption as we progressively throttle memory frequency. The range of available memory frequencies can be found in Table 1. Memory frequency experiment: : Table 2 reveals that memory frequency plays a vital role in reducing energy consumption. The savings provided by memory frequency tuning are similar and consistent across models on both hardware platforms, ranging from approximately 12% to 25%. This indicates that the Results default memory frequency is higher than optimal for modern Deep Learning workloads. For heavy workloads such as Bert, memory tuning can account for the majority of the energy consumption reduction. This can be partially attributed to the memory-bound nature of Transformer-based models (Ivanov et al., 2021). Our result demonstrates that systems that aim to optimize energy use in neural network inference need to take memory frequency into account. CPUs are only used for data pre-processing. Thus, we first measure the time spent in the data processing part of the inference pipeline. Next, we measure the energy saved by throttling the CPU frequency and assess the inference latency slowdown caused by reducing CPU frequency. The data preprocessing we perform is standard in almost all image processing and object detection pipelines, where we read the raw image file, convert it to CPU Frequency Experiment: an RGB scale, resize it, and reorient it to the desired input resolution and data layout. : The preprocessing time across different EfficientNet models remains constant since the operations performed are identical. As a result, the relative impact of CPU tuning on overall energy consumption depends on the ratio between preprocessing time and inference time. As the model size increases and inference duration increases, the influence of CPU tuning on overall energy consumption decreases. We observe that on both Jetson TX2 and Orin platforms, CPU tuning can decrease preprocessing energy consumption by approximately 30%. Depending on the model, quantization level, and batch size, this results in up to a 6% reduction in overall energy consumption. Results We maintain the default hardware configuration and only adjust the minimum GPU frequency on Jetson Orin. Increasing the minimum GPU frequency forces the GPU DVFS mechanism to operate within a smaller range. We scale the model from Minimum GPU frequency experiment: EfficientNet B0 to EfficientNet B7 to illustrate the effect of the GPU minimum frequency on inference latency. : Table 3 indicates that tuning the minimum GPU frequency can significantly reduce energy consumption when the workload cannot fully utilize the computational power of the hardware. Notably, energy consumption and inference latency are reduced by forcing the GPU to operate Results at a higher frequency. This differs from the tradeoff observed in other experiments, where we exchange inference latency for lower energy consumption. Tuning minimum GPU frequency can nearly halve the energy consumption for small models. As computational power becomes saturated with increasing model size, the return on tuning the minimum GPU frequency diminishes. Figure 4 shows the per query energy cost as we vary the minimum and maximum GPU frequency. It shows that increasing the minimum GPU frequency from the default minimum leads to lower energy costs and inference latency.

PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices: Opportunities

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Minghao Yan, University of Wisconsin-Madison; (2) Hongyi Wang, Carnegie Mellon University; (3) Shivaram Venkataraman, myan@cs.wisc.edu. Table of Links Abstract & Introduction Motivation Opportunities Architecture Overview Proble Formulation: Two-Phase Tuning Modeling Workload Interference Experiments Conclusion & References A. Hardware Details B. Experimental Results C. Arithmetic Intensity D. Predictor Analysis 2 MOTIVATION Many deep neural networks have been deployed on edge devices to perform tasks such as image classification, object detection, and dialogue systems. Scenarios including smart home assistants (He et al., 2020), inventory and supply chain monitoring (jet), and autopilot (Gog et al., 2022) often use battery-based devices that contain GPUs to perform the aforementioned tasks. In these scenarios, pre-trained models are installed on the devices where the inference workload is deployed. Prior works have focused on optimizing the energy consumption of GPUs (Wang et al., 2020b; 2021; Tang et al., 2019; Strubell et al., 2019; Mei et al., 2017) in cloud scenarios (Qiao et al., 2021; Wan et al., 2020; Hodak et al., 2019) and training settings (Wang et al., 2020a; Peng et al., 2019; Kang et al., 2022). On-device inference workloads exhibit different characteristics and warrant separate attention. In this section, we outline previous efforts in optimizing ondevice neural network inference and discuss our approach to holistically optimize energy consumption. 2.1 On-device Neural Network Deployment Prior work in optimizing on-device neural network inference focuses on quantization (Kim et al., 2021; Banner et al., 2018; Courbariaux et al., 2015; 2014; Gholami et al., 2021), designing hardware-friendly network architectures (Xu et al., 2019; Lee et al., 2019; Sanh et al., 2019; Touvron et al., 2021; Howard et al., 2019), and leveraging hardware components specific to mobile settings, such as DSPs (Lane & Georgiev, 2015). Our work explores an orthogonal dimension and aims to answer a different question: Given a neural network to deploy on a specific device, how can we tune the device to reduce energy consumption? In our work, we focus on edge devices that contain CPUs, memory, and GPUs. These devices are generally more powerful than DSPs often found on mobile devices. One such example is the Nvidia Jetson series, which is capable of handling a wide array of applications, ranging from AI to robotics and embedded IoT solutions (jet). The devices also come with dynamic voltage and frequency scaling (DVFS) capabilities that allow for the optimization of power consumption and thermal management during complex computational tasks. The Jetson series features a unified memory shared by both CPU and GPUs. We refer to the operating frequency of CPU, GPU, and shared memory as CPU frequency, GPU frequency, and memory frequency in this paper. To understand the system requirements in edge NN inference, we next describe a case study of how NNs are deployed in an inventory management company. From our conversations, Company A works with Customer B to deploy neural networks on edge devices to optimize inventory management. To comply with regulations and protect privacy, data from each inventory site are required to be stored locally. The vast difference in the layout of the inventories makes it impossible to pre-train the model on data from every warehouse. Therefore, these devices come with a pre-trained model based on data from a small sample of inventories, which may have significantly different layouts and external environments compared to the actual deployment venue. Consequently, daily fine-tuning is required to enhance performance in the deployed sites, as the environment continually evolves. Similar arguments apply to smart home devices, where a model is pre-trained on selected properties, but the deployed households may be much more diverse. To address privacy concerns, on-device Case Study on Inventory Management: fine-tuning of neural networks is preferred, as it keeps sensitive data locally. Therefore, edge devices often need to run both inference and periodic fine-tuning. Combining multiple workloads on edge devices can lead to SLO violations due to interference and increased energy use. 2.2 Holistic Energy Consumption Optimization Some recent works have explored reducing energy consumption by optimizing for batch size and GPU maximum frequency (You et al., 2022; Nabavinejad et al., 2021; Komoda et al., 2013; Gu et al., 2023) and developing power models for modern GPUs (Kandiah et al., 2021; Hong & Kim, 2010; Arafa et al., 2020; Lowe-Power et al., 2020). In this work, we argue that other hardware components also cause energy inefficiency and require separate optimization. We perform a grid search over GPU, memory, and CPU frequencies and various batch sizes to examine the Pareto frontier of inference latency and energy consumption. Figure 2 shows the tradeoff between the per-query energy consumption and inference latency (normalized to the optimal latency) on Jetson TX2 and Jetson Orin. Each point in the figure represents the optimal configuration that we find through grid search under a given inference latency budget and batch size. As Figure 2 shows, the Pareto frontier is not smooth globally and is difficult to capture by a simple model, which warrants more sophisticated optimization techniques to quickly converge to a hardware configuration that lies on the Pareto Frontier (Censor, 1977). Zeus (You et al., 2022) attempts to reduce the energy consumption of neural network training by changing the GPU power limit and tuning training batch size. PolyThrottle also includes these two factors. In Zeus (You et al., 2022), the focus is on training workloads in data center settings, where batch size tuning helps achieve an accuracy threshold in an energy-efficient way. We include batch size as part of PolyThrottle as it provides a trade-off between inference latency and throughput. Our empirical evaluation reveals new avenues available for optimization which complicates the search space, as we describe next.

PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices: Motivation

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Minghao Yan, University of Wisconsin-Madison; (2) Hongyi Wang, Carnegie Mellon University; (3) Shivaram Venkataraman, myan@cs.wisc.edu. Table of Links Abstract & Introduction Motivation Opportunities Architecture Overview Proble Formulation: Two-Phase Tuning Modeling Workload Interference Experiments Conclusion & References A. Hardware Details B. Experimental Results C. Arithmetic Intensity D. Predictor Analysis 7 EXPERIMENTS 7.1 Setup Our experiments are conducted on the Jetson TX2 Developer Kit and Jetson Orin Developer Kit. To assess the energy consumption of our program, we employ the built-in power monitors on the Jetson TX2 and Jetson Orin Developer Kits. We also cross-validate our measurements with an external digital multimeter (See Appendix A for more details on hardware and energy measurement). Hardware Platform: We base our experiments on the EfficientNet family and Bert models (Tan & Le, 2019; Devlin et al., 2018). EfficientNet is chosen not only for its status as a state-of-the-art convolutional network in on-device and mobile settings but also for its principled approach to scaling the width, depth, and resolution of convolution layers. Table 4 summarizes the scaling pattern of EfficientNet from the smallest B0 to the largest B7. We select Bert to investigate energy usage patterns in a Transformer-based model (Wolf et al., 2020), where the workload is more memory-bounded compared to convolution-based neural networks. Bert and its variants (Kim et al., 2021; Devlin et al., 2018; Sanh et al., 2019; Tambe et al., 2021) are widely used for Question and Answering tasks (Rajpurkar et al., 2016), making it applicable for numerous edge devices, such as smart home assistants and smart speakers. Workload Selection: : We evaluate PolyThrottle on real-world traffic streams data (Shen et al., 2019) and sample frames uniformly to feed into EfficientNet. For Bert, we evaluate on SQuAD (Rajpurkar et al., 2016) for Question Answering. Note that datasets do not affect PolyThrottle’s performance since inference latency would not change significantly across datasets once the model is chosen. Dataset PolyThrottle is built on the Nvidia Triton inference server. To maximize performance, we generate TensorRT kernels that profile various data layouts and tiling strategies to identify the fastest execution graph for a given hardware platform. Our modules include a Bayesian optimizer for determining the best configuration, an inference client responsible for preprocessing and submitting requests to the inference server, and a performance predictor module integrated into the inference client for scheduling fine-tuning requests. We maintain separate queues for inference and fine-tuning requests. Implementation: 7.2 Efficiently Searching for Optimal Configuration In this experiment, we carry out an extensive empirical analysis of tuning various models across different hardware configurations while also adjusting the quantization level. We perform a grid search on EfficientNet B0, B4, B7, and Bert Base to examine the potential energy savings and identify the optimal GPU and memory frequencies for each model. We also adjust the quantization level for each tested model. We evaluate 16-bit and 32-bit floating point (FP16/FP32) precision. The optimal energy consumption and configuration referenced later in this section use the results obtained here as the baseline and optimal solution. Having obtained the optimal frequency using grid search, we next evaluate the average number of attempts it takes for PolyThrottle to find a solution within 5% of the optimal solution. We compare our Constrained Bayesian Optimization (CBO) formulation against Random Search (RS). Experiment Settings: We measure the average number of attempts needed to find a near-optimal configuration. For Random Search, we calculate the expected number of trials needed to find a near-optimal configuration based on the grid size by computing the fraction of near-optimal configurations and taking the reciprocal. For CBO, we set the ξ parameter associated with the Expected Improvement function to 0.1 and initial random samples to be 5, which we find to work well across different models and hardware platforms. We conduct two experiments where we set different inference latency constraints, the results can be found in Figure 5: 1. We restrict inference latency to close to the optimal latency (20%). In this setting, the tight latency constraints make it impossible to batch the inference query, essentially reducing the search space for the optimal configuration. 2. In the second benchmark, we relax the inference latency constraint to include the configurations that provide the lowest energy-per-query in Figure 2. In this setting, we need to explore the batch size dimension to find the configuration that minimizes energy. We test on EfficientNet B0, B4, and B7, as well as Bert Base on both Jetson TX2 and Jetson Orin. Figure 5 shows that CBO outperforms RS in both scenarios. Since CBO models the relationship between hardware configuration and latency, it can find a near-optimal solution with only 5 to 15 samples. In the second scenario, the performance of RS deteriorates as it is unable to leverage the relationship between latency and batch size when dealing with a multiplicatively increasing search space. Overall CBO takes 3-10x fewer samples in the second setting. The overhead of performing CBO is also minimal. As shown in Figure 5, CBO only requires around 15 samples to find a near-optimal solution and the optimization procedure can be completed in a few minutes. In cases where a new model is deployed, only a few minutes of overhead are needed to find optimal configurations for the new model. Results: It is important to note that though RS might achieve performance comparable to CBO under certain conditions, this result is merely the expected value and the variance of RS is large. For instance, if 10 out of 200 configurations are near-optimal, the expected number of trials needed to reach a near-optimal configuration is 20, with a standard deviation of 19.49. Consequently, it’s plausible that even after 40 trials, RS might still fail to identify a near-optimal configuration. On the other hand, the standard deviation of CBO is smaller; in all experiments, CBO’s standard deviations are less than 3. 7.3 Workload-aware Fine-tuning Scheduling Next, we evaluate how well PolyThrottle handles fine-tuning requests alongside inference. The central question we aim to address is whether our performance predictor can effectively identify and adjust accordingly when the SLO requirement is at risk of being violated, and if reducing the inference batch size and trading throughput can satisfy the latency SLO. To simulate this scenario, we generate two distinct inference arrival patterns (Uniform and Poisson) and use the publicly available Twitter trace (twi, 2018) and compare our adaptive scheduling approach to greedy scheduling, where a fine-tuning request is scheduled as soon as it arrives. The three arrival patterns represent scenarios that are highly controlled and bursty, respectively. In this context, we contrast PolyThrottle’s adaptive scheduling mechanism with the greedy scheduling approach to assess the efficacy of PolyThrottle in meeting the desired SLO requirement. We evaluate on both synthetic and real workloads. For synthetic workloads, we generate a stream of inference requests using both Uniform and Poisson distributions. For real world workload, we first uniformly sample a day of Twitter streaming traces and then compute the variance of requests during each minute. We then picked the segment with the highest variance to test PolyThrottle’s capability in handling request bursts (twi, 2018; Romero et al., 2021). On Jetson Orin, we replay the stream for 30 seconds and measure the SLO violation rate during the replay using EfficientNet B7. Since each burst only lasts for a few seconds, this suffices to capture many bursts in the workload. We find that running the experiment for longer durations produces similar results. We set the fine-tuning batch size to 64, the number of fine-tuning iterations to 10, SLO to 0.7s, the output dimension to 1000, and an average of 8 inference requests per second. On Jetson TX2, we do the same experiment on EfficientNet B4. Due to memory constraints, we perform the fine-tuning batch size to 8, SLO to 1s, the output dimension to 100, and an average of 4 inference requests per second. We select a less performative model on TX2 to meet a reasonable SLO target (under 1s). The number of fine-tuning iterations is chosen based on the duration of the replay. We then measure the energy costs when deploying PolyThrottle at the default and optimal hardware frequency, respectively, to measure how much energy we save during this period. The optimal hardware frequency is obtained from results in Section 7.2. Experiment Settings: For greedy scheduling, we employ a standard drop policy (Crankshaw et al., 2017; Shen et al., 2019), whereby a request is dropped if it has already exceeded its deadline. In the adaptive setting, we use the predictor to determine whether to drop an inference request. We also replay the inference request stream without fine-tuning requests to serve as a baseline. : Table 5 and 6 show the SLO violation rates under various workloads and latency targets. The findings indicate that greedy scheduling may lead to significant SLO violations owing to the interference introduced by the fine-tuning workload. In contrast, PolyThrottle’s adaptive scheduling mechanism demonstrates the ability to achieve low SLO violation rates by dynamically adjusting configurations. The baseline figures in the table represent SLO violation rates in the absence of interference from fine-tuning requests. Results Inherent variance in neural network inference resulted in 1% of SLO violations in the case of Uniform distribution. However, bursts in the Poisson distribution and the Twitter workload generated more SLO violations. PolyThrottle’s adaptive scheduling mechanism significantly reduces the SLO violation rate, meeting the SLO requirements while concurrently handling fine-tuning requests. Nevertheless, in several instances, we were unable to achieve near-zero SLO violation rates. This limitation can be attributed to the granularity of scheduling as we process the current batch of requests over an extended timespan due to interference from the fine-tuning workload. We also by 14% on EfficientNet B7 on Jetson Orin and by 23% on EfficientNet B4 on Jetson TX2 across the workloads. We show in Appendix D how PolyThrottle reacts to changing SLOs when there are outstanding fine-tuning requests. reduce energy consumption

PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices: Experiments

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Minghao Yan, University of Wisconsin-Madison; (2) Hongyi Wang, Carnegie Mellon University; (3) Shivaram Venkataraman, myan@cs.wisc.edu. Table of Links Abstract & Introduction Motivation Opportunities Architecture Overview Proble Formulation: Two-Phase Tuning Modeling Workload Interference Experiments Conclusion & References A. Hardware Details B. Experimental Results C. Arithmetic Intensity D. Predictor Analysis 8 CONCLUSION In this work, we examine the unique characteristics of energy consumption in neural network inference, especially for edge devices. We identified unique tradeoffs and dimensions between energy consumption and inference latency SLOs and empirically demonstrated hidden components in optimizing energy consumption. We then propose an optimization framework that automatically and holistically tunes various hardware components to find a configuration aligned with the Pareto Frontier. We empirically verify the effectiveness and efficiency of PolyThrottle. PolyThrottle also adapts to the need for fine-tuning and proposes a simple performance prediction model to adaptively schedule finetuning requests while keeping the online inference workload under the inference latency SLO whenever possible. We hope our study sheds more light on the hidden dimension of NN energy optimization. REFERENCES Jetson partner solutions ebook. URL https://resources.nvidia.com/ en-us-jetson-success-stories/ jetson-partner-solutions-ebook?lx=XRDs_y. Twitter streaming traces, 2018. URL https://archive.org/details/ archiveteam-twitter-stream-201804. Alipourfard, O., Liu, H. H., Chen, J., Venkataraman, S., Yu, M., and Zhang, M. Cherrypick: Adaptively unearthing the best cloud configurations for big data analytics. In NSDI, volume 2, pp. 4–2, 2017. Anderson, T., Belay, A., Chowdhury, M., Cidon, A., and Zhang, I. Treehouse: A case for carbon-aware datacenter software. 2022. Anthony, L. F. W., Kanding, B., and Selvan, R. Carbontracker: Tracking and predicting the carbon footprint of training deep learning models. 2020. Arafa, Y., ElWazir, A., ElKanishy, A., Aly, Y., Elsayed, A., Badawy, A., Chennupati, G., Eidenbenz, S., and Santhi, N. Verified instruction-level energy consumption measurement for nvidia gpus. 2020. Bai, Z., Zhang, Z., Zhu, Y., and Jin, X. Pipeswitch: Fast pipelined context switching for deep learning applications. In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation, pp. 499–514, 2020. Banner, R., Hubara, I., Hoffer, E., and Soudry, D. Scalable methods for 8-bit training of neural networks. Advances in neural information processing systems, 31, 2018. Brochu, E., Cora, V. M., and De Freitas, N. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv:1012.2599, 2010. Cai, H., Wang, T., Wu, Z., Wang, K., Lin, J., and Han, S. On-device image classification with proxyless neural architecture search and quantization-aware fine-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0–0, 2019. Cai, H., Gan, C., Zhu, L., and Han, S. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, 33: 11285–11297, 2020. Cao, Q., Balasubramanian, A., and Balasubramanian, N. Towards accurate and reliable energy measurement of nlp models. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, 2020. Censor, Y. Pareto optimality in multiobjective problems. Applied Mathematics and Optimization, 4(1):41–59, 1977. Courbariaux, M., Bengio, Y., and David, J.-P. Training deep neural networks with low precision multiplications. arXiv preprint arXiv:1412.7024, 2014. Courbariaux, M., Bengio, Y., and David, J.-P. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems, 28, 2015. Crankshaw, D., Wang, X., Zhou, G., Franklin, M. J., Gonzalez, J. E., and Stoica, I. Clipper: A low latency online prediction serving system. In NSDI, volume 17, pp. 613– 627, 2017. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. Gardner, J. R., Kusner, M. J., Xu, Z. E., Weinberger, K. Q., and Cunningham, J. P. Bayesian optimization with inequality constraints. In ICML, volume 2014, pp. 937–945, 2014. Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., and Keutzer, K. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630, 2021. Gog, I., Kalra, S., Schafhalter, P., Gonzalez, J. E., and Stoica, I. D3: a dynamic deadline-driven approach for building autonomous vehicles. In Proceedings of the Seventeenth European Conference on Computer Systems, pp. 453–471, 2022. Gu, D., Xie, X., Huang, G., Jin, X., and Liu, X. Energyefficient gpu clusters scheduling for deep learning. arXiv preprint arXiv:2304.06381, 2023. Gujarati, A., Karimi, R., Alzayat, S., Hao, W., Kaufmann, A., Vigfusson, Y., and Mace, J. Serving {DNNs} like clockwork: Performance predictability from the bottom up. In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pp. 443–462, 2020. Gupta, U., Kim, Y. G., Lee, S., Tse, J., Lee, H.-H. S., Wei, G.-Y., Brooks, D., and Wu, C.-J. Chasing carbon: The elusive environmental footprint of computing. IEEE Micro, 42(4):37–47, 2022. He, C., Li, S., So, J., Zeng, X., Zhang, M., Wang, H., Wang, X., Vepakomma, P., Singh, A., Qiu, H., et al. Fedml: A research library and benchmark for federated machine learning. arXiv preprint arXiv:2007.13518, 2020. Hodak, M., Gorkovenko, M., and Dholakia, A. Towards power efficiency in deep learning on data center hardware. In IEEE International Conference on Big Data, 2019. Hong, S. and Kim, H. An integrated gpu power and performance model. In ISCA, 2010. Howard, A., Sandler, M., Chu, G., Chen, L.-C., Chen, B., Tan, M., Wang, W., Zhu, Y., Pang, R., Vasudevan, V., et al. Searching for mobilenetv3. pp. 1314–1324, 2019. Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. Ivanov, A., Dryden, N., Ben-Nun, T., Li, S., and Hoefler, T. Data movement is all you need: A case study on optimizing transformers. Proceedings of Machine Learning and Systems, 3:711–732, 2021. Kandasamy, K., Vysyaraju, K. R., Neiswanger, W., Paria, B., Collins, C. R., Schneider, J., Poczos, B., and Xing, E. P. Tuning hyperparameters without grad students: Scalable and robust bayesian optimisation with dragonfly. The Journal of Machine Learning Research, 21(1):3098–3124, 2020. Kandiah, V., Peverelle, S., Khairy, M., Pan, J., Manjunath, A., Rogers, T. G., Aamodt, T. M., and Hardavellas, N. Accelwattch: A power modeling framework for modern gpus. In MICRO, 2021. Kang, D.-K., Lee, K.-B., and Kim, Y.-C. Cost efficient gpu cluster management for training and inference of deep learning. Energies, 15(2):474, 2022. Kim, S., Gholami, A., Yao, Z., Mahoney, M. W., and Keutzer, K. I-bert: Integer-only bert quantization. pp. 5506–5518. PMLR, 2021. Klein, A., Bartels, S., Falkner, S., Hennig, P., and Hutter, F. Towards efficient bayesian optimization for big data. In NIPS 2015 Bayesian Optimization Workshop, 2015. Komoda, T., Hayashi, S., Nakada, T., Miwa, S., and Nakamura, H. Power capping of cpu-gpu heterogeneous systems through coordinating dvfs and task mapping. In 2013 IEEE 31st International Conference on computer design (ICCD). IEEE, 2013. Lacoste, A., Luccioni, A., Schmidt, V., and Dandres, T. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700, 2019. Lane, N. D. and Georgiev, P. Can deep learning revolutionize mobile sensing? In Proceedings of the 16th international workshop on mobile computing systems and applications, pp. 117–122, 2015. Lee, J., Chirkov, N., Ignasheva, E., Pisarchyk, Y., Shieh, M., Riccardi, F., Sarokin, R., Kulik, A., and Grundmann, M. On-device neural net inference with mobile gpus. arXiv preprint arXiv:1907.01989, 2019. Lowe-Power, J., Ahmad, A. M., Akram, A., Alian, M., Amslinger, R., Andreozzi, M., Armejach, A., Asmussen, N., Beckmann, B., Bharadwaj, S., et al. The gem5 simulator: Version 20.0+. arXiv preprint arXiv:2007.03152, 2020. Mei, X., Wang, Q., and Chu, X. A survey and measurement study of gpu dvfs on energy conservation. Digital Communications and Networks, 3(2):89–100, 2017. Nabavinejad, S. M., Reda, S., and Ebrahimi, M. Batchsizer: Power-performance tradeoff for dnn inference. In Proceedings of the 26th Asia and South Pacific Design Automation Conference, 2021. NVIDIA. Stream management, 2023a. URL https://docs.nvidia.com/cuda/ cuda-runtime-api. NVIDIA. Nvidia multi-instance gpu, 2023b. URL https://docs.nvidia.com/datacenter/ tesla/mig-user-guide/index.html. Peng, Y., Zhu, Y., Chen, Y., Bao, Y., Yi, B., Lan, C., Wu, C., and Guo, C. A generic communication scheduler for distributed dnn training acceleration. In SOSP, 2019. Qiao, A., Choe, S. K., Subramanya, S. J., Neiswanger, W., Ho, Q., Zhang, H., Ganger, G. R., and Xing, E. P. Pollux: Coadaptive cluster scheduling for goodput-optimized deep learning. In OSDI, 2021. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. Squad: 100,000+ questions for machine comprehension of text. In EMNLP, 2016. Romero, F., Li, Q., Yadwadkar, N. J., and Kozyrakis, C. {INFaaS}: Automated model-less inference serving. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp. 397–411, 2021. Sanh, V., Debut, L., Chaumond, J., and Wolf, T. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. Schwartz, R., Dodge, J., Smith, N. A., and Etzioni, O. Green ai. Commun. ACM, 63(12):54–63, 2020. Shen, H., Chen, L., Jin, Y., Zhao, L., Kong, B., Philipose, M., Krishnamurthy, A., and Sundaram, R. Nexus: A gpu cluster engine for accelerating dnn-based video analysis. In Proceedings of the 27th ACM Symposium on Operating Systems Principles, pp. 322–337, 2019. Snoek, J., Larochelle, H., and Adams, R. P. Practical bayesian optimization of machine learning algorithms. Advances in neural information processing systems, 25, 2012. Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for deep learning in nlp. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. Tambe, T., Hooper, C., Pentecost, L., Jia, T., Yang, E.- Y., Donato, M., Sanh, V., Whatmough, P., Rush, A. M., Brooks, D., et al. Edgebert: Sentence-level energy optimizations for latency-aware multi-task nlp inference. In MICRO, 2021. Tan, M. and Le, Q. Efficientnet: Rethinking model scaling for convolutional neural networks. pp.6105–6114. PMLR, 2019. Tang, Z., Wang, Y., Wang, Q., and Chu, X. The impact of gpu dvfs on the energy and performance of deep learning: An empirical study. In Proceedings of the Tenth ACM International Conference on Future Energy Systems, 2019. Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jegou, H. Training data-efficient image transform- ´ ers & distillation through attention. pp. 10347–10357. PMLR, 2021. Venkataraman, S., Yang, Z., Franklin, M., Recht, B., and Stoica, I. Ernest: Efficient performance prediction for {Large-Scale} advanced analytics. In 13th USENIX Symposium on Networked Systems Design and Implementation (NSDI 16), pp. 363–378, 2016. Wan, C., Santriaji, M., Rogers, E., Hoffmann, H., Maire, M., and Lu, S. Alert: Accurate learning for energy and timeliness. In ATC, 2020. Wang, F., Zhang, W., Lai, S., Hao, M., and Wang, Z. Dynamic gpu energy optimization for machine learning training workloads. IEEE Transactions on Parallel and Distributed Systems, 2021. Wang, G., Venkataraman, S., Phanishayee, A., Devanur, N., Thelin, J., and Stoica, I. Blink: Fast and generic collectives for distributed ml. In Proceedings of Machine Learning and Systems, 2020a. Wang, Y., Wang, Q., Shi, S., He, X., Tang, Z., Zhao, K., and Chu, X. Benchmarking the performance and energy efficiency of ai accelerators for ai training. In 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID), 2020b. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural language processing. In EMNLP, 2020. Wu, C.-J., Raghavendra, R., Gupta, U., Acun, B., Ardalani, N., Maeng, K., Chang, G., Aga, F., Huang, J., Bai, C., Gschwind, M., Gupta, A., Ott, M., Melnikov, A., Candido, S., Brooks, D., Chauhan, G., Lee, B.,Lee, H.-H., Akyildiz, B., Balandat, M., Spisak, J., Jain, R., Rabbat, M., and Hazelwood, K. Sustainable ai: Environmental implications, challenges and opportunities. In Proceedings of Machine Learning and Systems, 2022. Wu, X., Rao, J., Chen, W., Huang, H., Ding, C., and Huang, H. Switchflow: preemptive multitasking for deep learning. In Proceedings of the 22nd Inter Xu, M., Liu, J., Liu, Y., Lin, F. X., Liu, Y., and Liu, X. A first look at deep learning apps on smartphones. In The World Wide Web Conference, WWW ’19, pp. 2125–2136, 2019. You, J., Chung, J.-W., and Chowdhury, M. Zeus: Understanding and optimizing gpu energy consumption of dnn training. arXiv preprint arXiv:2208.06102, 2022. Yu, P. and Chowdhury, M. Fine-grained gpu sharing primitives for deep learning applications. Proceedings of Machine Learning and Systems, 2:98–111, 2020. Zhao, Y., Liu, X., Liu, S., Li, X., Zhu, Y., Huang, G., Liu, X., and Jin, X. Muxflow: Efficient and safe gpu sharing in large-scale production deep learning clusters. arXiv preprint arXiv:2303.13803, 2023.

ENTERGY

entergy.com

10K+ employees

Since 1913

COMPANY RANKING

ENTERGY

EVERGREEN INDEX #3698

Representing this company? Click here to claim and customize this page!

Entergy WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

ENTERGY

entergy.com

10K+ employees

Since 1913

COMPANY RANKING

ENTERGY

EVERGREEN INDEX #3698

RELATED COMPANIESRANK

STORIES ABOUT ENTERGY

Representing this company? Click here to claim and customize this page!

Entergy WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!