Madison Reed

RAGS MARTIN-JONES AND THE PR-NCE OF W-LES

State of Podcasts 2018: Takeaways from Podcast Movement on monetization, diversity, and discovery

By and Markus Neuy Dirk Krause How to transfer Shaders from Shadertoy to ThreeJS ( ). deutsche Version Introduction As a part of our research at the Publicis Pixelpark Innovationlab we studied how we could use lower level languages for web technologies. The obvious choice seemed to be or . asm.js WebAssembly But you could also use WebGL Shaders to solve machine-oriented problems. Shaders are programmed in a language that is similar to C/C++, and while they are not mainly meant for solving generic problems, they can be used for more things than just rendering images and 3D scenes.A second motivation stems from the asthetics that can be achieved with shaders. In 2002, a group of students from the University of Wisconsin-Madison published (“Non-PhotoRealistic Rendering Quake”), a variation of the well-known game Quake by injecting code into the rendering pipeline. NPRQuake NPRQuake Screenshot The aesthetic quality of this variation was stunning; we immediately understood that these kinds of effects could be a game-changer in projects. While in 2002 this variation was only possible by writing drivers for OpenGL, can now in 2018 be achieved by shaders — even in web browsers. So when we recently were involved in an art project, we decided to give shaders a go. Availability of shader code If you are not really used to program shaders the obvious choice is to search for freely available examples and use these (with a careful look at the licences involved). One library that stands out in this regard is with as another example. Shadertoy ShaderFrog Since we successfully worked with ThreeJS before we decided to publish our findings with postprocessing shaders from Shadertoy in ThreeJS. Shader in ThreeJS ThreeJS can be used to utilize postprocessing shaders which alter the whole rendered image and also material shaders, that can alter the material of 3D objects. Both types need a vertex and a fragment shader part; a vertex shader can change the position of vertices in 3D while a fragment shader usually replaces the color of a rendered image. This image shows the four possible variations. Types of shaders In the upper left a postprocessing shader adds a color gradient to the rendered image. To the right of it, a vertex shader reduces the render area. The two bottom images show material shaders; the left one only alters the color while the right one changes the position of the vertices. Since shaders are always composed of both vertex and fragment parts, the last example also changes the color. Shadertoy Trivial example from Shadertoy We studied how to transfer a shader from Shadertoy to ThreeJS back in 2014 with first results published on . We found the following pattern useful: StackOverflow Add like etc. ShaderToy specific variables iGlobalTime Rename to mainImage(out vec4 z, in vec2 w) main() Rename to z gl_FragColor By following this patters you can transfer . a simple shader to ThreeJS Trivial Shader Non-trivial example from Shadertoy With a more complex shader you need to do much more, as we will outline now. For a non-trivial example we chose by since you run into a couple of problems with it. You can find it here: Noise Contour candycat https://www.shadertoy.com/view/MscSzf This example also creates a whole scene in shader language. But in ThreeJS you usually want control over the 3D objects, so we decided to create the scene in ThreeJS while still utilizing the shaders to alter it. Understanding the structure of shaders We start by trying to get a grasp of the structure of the shader; this can be achieved with Shadertoy’s editor. Since edits to the code can be seen in realtime we can make small changes to understand how it works. Below the actual code we see that this code is based on a channel called , with the indicating a buffer. iChannel0 B To see this buffer in action we comment out line 37 and add this: The result should be: This simple change results in showing the color of the previous buffer and not the result of this buffer. By examining the previous buffer — — we see that this one also uses , so we are still not looking at the original scene creating code. Buf B iChannel0 Utilizing the same trick as before, we comment out line 29 and add a line that calculates uv and the actual color like so: This should leave us with: This looks very much more like a regular scene. Also, doesn’t use another buffer, so we are looking at the original scene creating code. Buf A Reconstruction in ThreeJS Full disclaimer here: what follows is by no chance ‘optimal’ code, but just one way to solve the problem in the most straight forward way. Creating the scene We start by creating a somewhat simpler scene with only a sphere and a plane. Additionally we want to use the from ThreeJS. MeshNormalMaterial A possible result is shown here: Shader in ThreeJS step 0 The code is contained in a HTML-file called : index.html We need to take care of the dependencies to the ThreeJS library and we also add our own code in : index.js This JavaScript code creates a renderer, a camera, an orbit control and also the plane and the sphere with a . It also takes care of window size changes and rendering. MeshNormalMaterial This concludes step zero of porting the scene from Shadertoy. Shader in ThreeJS step 0 Recreating the first shader pass In the next step we try to recreate the first shader render step in the buffer; this is basically copying the shader code to ThreeJS. This should be the result: Shadertoy withou the last pass To achieve this we used the for ThreeJS, which provides a simple way to use postprocessing shaders. EffectComposer This creates an EffectComposer Instance which adds a normal rendering pass and an additional shader pass. We copy the shader code in the variables and . The shader definition also defines a called used by the EffectComposer. It contains the image from the previous rendering pass that will be altered in the current pass. VERTEX FRAGMENT Uniform tDiffuse With this new render step in action, we show this pass rather than the original scene. Thus we need to add some code for resizing purposes, hence we add: Now we need to define the constants und . We can’t use Shadertoy’s vertex shader, so we need to define our own: VERTEX FRAGMENT We do use the fragment shader from Shadertoy, though, and add it to : FRAGMENT This basically creates the shader but we still need to address the following problems: the vertex shader coordinate isn’t used in the fragment shader yet- the fragment shader uses which is unknown in the current WebGL context texture must be renamed to mainImage main isn’t set yet. iResolution So the shader isn’t working yet. Addressing the first problem results in this definition: Now we can use the vector instead of . This results in: vUv fragCoord / iResolution.xy Now we simply replace every occurence of with . texture texture2D Additionally we alter the to without parameters: mainImage main should also return instead of which defines the color in the shader. main gl_FragColor fragColor Lastly we need to set by adding it to the uniforms. We do this by defining a ThreeJS vector storing width and height: iResolution Now we can add the resolution to the uniforms: We need to enhance our function: resize It is important that we use the uniforms of the actual render pass. The original one has been deeply cloned by the EffectComposer; changing the variable would have no effect. resolution Since we did define two uniforms, we need to introduce them to our fragment shaders so we define them: This concludes this shader pass and if all went well we see this: Shader Pass 1 without shadows From the blue lines we see that it generally works but the pink part is still missing. Let’s change that. Solve the problem with shadows The pink part is missing since the shader in Shadertoy secretly renders shadows to an alpha channel that wasn’t visible in the first place as we can see in the next picture: Shadows in Shadertoy There are several ways to solve this — we used the straight forward one by adding a material that holds the shadows. These must be handled in an additional render pass. So let’s create shadows in ThreeJS: Shadows need light, in this case a directional one: A can hold shadows. MeshPhongMaterial While a new render target saves them. And again, a function is needed: resize Now we can transfer the shadows to the new render target and prepare it for the shader: These lines set the material, render the scene, set the shadow to a uniform and change the material back to . MeshNormalMaterial Now the shader needs to know about the shadows to be able to process them, so we change the uniforms: Same for the fragment shader: Then we replace the former line with our shadow. The result should look like the the second step on Shadertoy. Shader step 1 Now we only miss the second shader pass to complete this. The final shader pass For the final shader pass we add another EffectComposer instance. Let’s define another shader: We deactivate for the previous render pass: renderToScreen Again, more variables are introduced; to change variables over time and to add noise. iTime iChannel1 Shadertoy Noise and iTime We use a ThreeJS clock for . iTime With every change we also update : iTime We add und noise to the uniforms: iTime The noise is simply a noisy texture (for example the one from Shadertoy) that we load with ThreeJS into . tNoise Now we need to adapt the fragment shader to our new variables, so we apply the following measures: change to mainImage main define uniforms and adapt the variables define the coordinates vUv change the returned result to gl_FragColor replace with texture texture2D This gives us: After these changes the shader still won’t compile, because this shader needs a specific WebGL extension. Thankfully, this is easy to add in ThreeJS: This gives us the following result: Final Shader Which is very close to the original Shadertoy: Conclusion We successfully transferred a complex Shadertoy shader to ThreeJS by following these steps: understand the structure of the specific shader implement shader passes address possible GLSL incompatibilities create optional shader passes and/or materials activate optional extensions We expect that these challenges will be mitigated with the upcoming WebGL2 support in ThreeJS since possible GLSL incompatibilities should vanish. The full source code is . here The final result Helpful links and resources Shadertoy Shader: https://www.shadertoy.com/view/MscSzf WebGL2Fundamentals: https://webgl2fundamentals.org ThreeJS: https://threejs.org/](https://threejs.org/ ThreeJS Examples: https://github.com/mrdoob/three.js/tree/dev/examples WebGL-Extensions: and https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/getExtension https://github.com/mrdoob/three.js/issues/7379 EffectComposer: https://github.com/hughsk/three-effectcomposer Another EffectComposer approach: https://github.com/spite/Wagner WebGL Quick Reference Card: https://www.khronos.org/files/webgl/webgl-reference-card-1_0.pdf Credits Part of this research was funded by . EFRE.NRW Projekt ‘ForK — Forschungsimpulse für die Kreativwirtschaft’

Converting Shaders from Shadertoy to ThreeJS

1️⃣ Code: The Hidden Language of Computer Hardware and Software This is a classic book I believe should be the first thing to read when you want to learn about how computer works. Even though the first edition was published nearly two decades ago, the content is still relevant, and good news is that the second edition was just released Aug 2022 with new chapters on clock, ALU etc, and the companion website, , uses animated graphics of key circuits in the book to make computers even easier to comprehend. CodeHiddenLanguage.com It covers how technology involved and computer was built, how do hardware components work together to run a program, the book answered my questions about what happens in a computer behind the scenes, and it laid foundation for me to grasp higher abstraction level mechanism such as operating system 2️⃣ Computer Systems: A Programmer's Perspective This is actually a text book used at Carnegie Mellon, it explains the underlying elements common among all computer systems and how they affect general application performance but from the programmer’s perspective. It’s a big book and indeed pricy, I was following the corresponding open sources course online, after a while, I just thought I better go with the hard copy so I can flip through them and work on the exercises. But if you don’t want to spend the money on hard copy, learn with the open source course videos should be equivalent from a self learning perspective. The book has three main parts: program structure and execution, running programs on a system, interaction and communication between programs. I did not read this book from start to end, but rather dive into the chapters I want to learn more details or curious about, since I’ve already have some basic ideas after reading the first book. Depends on your learning style and your knowledge system, you can go through the chapters sequentially or read the first book and use this as some sort of advanced learning material. The benefit of the content in this book/course at least for me is that it provided extensive coverage on how a computer works and the practice problems to test if you really understand a concept. 3️⃣ Operating Systems: Three Easy Pieces This book is the culmination of years of teaching intro to operating systems by by  Remzi H and Andrea C Arpaci-Dusseau  to both undergraduates and graduates at the University of Wisconsin-Madison Computer Sciences department. It has three pieces: virtualization, concurrency, persistence, these are the fundamental concepts in OS construction. I find it works great with the operating system course the authors are teaching. Note that on the course page it lists CSAPP similar course as its prerequisite. After learning the fundamentals I’ve found it helps when I look at OS level metrics to identify system bottlenecks, and it was easier to understand the new technologies such as Docker, I also think learnings at this abstraction level enables me to have a clearer picture when I debug or design a system, or even during system design interviews. Which one are you mostly interested in? Hope it helps for starting to get a better understanding of computer systems! 🤓

📚3 Must Read Books for Programmers from Non-traditional Background

A Sparse Bayesian Learning for Diagnosis of Nonstationary and Spatially Correlated Faults:References

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Minghao Yan, University of Wisconsin-Madison; (2) Hongyi Wang, Carnegie Mellon University; (3) Shivaram Venkataraman, myan@cs.wisc.edu. Table of Links Abstract & Introduction Motivation Opportunities Architecture Overview Proble Formulation: Two-Phase Tuning Modeling Workload Interference Experiments Conclusion & References A. Hardware Details B. Experimental Results C. Arithmetic Intensity D. Predictor Analysis B EXPERIMENTAL RESULTS In this section, we further demonstrate the tradeoff between memory frequency and maximum GPU frequency by presenting an array of results. These results underline the interesting observation that the energy consumption patterns vary for the same model operating on different devices. Furthermore, even for the same model device pairing, the optimization landscape can be significantly influenced by the batch size. This underlines the complexities of energy optimization and the need for an adaptive framework that can take these factors into account. Figures 6 − 12 show the energy consumption patterns of EfficientNet and Bert on Jetson TX2 and Orin under various batch sizes. Table 7 shows the optimal CPU frequency and corresponding energy consumption reduction in image preprocessing.

PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices: Experimental Results

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Minghao Yan, University of Wisconsin-Madison; (2) Hongyi Wang, Carnegie Mellon University; (3) Shivaram Venkataraman, myan@cs.wisc.edu. Table of Links Abstract & Introduction Motivation Opportunities Architecture Overview Proble Formulation: Two-Phase Tuning Modeling Workload Interference Experiments Conclusion & References A. Hardware Details B. Experimental Results C. Arithmetic Intensity D. Predictor Analysis 3 OPPORTUNITIES In this section, we perform empirical experiments to uncover new opportunities for optimizing energy use in NN inference. As discussed in Section 2, prior work did not study how memory frequency, minimum GPU frequency, and CPU frequency play a role in energy consumption. This is partially limited by hardware constraints. Specialized power rails need to be built into the device during manufacturing to enable accurate measurement of energy consumption associated with each component. We leverage two Jetson developer kits, TX2 and Orin, which offer native support for component-wise energy consumption measurement and frequency tuning, to study how these frequencies impact inference latency and energy consumption in modern deep learning workloads. We find that the default frequencies are much larger than optimal and throttling all these frequency knobs offer energy consumption reduction with minimal impact on inference latency. Figure 3 illustrates the energy optimization landscape when varying GPU and memory frequencies, without imposing any constraints on latency SLO. The plot reveals that, without any other constraints, the energy optimization landscape generally exhibits a bowl shape. However, this shape varies depending on the models, devices, and other hyperparameters, such as batch sizes (See Appendix B for more results). Next, we dive into how each hardware component affects inference energy consumption. : Experiments in this section are performed with 16-bit floating point number precision, as it has been demonstrated to have minimal impact on model accuracy in practice. We use Bert and EfficientNet models and vary the EfficientNet model size between B0, B4, B7 (Table 4). Setup For each model, we fix the GPU frequency at the optimal frequency determined by grid-search of all possible frequency configurations. We then examine the tradeoff between inference latency and energy consumption as we progressively throttle memory frequency. The range of available memory frequencies can be found in Table 1. Memory frequency experiment: : Table 2 reveals that memory frequency plays a vital role in reducing energy consumption. The savings provided by memory frequency tuning are similar and consistent across models on both hardware platforms, ranging from approximately 12% to 25%. This indicates that the Results default memory frequency is higher than optimal for modern Deep Learning workloads. For heavy workloads such as Bert, memory tuning can account for the majority of the energy consumption reduction. This can be partially attributed to the memory-bound nature of Transformer-based models (Ivanov et al., 2021). Our result demonstrates that systems that aim to optimize energy use in neural network inference need to take memory frequency into account. CPUs are only used for data pre-processing. Thus, we first measure the time spent in the data processing part of the inference pipeline. Next, we measure the energy saved by throttling the CPU frequency and assess the inference latency slowdown caused by reducing CPU frequency. The data preprocessing we perform is standard in almost all image processing and object detection pipelines, where we read the raw image file, convert it to CPU Frequency Experiment: an RGB scale, resize it, and reorient it to the desired input resolution and data layout. : The preprocessing time across different EfficientNet models remains constant since the operations performed are identical. As a result, the relative impact of CPU tuning on overall energy consumption depends on the ratio between preprocessing time and inference time. As the model size increases and inference duration increases, the influence of CPU tuning on overall energy consumption decreases. We observe that on both Jetson TX2 and Orin platforms, CPU tuning can decrease preprocessing energy consumption by approximately 30%. Depending on the model, quantization level, and batch size, this results in up to a 6% reduction in overall energy consumption. Results We maintain the default hardware configuration and only adjust the minimum GPU frequency on Jetson Orin. Increasing the minimum GPU frequency forces the GPU DVFS mechanism to operate within a smaller range. We scale the model from Minimum GPU frequency experiment: EfficientNet B0 to EfficientNet B7 to illustrate the effect of the GPU minimum frequency on inference latency. : Table 3 indicates that tuning the minimum GPU frequency can significantly reduce energy consumption when the workload cannot fully utilize the computational power of the hardware. Notably, energy consumption and inference latency are reduced by forcing the GPU to operate Results at a higher frequency. This differs from the tradeoff observed in other experiments, where we exchange inference latency for lower energy consumption. Tuning minimum GPU frequency can nearly halve the energy consumption for small models. As computational power becomes saturated with increasing model size, the return on tuning the minimum GPU frequency diminishes. Figure 4 shows the per query energy cost as we vary the minimum and maximum GPU frequency. It shows that increasing the minimum GPU frequency from the default minimum leads to lower energy costs and inference latency.

PolyThrottle: Energy-efficient Neural Network Inference on Edge Devices: Opportunities

This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Minghao Yan, University of Wisconsin-Madison; (2) Hongyi Wang, Carnegie Mellon University; (3) Shivaram Venkataraman, myan@cs.wisc.edu. Table of Links Abstract & Introduction Motivation Opportunities Architecture Overview Proble Formulation: Two-Phase Tuning Modeling Workload Interference Experiments Conclusion & References A. Hardware Details B. Experimental Results C. Arithmetic Intensity D. Predictor Analysis 2 MOTIVATION Many deep neural networks have been deployed on edge devices to perform tasks such as image classification, object detection, and dialogue systems. Scenarios including smart home assistants (He et al., 2020), inventory and supply chain monitoring (jet), and autopilot (Gog et al., 2022) often use battery-based devices that contain GPUs to perform the aforementioned tasks. In these scenarios, pre-trained models are installed on the devices where the inference workload is deployed. Prior works have focused on optimizing the energy consumption of GPUs (Wang et al., 2020b; 2021; Tang et al., 2019; Strubell et al., 2019; Mei et al., 2017) in cloud scenarios (Qiao et al., 2021; Wan et al., 2020; Hodak et al., 2019) and training settings (Wang et al., 2020a; Peng et al., 2019; Kang et al., 2022). On-device inference workloads exhibit different characteristics and warrant separate attention. In this section, we outline previous efforts in optimizing ondevice neural network inference and discuss our approach to holistically optimize energy consumption. 2.1 On-device Neural Network Deployment Prior work in optimizing on-device neural network inference focuses on quantization (Kim et al., 2021; Banner et al., 2018; Courbariaux et al., 2015; 2014; Gholami et al., 2021), designing hardware-friendly network architectures (Xu et al., 2019; Lee et al., 2019; Sanh et al., 2019; Touvron et al., 2021; Howard et al., 2019), and leveraging hardware components specific to mobile settings, such as DSPs (Lane & Georgiev, 2015). Our work explores an orthogonal dimension and aims to answer a different question: Given a neural network to deploy on a specific device, how can we tune the device to reduce energy consumption? In our work, we focus on edge devices that contain CPUs, memory, and GPUs. These devices are generally more powerful than DSPs often found on mobile devices. One such example is the Nvidia Jetson series, which is capable of handling a wide array of applications, ranging from AI to robotics and embedded IoT solutions (jet). The devices also come with dynamic voltage and frequency scaling (DVFS) capabilities that allow for the optimization of power consumption and thermal management during complex computational tasks. The Jetson series features a unified memory shared by both CPU and GPUs. We refer to the operating frequency of CPU, GPU, and shared memory as CPU frequency, GPU frequency, and memory frequency in this paper. To understand the system requirements in edge NN inference, we next describe a case study of how NNs are deployed in an inventory management company. From our conversations, Company A works with Customer B to deploy neural networks on edge devices to optimize inventory management. To comply with regulations and protect privacy, data from each inventory site are required to be stored locally. The vast difference in the layout of the inventories makes it impossible to pre-train the model on data from every warehouse. Therefore, these devices come with a pre-trained model based on data from a small sample of inventories, which may have significantly different layouts and external environments compared to the actual deployment venue. Consequently, daily fine-tuning is required to enhance performance in the deployed sites, as the environment continually evolves. Similar arguments apply to smart home devices, where a model is pre-trained on selected properties, but the deployed households may be much more diverse. To address privacy concerns, on-device Case Study on Inventory Management: fine-tuning of neural networks is preferred, as it keeps sensitive data locally. Therefore, edge devices often need to run both inference and periodic fine-tuning. Combining multiple workloads on edge devices can lead to SLO violations due to interference and increased energy use. 2.2 Holistic Energy Consumption Optimization Some recent works have explored reducing energy consumption by optimizing for batch size and GPU maximum frequency (You et al., 2022; Nabavinejad et al., 2021; Komoda et al., 2013; Gu et al., 2023) and developing power models for modern GPUs (Kandiah et al., 2021; Hong & Kim, 2010; Arafa et al., 2020; Lowe-Power et al., 2020). In this work, we argue that other hardware components also cause energy inefficiency and require separate optimization. We perform a grid search over GPU, memory, and CPU frequencies and various batch sizes to examine the Pareto frontier of inference latency and energy consumption. Figure 2 shows the tradeoff between the per-query energy consumption and inference latency (normalized to the optimal latency) on Jetson TX2 and Jetson Orin. Each point in the figure represents the optimal configuration that we find through grid search under a given inference latency budget and batch size. As Figure 2 shows, the Pareto frontier is not smooth globally and is difficult to capture by a simple model, which warrants more sophisticated optimization techniques to quickly converge to a hardware configuration that lies on the Pareto Frontier (Censor, 1977). Zeus (You et al., 2022) attempts to reduce the energy consumption of neural network training by changing the GPU power limit and tuning training batch size. PolyThrottle also includes these two factors. In Zeus (You et al., 2022), the focus is on training workloads in data center settings, where batch size tuning helps achieve an accuracy threshold in an energy-efficient way. We include batch size as part of PolyThrottle as it provides a trade-off between inference latency and throughput. Our empirical evaluation reveals new avenues available for optimization which complicates the search space, as we describe next.

MADISON REED

madison-reed.com

-- employees

Since n.d.

COMPANY RANKING

MADISON REED

EVERGREEN INDEX #7011

Representing this company? Click here to claim and customize this page!

Madison Reed WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

MADISON REED

madison-reed.com

-- employees

Since n.d.

COMPANY RANKING

MADISON REED

EVERGREEN INDEX #7011

RELATED COMPANIESRANK

STORIES ABOUT MADISON REED

Representing this company? Click here to claim and customize this page!

Madison Reed WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!