Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Related Work and Background


Analysis
3.1 Limitations about Existing ReLUficatio
3.2 dReLU


Are Neurons in Expert still Sparsely Activated?


dReLU Sparsification


Experiments Results
6.1 Downstream Tasks Performance
6.2 Sparsity of Sparsified Models


Practical Inference Speedup Evaluation
7.1 Experiments Setting
7.2 Pure CPU Inference and 7.3 Hybrid GPU-CPU Inference
7.4 Deploy LLMs on mobile phones


Conclusion and References Related Work and Background Related Work and Background Related Work and Background Analysis
3.1 Limitations about Existing ReLUficatio
3.2 dReLU Analysis 3.1 Limitations about Existing ReLUficatio 3.1 Limitations about Existing ReLUficatio 3.2 dReLU 3.2 dReLU Are Neurons in Expert still Sparsely Activated? Are Neurons in Expert still Sparsely Activated? Are Neurons in Expert still Sparsely Activated? dReLU Sparsification dReLU Sparsification dReLU Sparsification Experiments Results
6.1 Downstream Tasks Performance
6.2 Sparsity of Sparsified Models Experiments Results 6.1 Downstream Tasks Performance 6.1 Downstream Tasks Performance 6.2 Sparsity of Sparsified Models 6.2 Sparsity of Sparsified Models Practical Inference Speedup Evaluation
7.1 Experiments Setting
7.2 Pure CPU Inference and 7.3 Hybrid GPU-CPU Inference
7.4 Deploy LLMs on mobile phones Practical Inference Speedup Evaluation Practical Inference Speedup Evaluation 7.1 Experiments Setting 7.1 Experiments Setting 7.2 Pure CPU Inference and 7.3 Hybrid GPU-CPU Inference 7.2 Pure CPU Inference and 7.3 Hybrid GPU-CPU Inference 7.4 Deploy LLMs on mobile phones 7.4 Deploy LLMs on mobile phones Conclusion and References Conclusion and References Conclusion and References A. Appendix / supplemental material A. Appendix / supplemental material B. Limitation B. Limitation C. Broader Impact C. Broader Impact 2 Related Work and Background Efficient Inference of LLMs. Efficient LLM inference poses challenges that necessitate a synergistic combination of algorithmic and systemic approaches. From an algorithmic standpoint, researchers have explored various methods to reduce computation and memory overheads, including compressing models [40, 63, 18, 34, 61], modifying model structures [3, 21], and speculative decoding methods [32, 12, 10]. On the systemic front, there are efforts that effectively integrate the features of downstream hardware and upper-level models to maximize the efficiency of computation and memory utilization [4, 49, 16, 64], leading to the development of more efficient frameworks like vLLM [29]. Efficient Inference of LLMs Sparse activation, in particular, has emerged as a research area that demands an even tighter integration of algorithmic and systemic approaches. The selection of activation functions and the construction of activation predictors are algorithmic problems, while fully exploiting the sparse activation of LLMs on specific hardware is a systemic challenge. By leveraging sparse activation, researchers have achieved promising results in building efficient LLM inference systems [36, 56]. Mixture-of-Experts (MoE). MoE techniques induce effective sparsity in LLMs by determining which subset of subnetworks (referred to as "experts") to activate during the inference pass, often through a trained "router" subnetwork. This approach allows the model to enhance its capacity without escalating the computational expenses [31, 53]. Mixture-of-Experts (MoE) Intrinsic Activation Sparsity. Intrinsic activation sparsity is known to be present in LLMs that utilize ReLU family nonlinearities in their MLP blocks [68, 33]. This phenomenon has been explored to accelerate inference speed and reduce memory usage [56, 36, 37]. With this phenomenon, each neuron can be viewed as an expert to reduce the computation overhead. Intrinsic Activation Sparsity Gated-MLP Blocks. We now delve into the components of LLMs that our study aims to analyze: the Gated-MLP blocks, which are commonly used. A Gated-MLP block consists of three fully Gated-MLP Blocks. connected layers and performs the following computation: Authors:
(1) Yixin Song, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University;
(2) Haotong Xie, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University;
(3) Zhengyan Zhang, Department of Computer Science and Technology, Tsinghua University;
(4) Bo Wen, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University;
(5) Li Ma, Shanghai Artificial Intelligence Laboratory;
(6) Zeyu Mi, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University Mi yzmizeyu@sjtu.edu.cn);
(7) Haibo Chen, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University. Authors: Authors: (1) Yixin Song, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University; (2) Haotong Xie, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University; (3) Zhengyan Zhang, Department of Computer Science and Technology, Tsinghua University; (4) Bo Wen, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University; (5) Li Ma, Shanghai Artificial Intelligence Laboratory; (6) Zeyu Mi, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University Mi yzmizeyu@sjtu.edu.cn); (7) Haibo Chen, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University. This paper is available on arxiv under CC BY 4.0 license. This paper is available on arxiv under CC BY 4.0 license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Optimizing LLM Inference: Sparse Activation, MoE, and Gated-MLP Efficiency

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

256K Tokens on One GPU? Jamba’s Engineering Magic Explained

100 Days of AI Day 2: Enhancing Prompt Engineering for ChatGPT

The Noonification: Exploring Tool-Integrated Reasoning: Innovating Math-Proficient LLMs (10/5/2023)

The Noonification: Simple Database Migration Scripts On Your CI/CD step (10/16/2023)

The Noonification: Breaking Axioms in Program Execution (11/5/2023)

The Noonification: The Easiest Way to Create Your First NPM Package (12/15/2023)

256K Tokens on One GPU? Jamba’s Engineering Magic Explained

100 Days of AI Day 2: Enhancing Prompt Engineering for ChatGPT

The Noonification: Exploring Tool-Integrated Reasoning: Innovating Math-Proficient LLMs (10/5/2023)

The Noonification: Simple Database Migration Scripts On Your CI/CD step (10/16/2023)

The Noonification: Breaking Axioms in Program Execution (11/5/2023)

The Noonification: The Easiest Way to Create Your First NPM Package (12/15/2023)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps