Optimizing LLM Inference: Sparse Activation, MoE, and Gated-MLP Efficiency

Table of Links

Abstract and 1. Introduction

2. Related Work and Background

3. Analysis

   3.1 Limitations about Existing ReLUficatio

   3.2 dReLU

4. Are Neurons in Expert still Sparsely Activated?

5. dReLU Sparsification

6. Experiments Results

   6.1 Downstream Tasks Performance

   6.2 Sparsity of Sparsified Models

7. Practical Inference Speedup Evaluation

   7.1 Experiments Setting

   7.2 Pure CPU Inference and 7.3 Hybrid GPU-CPU Inference

   7.4 Deploy LLMs on mobile phones

8. Conclusion and References

A. Appendix / supplemental material

B. Limitation

C. Broader Impact

2 Related Work and Background

Efficient Inference of LLMs. Efficient LLM inference poses challenges that necessitate a synergistic combination of algorithmic and systemic approaches. From an algorithmic standpoint, researchers have explored various methods to reduce computation and memory overheads, including compressing models [40, 63, 18, 34, 61], modifying model structures [3, 21], and speculative decoding methods [32, 12, 10]. On the systemic front, there are efforts that effectively integrate the features of downstream hardware and upper-level models to maximize the efficiency of computation and memory utilization [4, 49, 16, 64], leading to the development of more efficient frameworks like vLLM [29].

Sparse activation, in particular, has emerged as a research area that demands an even tighter integration of algorithmic and systemic approaches. The selection of activation functions and the construction of activation predictors are algorithmic problems, while fully exploiting the sparse activation of LLMs on specific hardware is a systemic challenge. By leveraging sparse activation, researchers have achieved promising results in building efficient LLM inference systems [36, 56].

Mixture-of-Experts (MoE). MoE techniques induce effective sparsity in LLMs by determining which subset of subnetworks (referred to as "experts") to activate during the inference pass, often through a trained "router" subnetwork. This approach allows the model to enhance its capacity without escalating the computational expenses [31, 53].

Intrinsic Activation Sparsity. Intrinsic activation sparsity is known to be present in LLMs that utilize ReLU family nonlinearities in their MLP blocks [68, 33]. This phenomenon has been explored to accelerate inference speed and reduce memory usage [56, 36, 37]. With this phenomenon, each neuron can be viewed as an expert to reduce the computation overhead.

Gated-MLP Blocks. We now delve into the components of LLMs that our study aims to analyze: the Gated-MLP blocks, which are commonly used. A Gated-MLP block consists of three fully

connected layers and performs the following computation: