Analyzing ReLUfication Limitations: Enhancing LLM Sparsity via Up Projection

Table of Links

Abstract and 1. Introduction

2. Related Work and Background

3. Analysis

   3.1 Limitations about Existing ReLUficatio

   3.2 dReLU

4. Are Neurons in Expert still Sparsely Activated?

5. dReLU Sparsification

6. Experiments Results

   6.1 Downstream Tasks Performance

   6.2 Sparsity of Sparsified Models

7. Practical Inference Speedup Evaluation

   7.1 Experiments Setting

   7.2 Pure CPU Inference and 7.3 Hybrid GPU-CPU Inference

   7.4 Deploy LLMs on mobile phones

8. Conclusion and References

A. Appendix / supplemental material

B. Limitation

C. Broader Impact

3 Analysis

3.1 Limitations about Existing ReLUfication

We first evaluate the sparsity of ReLULlama-7B [59] and the original Llama-2-7B [60], as shown in Table 1. The results reveal that existing ReLUfication methods can only improve the sparsity from 40% to 67%, indicating their limited effectiveness in significantly enhancing model sparsity.

To investigate the underlying reasons for this limitation, we profile the activation distribution of the gate and up projection components separately in ReLULlama-7B and Llama-2-7B, as illustrated in Figure 3. The figure shows that after ReLUfication, the combined activation becomes more concentrated around 0, with the sparsity increasing to 67%. This can be attributed to the ReLU activation function applied after the gate weight, which masks all negative activations to zero.

To further push the sparsity, shifted-ReLU [42] has been proposed, which adjusts the threshold of ReLU function to mask out more activations in the gate projection. However, the improvements brought by this method are limited. Another line of work is to adopt progressive sparsity regularization to the intermediate output to introduce more zero activation output [55]. However, this method carries the risk of performance degradation.

Existing ReLUfication methods primarily focus on modifying the gate component. Different from previous work, we find that existing ReLUfication doesn’t alter the activation distribution of the up projection component, as shown in Figure 3(c) and (f). According to the definition of Gated-MLP (Equation 1), the gate and up projection components jointly influence the sparsity of neuron activations in parallel. However, a significant number of activation values in the up projection component remain less than 0. This suggests that masking the outputs of the up and gate matrices that are less than 0 as inactive could introduce stronger sparsity without sacrificing non-linear capabilities. This observation motivates us to explore the possibility of further enhancing model sparsity by modifying the up projection.