Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Related Work and Background


Analysis
3.1 Limitations about Existing ReLUficatio
3.2 dReLU


Are Neurons in Expert still Sparsely Activated?


dReLU Sparsification


Experiments Results
6.1 Downstream Tasks Performance
6.2 Sparsity of Sparsified Models


Practical Inference Speedup Evaluation
7.1 Experiments Setting
7.2 Pure CPU Inference and 7.3 Hybrid GPU-CPU Inference
7.4 Deploy LLMs on mobile phones


Conclusion and References Related Work and Background Related Work and Background Related Work and Background Analysis
3.1 Limitations about Existing ReLUficatio
3.2 dReLU Analysis 3.1 Limitations about Existing ReLUficatio 3.1 Limitations about Existing ReLUficatio 3.2 dReLU 3.2 dReLU Are Neurons in Expert still Sparsely Activated? Are Neurons in Expert still Sparsely Activated? Are Neurons in Expert still Sparsely Activated? dReLU Sparsification dReLU Sparsification dReLU Sparsification Experiments Results
6.1 Downstream Tasks Performance
6.2 Sparsity of Sparsified Models Experiments Results 6.1 Downstream Tasks Performance 6.1 Downstream Tasks Performance 6.2 Sparsity of Sparsified Models 6.2 Sparsity of Sparsified Models Practical Inference Speedup Evaluation
7.1 Experiments Setting
7.2 Pure CPU Inference and 7.3 Hybrid GPU-CPU Inference
7.4 Deploy LLMs on mobile phones Practical Inference Speedup Evaluation Practical Inference Speedup Evaluation 7.1 Experiments Setting 7.1 Experiments Setting 7.2 Pure CPU Inference and 7.3 Hybrid GPU-CPU Inference 7.2 Pure CPU Inference and 7.3 Hybrid GPU-CPU Inference 7.4 Deploy LLMs on mobile phones 7.4 Deploy LLMs on mobile phones Conclusion and References Conclusion and References Conclusion and References A. Appendix / supplemental material A. Appendix / supplemental material B. Limitation B. Limitation C. Broader Impact C. Broader Impact A Appendix / supplemental material A.1 Training Details of 300M models In this subsection, we will introduce the details of training the 300M model, including the model architecture, types of data used, and hyperparameters. The evaluation results of the final 300M models are shown in Table 10. A.1.1 Architecture A.1.1 Architecture We adopt a similar model architecture to Llama 2 [60] with the following details: Activation Function and Intermediate Hidden Size. We focus on dReLU and SwiGLU [52] activation functions. Activation Function and Intermediate Hidden Size. Multi-Head Attention. For Attention block, we adopt the Llama-2-7B’s architecture, apply prenormalization using RMSNorm [66] and RoPE [57] for Positional embedding. Multi-Head Attention A.1.2 Training Hyperparameters A.1.2 Training Hyperparameters We utilize LLaMA-Factory as our training framework [70]. Our models are trained using AdamW optimizer [38], with the following hyper-parameters: β1 = 0.9, β2 = 0.95. We adopt a cosine learning rate schedule and we set weight decay to 0.01 and gradient clipping hyper-parameters. (see Table 12 for more details). A.2 Activation Distribution Analysis of MoE Models Figure 7 shows the activation distribution of Mistral and Mixtral. We can see that the FFN in MoE models show the similar activation distribution compared to dense Mistral models. A.3 Details Performance of ReLUfied Models In this subsection, we present the detailed performance metrics of our ReLUfied models across various commonsense benchmarks, as shown in Table 13. Authors:
(1) Yixin Song, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University;
(2) Haotong Xie, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University;
(3) Zhengyan Zhang, Department of Computer Science and Technology, Tsinghua University;
(4) Bo Wen, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University;
(5) Li Ma, Shanghai Artificial Intelligence Laboratory;
(6) Zeyu Mi, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University Mi yzmizeyu@sjtu.edu.cn);
(7) Haibo Chen, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University. Authors: Authors: (1) Yixin Song, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University; (2) Haotong Xie, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University; (3) Zhengyan Zhang, Department of Computer Science and Technology, Tsinghua University; (4) Bo Wen, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University; (5) Li Ma, Shanghai Artificial Intelligence Laboratory; (6) Zeyu Mi, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University Mi yzmizeyu@sjtu.edu.cn); (7) Haibo Chen, Institute of Parallel and Distributed Systems (IPADS), Shanghai Jiao Tong University. This paper is available on arxiv under CC BY 4.0 license. This paper is available on arxiv under CC BY 4.0 license. available on arxiv available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

TurboSparse Limitations: The Impact of 150B Token Recovery Training

TurboSparse: Faster LLMs via dReLU Activation

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

256K Tokens on One GPU? Jamba’s Engineering Magic Explained

TurboSparse: Elite Inference Speed via dReLU Sparsity

100 Days of AI Day 7: Building Your Own ChatGPT with Langchain

The Noonification: Blockchain ❤️s WASM: Chapter Arbitrum (11/6/2023)

The Noonification: Craig Wright is Satoshi Nakamoto (12/24/2023)

The Noonification: Patience is Beautiful (6/18/2023)

256K Tokens on One GPU? Jamba’s Engineering Magic Explained

TurboSparse: Elite Inference Speed via dReLU Sparsity

100 Days of AI Day 7: Building Your Own ChatGPT with Langchain

The Noonification: Blockchain ❤️s WASM: Chapter Arbitrum (11/6/2023)

The Noonification: Craig Wright is Satoshi Nakamoto (12/24/2023)

The Noonification: Patience is Beautiful (6/18/2023)

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps