Authors:
(1) Wanyun Cui, Shanghai University of Finance and Economics, with equal contribution;
(2) Qianle Wang, Shanghai University of Finance and Economics, with equal contribution. Table of Links Abstract and 1 Introduction 2 Related Work 3 Quantifying the Impact of Parameters on Model Performance & 4. Unified Mixed-Precision Training 5 Prevalence of Parameter Heterogeneity in LLMs 6 Quantization Experiments and 6.1 Implementation Details 6.2 Effect of Base LLM Quantization 6.3 Effect of Chat LLM Quantization 6.4 Comparison of Parameter Selection Criteria, Conclusion, & References 6.4 Comparison of Parameter Selection Criteria To evaluate the effectiveness of our proposed impact-based parameter selection criterion, we conduct experiments comparing it with the commonly used magnitude-based criterion [18]. Table 5 presents the perplexity of LLaMA2-7B-3bit and LLaMA2-13B-3bit models, using both criteria for cherry parameter selection. From the results, it is evident that the impact-based criterion consistently outperforms the magnitude-based criterion across all settings. These results demonstrate that our proposed impact-based criterion is a more effective measure of parameter importance compared to the magnitude-based criterion. The impacts identify and preserve the most critical parameters during the quantization process. We think this justify the heterogeneity of parameter impacts against parameter magnitudes as we highlighted in § 5. The extensive experimental results presented in this section clearly demonstrate the superiority of CherryQ compared to existing quantization methods. By effectively identifying the critical cherry parameters and unifying the mixed-precision parameter optimization, CherryQ achieves state-of-theart performance for both base LLMs and chat LLMs. 7. Conclusion In this paper, we investigated the parameter heterogeneity phenomenon in LLMs. Our experiments on LLaMA2, Mistral, Gemma, and Vicuna models, consistently demonstrated that a small subset of parameters plays a crucial role in maintaining the model’s performance, while the majority of parameters can be quantized to ultra-low precision without significant degradation. This finding highlights the potential for efficient model compression and quantization techniques that take into account the heterogeneous nature of parameter importance. Motivated by this observation, we proposed a novel impact-based parameter selection criterion for quantization. Our method effectively identifies and preserves the most critical cherry parameters during the quantization process. We use a QAT framework for unified optimization of both cherry parameters and normal parameters. Extensive experiments demonstrate that CherryQ outperforms the commonly used magnitude-based criterion, achieving significantly lower perplexity scores and better downstream performance. The heterogeneity and proposed approach pave the way for more efficient deployment of LLMs in resource-constrained environments. A Effect of Chat LLM Quantization on MMLU We further evaluate the performance of CherryQ on the MMLU benchmark by quantizing the Vicuna1.5 model. As shown in Table 6, CherryQ outperforms both QAT and GPTQ in terms of average accuracy across almost all categories. References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. [3] Yoshua Bengio, Nicholas L´eonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. [4] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7947–7969, 2021. [5] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://lmsys. org/blog/2023-03- 30-vicuna, 3(5), 2023. [6] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318–30332, 2022. [7] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024. [8] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. [9] Han Guo, Philip Greengard, Eric P Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.12023, 2023. [10] Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pages 293–299. IEEE, 1993. [11] John A Hertz. Introduction to the theory of neural computation. 2018. [12] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [13] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023. [14] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342, 2018. [15] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989. [16] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations, 2020. [17] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activationaware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. [18] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888, 2023. [19] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. [20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. [21] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, 2023. [22] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi`ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. [23] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [24] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402–17414, 2022. [25] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR, 2023. [26] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Wanyun Cui, Shanghai University of Finance and Economics, with equal contribution; (2) Qianle Wang, Shanghai University of Finance and Economics, with equal contribution. Authors: Authors: (1) Wanyun Cui, Shanghai University of Finance and Economics, with equal contribution; (2) Qianle Wang, Shanghai University of Finance and Economics, with equal contribution. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Related Work 2 Related Work 3 Quantifying the Impact of Parameters on Model Performance & 4. Unified Mixed-Precision Training 3 Quantifying the Impact of Parameters on Model Performance & 4. Unified Mixed-Precision Training 5 Prevalence of Parameter Heterogeneity in LLMs 5 Prevalence of Parameter Heterogeneity in LLMs 6 Quantization Experiments and 6.1 Implementation Details 6 Quantization Experiments and 6.1 Implementation Details 6.2 Effect of Base LLM Quantization 6.2 Effect of Base LLM Quantization 6.3 Effect of Chat LLM Quantization 6.3 Effect of Chat LLM Quantization 6.4 Comparison of Parameter Selection Criteria, Conclusion, & References 6.4 Comparison of Parameter Selection Criteria, Conclusion, & References 6.4 Comparison of Parameter Selection Criteria To evaluate the effectiveness of our proposed impact-based parameter selection criterion, we conduct experiments comparing it with the commonly used magnitude-based criterion [18]. Table 5 presents the perplexity of LLaMA2-7B-3bit and LLaMA2-13B-3bit models, using both criteria for cherry parameter selection. From the results, it is evident that the impact-based criterion consistently outperforms the magnitude-based criterion across all settings. These results demonstrate that our proposed impact-based criterion is a more effective measure of parameter importance compared to the magnitude-based criterion. The impacts identify and preserve the most critical parameters during the quantization process. We think this justify the heterogeneity of parameter impacts against parameter magnitudes as we highlighted in § 5. The extensive experimental results presented in this section clearly demonstrate the superiority of CherryQ compared to existing quantization methods. By effectively identifying the critical cherry parameters and unifying the mixed-precision parameter optimization, CherryQ achieves state-of-theart performance for both base LLMs and chat LLMs. 7. Conclusion In this paper, we investigated the parameter heterogeneity phenomenon in LLMs. Our experiments on LLaMA2, Mistral, Gemma, and Vicuna models, consistently demonstrated that a small subset of parameters plays a crucial role in maintaining the model’s performance, while the majority of parameters can be quantized to ultra-low precision without significant degradation. This finding highlights the potential for efficient model compression and quantization techniques that take into account the heterogeneous nature of parameter importance. Motivated by this observation, we proposed a novel impact-based parameter selection criterion for quantization. Our method effectively identifies and preserves the most critical cherry parameters during the quantization process. We use a QAT framework for unified optimization of both cherry parameters and normal parameters. Extensive experiments demonstrate that CherryQ outperforms the commonly used magnitude-based criterion, achieving significantly lower perplexity scores and better downstream performance. The heterogeneity and proposed approach pave the way for more efficient deployment of LLMs in resource-constrained environments. A Effect of Chat LLM Quantization on MMLU We further evaluate the performance of CherryQ on the MMLU benchmark by quantizing the Vicuna1.5 model. As shown in Table 6, CherryQ outperforms both QAT and GPTQ in terms of average accuracy across almost all categories. References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. [3] Yoshua Bengio, Nicholas L´eonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432, 2013. [4] Yelysei Bondarenko, Markus Nagel, and Tijmen Blankevoort. Understanding and overcoming the challenges of efficient transformer quantization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7947–7969, 2021. [5] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023. URL https://lmsys. org/blog/2023-03- 30-vicuna, 3(5), 2023. [6] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35:30318–30332, 2022. [7] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024. [8] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. [9] Han Guo, Philip Greengard, Eric P Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311.12023, 2023. [10] Babak Hassibi, David G Stork, and Gregory J Wolff. Optimal brain surgeon and general network pruning. In IEEE international conference on neural networks, pages 293–299. IEEE, 1993. [11] John A Hertz. Introduction to the theory of neural computation. 2018. [12] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023. [13] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. arXiv preprint arXiv:2306.07629, 2023. [14] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342, 2018. [15] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989. [16] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations, 2020. [17] Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activationaware weight quantization for llm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023. [18] Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888, 2023. [19] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016. [20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. [21] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. In The Twelfth International Conference on Learning Representations, 2023. [22] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi`ere, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. [23] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [24] Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Fengwei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402–17414, 2022. [25] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087–38099. PMLR, 2023. [26] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Quant

Abstract

Empirical

Gong

Kale

Llama

Optimal

Rivi

scale

Rethinking AI Quantization: The Missing Piece in Model Efficiency

The Impact of Parameters on LLM Performance

Disproportionate Tech

Too Long; Didn't Read

The Future of AI Compression: Smarter Quantization Strategies

The Future of AI Compression: Smarter Quantization Strategies

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Can ChatGPT-Style Models Survive Quantization?

ResponseDial For ChatGPT: Optimize Your Workflow and Take Control of Your GPT Responses

Quantizing Large Language Models: Can We Maintain Accuracy?

The Science of "Cherry" Parameters: Why Some LLM Weights Matter More

The Perplexity Puzzle: How Low-Bit Quantization Affects AI Accuracy

Can ChatGPT-Style Models Survive Quantization?

ResponseDial For ChatGPT: Optimize Your Workflow and Take Control of Your GPT Responses

Quantizing Large Language Models: Can We Maintain Accuracy?

The Science of "Cherry" Parameters: Why Some LLM Weights Matter More

The Perplexity Puzzle: How Low-Bit Quantization Affects AI Accuracy

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps