Efficient On-Device LLMs: Function Calling and Fine-Tuning Strategies

Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Related works 2 Related works 3 Methodology and 3.1 Causal language model as a classification model 3 Methodology and 3.1 Causal language model as a classification model 3.2 Functional token 3.2 Functional token 3.3 Dataset collection 3.3 Dataset collection 3.4 Model development and training 3.4 Model development and training 4 Experiments and 4.1 Android function calls 4 Experiments and 4.1 Android function calls 4.2 Extension to Vehicle, Yelp, and DoorDash function sets 4.2 Extension to Vehicle, Yelp, and DoorDash function sets 4.3 Full and partial training datasets and 4.4 Full training and LoRA training 4.3 Full and partial training datasets and 4.4 Full training and LoRA training 4.5 Parallel and nested function call and 4.6 Weighted loss function for special tokens 4.5 Parallel and nested function call and 4.6 Weighted loss function for special tokens 5 Discussion and future works and References 5 Discussion and future works and References Appendix Appendix A.1 Android function examples A.1 Android function examples A.2 Vehicle function examples A.2 Vehicle function examples 2 Related works Deployment of on-device language models Due to memory limitations and lower inference speeds, deploying larger models on edge devices like PCs or smartphones is challenging. Nonetheless, efforts to deploy smaller-scale Large Language Models (LLMs) to edge devices are underway. Open-source models of manageable sizes, such as Gemma-2B, Gemma-7B, StableCode-3B [31], and Llama-7B [47], have been introduced. To enhance these models’ inference speed on devices, research initiatives like Llama cpp [24] have been developed. The MLC LLM framework [46] allows the operation of 7B language models on mobile phones and other edge devices, demonstrating compatibility across various hardware, including AMD, NVIDIA, Apple, and Intel GPUs. Deployment of on-device language models Function calling in language models Rapid advancements have been observed in the function-calling capabilities of smaller-scale models. Projects such as NexusRaven [42], Toolformer [37], ToolAlpaca [44], Gorrilla [30], ToolLlama [32] and Taskmatrix [20] have demonstrated that 7B and 13B models can call external APIs with efficacy comparable to GPT-4. The pioneering Octopus v1 project even enabled a 2B model to perform on par with GPT-4. This body of work utilizes a RAG-based method for function calling, where the model retrieves relevant functions from a large pool based on the user’s query, then generates a response using these functions as context. Function calling in language models Fine-tuning and adaptors of language models Fine-tuning language models has become a prevalent practice, with various efforts dedicated to this endeavor. LoRA [14] is often the method of choice for training models under GPU resource constraints. We use both full model training and LoRA training in our work, and compare their performance. A notable benefit of LoRA is its facilitation of extended functionalities in models, suggesting its potential to adapt our current framework for a broad range of applications. Fine-tuning and adaptors of language models This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv Authors: (1) Wei Chen, Stanford University, with equal contribution and a corresponding author {weichen6}@stanford.edu; (2) Zhiyuan Li, Stanford University and a corresponding author {zhiyuan8}@stanford.edu. Authors: Authors: (1) Wei Chen, Stanford University, with equal contribution and a corresponding author {weichen6}@stanford.edu; (2) Zhiyuan Li, Stanford University and a corresponding author {zhiyuan8}@stanford.edu.