Functional Tokens: Enhancing Language Models for Function Calling

Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Related works 2 Related works 3 Methodology and 3.1 Causal language model as a classification model 3 Methodology and 3.1 Causal language model as a classification model 3.2 Functional token 3.2 Functional token 3.3 Dataset collection 3.3 Dataset collection 3.4 Model development and training 3.4 Model development and training 4 Experiments and 4.1 Android function calls 4 Experiments and 4.1 Android function calls 4.2 Extension to Vehicle, Yelp, and DoorDash function sets 4.2 Extension to Vehicle, Yelp, and DoorDash function sets 4.3 Full and partial training datasets and 4.4 Full training and LoRA training 4.3 Full and partial training datasets and 4.4 Full training and LoRA training 4.5 Parallel and nested function call and 4.6 Weighted loss function for special tokens 4.5 Parallel and nested function call and 4.6 Weighted loss function for special tokens 5 Discussion and future works and References 5 Discussion and future works and References Appendix Appendix A.1 Android function examples A.1 Android function examples A.2 Vehicle function examples A.2 Vehicle function examples 3.2 Functional token The critical innovation presented in this paper is the introduction of the functional token. Drawing parallels to the tokenization of natural language, we propose representing specific functions as functional tokens. We introduce a training methodology for these tokens, inspired by techniques used in natural language models for handling rare words. This training approach is inspired by the word2vec [28] framework, where contextual words enrich a token’s semantic representation. For instance, pretrained language models may initially struggle to recognize specialized terms such as PEGylation and Endosomal Escape from the domain of chemistry. However, these models can learn such terms through causal language modeling, leveraging corpora that include these specialized terms. Similarly, functional tokens can be learned using comparable methods. Our experiments have not identified any limitations on the number of functional tokens that can be defined, allowing users to map any specific function to a token. By incorporating the functional token, we aim to assign the same importance to functional tokens as other linguistic tokens. Unlike conventional linguistic tokens, our functional token does not possess inherent natural language meaning; instead, it represents specific actions encapsulated within the model. For smaller-scale models like Google Gemma 2B, conducting function calling can be challenging, although they excel at typical language completion tasks. The functional token enables the language model to transform a functional calling method into a standard completion task. As we define the actions for the language model, we can also characterize our model as a small action model. The integration of the functional token allows the model to concentrate on a fixed set of actions and perform effectively on these tasks. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv Authors: (1) Wei Chen, Stanford University, with equal contribution and a corresponding author {weichen6}@stanford.edu; (2) Zhiyuan Li, Stanford University and a corresponding author {zhiyuan8}@stanford.edu. Authors: Authors: (1) Wei Chen, Stanford University, with equal contribution and a corresponding author {weichen6}@stanford.edu; (2) Zhiyuan Li, Stanford University and a corresponding author {zhiyuan8}@stanford.edu.