Authors:
(1) Nicholas Farn, Microsoft Corporation {Microsoft Corporation {nifarn@microsoft.com};
(2) Richard Shin, Microsoft Corporation {eush@microsoft.com}. Table of Links Abstract and Intro Dataset Design Evaluation Methodology Experiments and Analysis Related Work Conclusion, Reproducibility, and References A. Complete list of tools B. Scenario Prompt C. Unrealistic Queries D. Nuances comparing prior work A COMPLETE LIST OF TOOLS We include the complete list of plugins and tools used in ToolTalk, and their corresponding descriptions. AccountTools This API contains tools for account management. • ChangePassword Changes the password of an account. • DeleteAccount Deletes a user’s account, requires user to be logged in. • GetAccountInformation Retrieves account information of logged in user. • LogoutUser Logs user out. • QueryUser Finds users given a username or email. • RegisterUser Register a new user. • ResetPassword Resets the password of a user using a verification code. • SendVerificationCode Initiates a password reset for a user by sending a verification code to a backup email. • UpdateAccountInformation Updates account information of a user. • UserLogin Logs in a user. Alarm This API contains tools for managing alarms. • AddAlarm Sets an alarm for a specific time. • DeleteAlarm Removes an alarm given an alarm id. • FindAlarms Finds alarms a user has set. Calendar This API lets a users manage events in their calendar. • CreateEvent Adds events to a user’s calendar. • DeleteEvent Deletes events from a user’s calendar. • ModifyEvent Allows modification of an existing event. • QueryCalendar Queries for events that occur in a time range. Email This API lets a user search and send emails. • SearchInbox Searches for emails matching filters returning 5 most recent results. • SendEmail Sends an email on behalf of a given user. Message This API lets a user search and send messages. • SearchMessages Searches messages matching filters returning 5 most recent results. • SendMessage Sends a message to another user. Reminder A suite of APIs for managing reminders for a TODO list. • AddReminder Add a reminder with an optional due date. • CompleteReminder Complete a reminder. • DeleteReminder Delete a reminder. • GetReminders Get a list of reminders. Weather Get weather information of a location. • CurrentWeather Get the current weather of a location. • ForecastWeather Get the 3-day weather forecast of a location. • HistoricWeather Get historic weather information of a location by month. B SCENARIO PROMPT Task You will be provided with a list of APIs. These APIs will have a description and a list of parameters and return types for each tool. Your task involves creating 3 varied, complex, and detailed user scenarios that require at least 5 API calls to complete involving at least 3 different APIs. One of these APIs will be explicitly provided and the other two will be chosen by you. For instance, given the APIs: SearchHotels, BookHotel, CancelBooking, GetNFLNews. Given that GetNFLNews is explicitly provided, your scenario should articulate something akin to: "The user wants to see if the Broncos won their last game (GetNFLNews). They then want to see if that qualifies them for the playoffs and who they will be playing against (GetNFLNews). The Broncos did make it into the playoffs, so the user wants watch the game in person. They want to look for hotels where the playoffs are occurring (GetNBANews + SearchHotels). After looking at the options, the user chooses to book a 3-day stay at the cheapest 4-star option (BookHotel)." This scenario exemplifies a scenario using 5 API calls. The scenario is complex, detailed, and concise as desired. The scenario also includes two APIs used in tandem, the required API, GetNBANews to search for the playoffs location and SearchHotels to find hotels based on the returned location. Usage of multiple APIs in tandem is highly desirable and will receive a higher score. Ideally each scenario should contain one or more instances of multiple APIs being used in tandem. Note that this scenario does not use all the APIs given and re-uses the " GetNBANews" API. Re-using APIs is allowed, but each scenario should involve at least 3 different APIs. Note that API usage is also included in the scenario, but exact parameters are not necessary. You must use a different combination of APIs for each scenario. All APIs must be used in at least one scenario. You can only use the APIs provided in the APIs section. Note that API calls are not explicitly mentioned and their uses are included in parentheses. This behaviour should be mimicked in your response. Deliver your response in this format: ‘‘‘ Scenario 1: <Scenario1>
Scenario 2: <Scenario2>
Scenario 3: <Scenario3> ‘‘‘ APIs ‘‘‘ {{API_DOCS}} ‘‘‘ Response Required API: {{REQUIRED_API}} Scenarios with >=5 API calls: ‘‘‘ - Scenario 1: C UNREALISTIC QUERIES Below are some examples of unrealistic queries gathered from various sources. These queries are useful for generating potentially complex tool interactions or unusual combinations of tools. However, as a consequence, they are unrealistic for various reasons such as forcing the usage of disparate APIs in situations a human is unlikely to ask for, explicitly asking for API endpoints which end users are unlikely to know of, or generally being unnaturally long and explicit. • “I’m working on a logistics project for my company and need to check the health of the SQUAKE API. Can you verify the API health by calling the ‘Checkhealth’ API endpoint? Additionally, I would like to retrieve the list of projects using the ‘Projects’ API endpoint.” (Qin et al., 2023b) • “How many singers have the average number of albums of singers in Beijing? Gives the square root of this number.” (Ruan et al., 2023) • “I am looking for x-large, red color women faux fur lined winter warm jacket coat, and price lower than 70.00 dollars.” Yao et al. (2022a) • “Can you retrieve the contact details of the ‘Gondrand’ customs agency in New Caledonia? I’m particularly interested in their postal code, email, name, and phone number. Also, please provide a list of all available transitaires. Begin!!!” Qin et al. (2023b) D NUANCES COMPARING PRIOR WORK Nuances comparing prior work from Table 5. ReAct and ART evaluate on hard QA tasks, but these tasks traditionally do not require the usage of tools to complete. We also note API-Bank fits all criteria. However, its level 1-2 examples are automated but simple. In comparison, its level 3 examples are complex but require manual evaluation. ToolBench’s tasks require hard to use tool, but have shallow solution paths. Their more complex environments have deeper solution paths, but re-use the existing datasets of WebShop (Yao et al., 2022a) and TableTop (Liang et al., 2023). This paper is available on arxiv under CC 4.0 license. Authors: (1) Nicholas Farn, Microsoft Corporation {Microsoft Corporation {nifarn@microsoft.com}; (2) Richard Shin, Microsoft Corporation {eush@microsoft.com}. Authors: Authors: (1) Nicholas Farn, Microsoft Corporation {Microsoft Corporation {nifarn@microsoft.com}; (2) Richard Shin, Microsoft Corporation {eush@microsoft.com}. Table of Links Abstract and Intro Abstract and Intro Dataset Design Dataset Design Evaluation Methodology Evaluation Methodology Experiments and Analysis Experiments and Analysis Related Work Related Work Conclusion, Reproducibility, and References Conclusion, Reproducibility, and References A. Complete list of tools A. Complete list of tools B. Scenario Prompt B. Scenario Prompt C. Unrealistic Queries C. Unrealistic Queries D. Nuances comparing prior work D. Nuances comparing prior work A COMPLETE LIST OF TOOLS We include the complete list of plugins and tools used in ToolTalk, and their corresponding descriptions. AccountTools This API contains tools for account management. AccountTools • ChangePassword Changes the password of an account. ChangePassword • DeleteAccount Deletes a user’s account, requires user to be logged in. DeleteAccount • GetAccountInformation Retrieves account information of logged in user. GetAccountInformation • LogoutUser Logs user out. LogoutUser • QueryUser Finds users given a username or email. QueryUser • RegisterUser Register a new user. • ResetPassword Resets the password of a user using a verification code. ResetPassword • SendVerificationCode Initiates a password reset for a user by sending a verification code to a backup email. SendVerificationCode • UpdateAccountInformation Updates account information of a user. UpdateAccountInformation • UserLogin Logs in a user. UserLogin Alarm This API contains tools for managing alarms. Alarm • AddAlarm Sets an alarm for a specific time. AddAlarm • DeleteAlarm Removes an alarm given an alarm id. DeleteAlarm • FindAlarms Finds alarms a user has set. FindAlarms Calendar This API lets a users manage events in their calendar. Calendar • CreateEvent Adds events to a user’s calendar. CreateEvent • DeleteEvent Deletes events from a user’s calendar. DeleteEvent • ModifyEvent Allows modification of an existing event. ModifyEvent • QueryCalendar Queries for events that occur in a time range. QueryCalendar Email This API lets a user search and send emails. Email • SearchInbox Searches for emails matching filters returning 5 most recent results. SearchInbox • SendEmail Sends an email on behalf of a given user. SendEmail Message This API lets a user search and send messages. Message • SearchMessages Searches messages matching filters returning 5 most recent results. SearchMessages • SendMessage Sends a message to another user. SendMessage Reminder A suite of APIs for managing reminders for a TODO list. Reminder • AddReminder Add a reminder with an optional due date. • CompleteReminder Complete a reminder. • DeleteReminder Delete a reminder. • GetReminders Get a list of reminders. AddReminder Weather Get weather information of a location. Weather • CurrentWeather Get the current weather of a location. CurrentWeather • ForecastWeather Get the 3-day weather forecast of a location. ForecastWeather • HistoricWeather Get historic weather information of a location by month. HistoricWeather B SCENARIO PROMPT Task You will be provided with a list of APIs. These APIs will have a description and a list of parameters and return types for each tool. Your task involves creating 3 varied, complex, and detailed user scenarios that require at least 5 API calls to complete involving at least 3 different APIs. One of these APIs will be explicitly provided and the other two will be chosen by you. For instance, given the APIs: SearchHotels, BookHotel, CancelBooking, GetNFLNews. Given that GetNFLNews is explicitly provided, your scenario should articulate something akin to: "The user wants to see if the Broncos won their last game (GetNFLNews). They then want to see if that qualifies them for the playoffs and who they will be playing against (GetNFLNews). The Broncos did make it into the playoffs, so the user wants watch the game in person. They want to look for hotels where the playoffs are occurring (GetNBANews + SearchHotels). After looking at the options, the user chooses to book a 3-day stay at the cheapest 4-star option (BookHotel)." This scenario exemplifies a scenario using 5 API calls. The scenario is complex, detailed, and concise as desired. The scenario also includes two APIs used in tandem, the required API, GetNBANews to search for the playoffs location and SearchHotels to find hotels based on the returned location. Usage of multiple APIs in tandem is highly desirable and will receive a higher score. Ideally each scenario should contain one or more instances of multiple APIs being used in tandem. Note that this scenario does not use all the APIs given and re-uses the " GetNBANews" API. Re-using APIs is allowed, but each scenario should involve at least 3 different APIs. Note that API usage is also included in the scenario, but exact parameters are not necessary. You must use a different combination of APIs for each scenario. All APIs must be used in at least one scenario. You can only use the APIs provided in the APIs section. Note that API calls are not explicitly mentioned and their uses are included in parentheses. This behaviour should be mimicked in your response. Deliver your response in this format: ‘‘‘ Scenario 1: <Scenario1> Scenario 2: <Scenario2> Scenario 3: <Scenario3> ‘‘‘ Scenario 1: <Scenario1> Scenario 2: <Scenario2> Scenario 3: <Scenario3> ‘‘‘ APIs ‘‘‘ {{API_DOCS}} ‘‘‘ Response Required API: {{REQUIRED_API}} Scenarios with >=5 API calls: ‘‘‘ - Scenario 1: C UNREALISTIC QUERIES Below are some examples of unrealistic queries gathered from various sources. These queries are useful for generating potentially complex tool interactions or unusual combinations of tools. However, as a consequence, they are unrealistic for various reasons such as forcing the usage of disparate APIs in situations a human is unlikely to ask for, explicitly asking for API endpoints which end users are unlikely to know of, or generally being unnaturally long and explicit. • “I’m working on a logistics project for my company and need to check the health of the SQUAKE API. Can you verify the API health by calling the ‘Checkhealth’ API endpoint? Additionally, I would like to retrieve the list of projects using the ‘Projects’ API endpoint.” (Qin et al., 2023b) • “How many singers have the average number of albums of singers in Beijing? Gives the square root of this number.” (Ruan et al., 2023) • “I am looking for x-large, red color women faux fur lined winter warm jacket coat, and price lower than 70.00 dollars.” Yao et al. (2022a) • “Can you retrieve the contact details of the ‘Gondrand’ customs agency in New Caledonia? I’m particularly interested in their postal code, email, name, and phone number. Also, please provide a list of all available transitaires. Begin!!!” Qin et al. (2023b) D NUANCES COMPARING PRIOR WORK Nuances comparing prior work from Table 5. ReAct and ART evaluate on hard QA tasks, but these tasks traditionally do not require the usage of tools to complete. We also note API-Bank fits all criteria. However, its level 1-2 examples are automated but simple. In comparison, its level 3 examples are complex but require manual evaluation. ToolBench’s tasks require hard to use tool, but have shallow solution paths. Their more complex environments have deeper solution paths, but re-use the existing datasets of WebShop (Yao et al., 2022a) and TableTop (Liang et al., 2023). This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Advancing Conversational AI with Complex Tool Orchestration

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Re: Defeating Nondeterminism in LLM Inference, The Future is Predictable

ToolTalk: Benchmarking the Future of Tool-Using AI Assistants

How to Create Realistic AI Conversations

Action vs Non-action Tools: Evaluating AI Assistant Correctness

Analyzing AI Assistant Performance: Lessons from ToolTalk's Analysis of GPT-3.5 and GPT-4

Re: Defeating Nondeterminism in LLM Inference, The Future is Predictable

ToolTalk: Benchmarking the Future of Tool-Using AI Assistants

How to Create Realistic AI Conversations

Action vs Non-action Tools: Evaluating AI Assistant Correctness

Analyzing AI Assistant Performance: Lessons from ToolTalk's Analysis of GPT-3.5 and GPT-4

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps