Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Survey with Industry Professionals 2 Survey with Industry Professionals 3 RQ1: Real-World use cases that necessitate output constraints 3 RQ1: Real-World use cases that necessitate output constraints 4 RQ2: Benefits of Applying Constraints to LLM Outputs and 4.1 Increasing Prompt-based development Efficiency 4 RQ2: Benefits of Applying Constraints to LLM Outputs and 4.1 Increasing Prompt-based development Efficiency 4.2 Integrating with Downstream Processes and Workflows 4.2 Integrating with Downstream Processes and Workflows 4.3 Satisfying UI and Product Requirements and 4.4 Improving User Experience, Trust, and Adoption 4.3 Satisfying UI and Product Requirements and 4.4 Improving User Experience, Trust, and Adoption 5 How to Articulate output constraints to LLMS and 5.1 The case for GUI: A Quick, Reliable, and Flexible Way of Prototyping Constraints 5 How to Articulate output constraints to LLMS and 5.1 The case for GUI: A Quick, Reliable, and Flexible Way of Prototyping Constraints 5.2 The Case for NL: More Intuitive and Expressive for Complex Constraints 5.2 The Case for NL: More Intuitive and Expressive for Complex Constraints 6 The Constraint maker Tool and 6.1 Iterative Design and User Feedback 6 The Constraint maker Tool and 6.1 Iterative Design and User Feedback 7 Conclusion and References 7 Conclusion and References A. The Survey Instrument A. The Survey Instrument 5 HOW TO ARTICULATE OUTPUT CONSTRAINTS TO LLMS Fig. 1 shows distributions of respondents’ preferences towards specifying output constraints either through GUI or natural language. An overarching observation is that respondents preferred using GUI to specify low-level constraints and natural language to express high-level constraints. We discuss their detailed rationale below: GUI to specify low-level constraints natural language to express high-level constraints 5.1 The case for GUI: A Quick, Reliable, and Flexible Way of Prototyping Constraints First and foremost, respondents considered GUIs particularly effective for defining “hard requirements,” providing more reliable results, and reducing ambiguity compared to natural language instructions. For example, one argued that choosing “boolean” as the output type via a GUI felt much more likely to be “honoured” compared to “typ[ing] that I want a Yes / No response [...] in a prompt.” Another claimed that “flagging a ‘JSON’ button” provides a much better user experience than “typing ‘output as JSON’ across multiple prompts.” In addition, respondents preferred using GUI when the intended constraint is “objective” and “quantifiable”, such as “use only items x,y,z,” or “a JSON with certain fields specified.” Moreover, respondents found GUI to be more flexible for rapid prototyping and experimentation (e.g., “when I want to play around with different numbers, moving a slider around seems easier than typing”). Finally, for novice LLM users, the range of choices afforded by a GUI constraint can help clarify the model’s capabilities and limitations, “making the model seems less like a black box.” One respondent drew from their experience working with text-to-image models to underscore this point: “by seeing “Illustration” as a possible output style [among others like “Photo realistic” or “Cartoon”], I became aware of [the model’s] capabilities.” “boolean” “honoured” “typ[ing] that I want a Yes / No response [...] in a prompt.” “flagging a ‘JSON’ button” “typing ‘output as JSON’ across multiple prompts.” “objective” and “quantifiable” “use only items x,y,z,” “a JSON with certain fields specified.” “when I want to play around with different numbers, moving a slider around seems easier than typing”). “by seeing “Illustration” as a possible output style [among others like “Photo realistic” or “Cartoon”], I became aware of [the model’s] capabilities.” This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. This paper is available on arxiv under CC BY-NC-SA 4.0 DEED license. available on arxiv Authors:
(1) Michael Xieyang Liu, Google Research, Pittsburgh, PA, USA (lxieyang@google.com);
(2) Frederick Liu, Google Research, Seattle, Washington, USA (frederickliu@google.com);
(3) Alexander J. Fiannaca, Google Research, Seattle, Washington, USA (afiannaca@google.com);
(4) Terry Koo, Google, Indiana, USA (terrykoo@google.com);
(5) Lucas Dixon, Google Research, Paris, France (ldixon@google.com);
(6) Michael Terry, Google Research, Cambridge, Massachusetts, USA (michaelterry@google.com);
(7) Carrie J. Cai, Google Research, Mountain View, California, USA (cjcai@google.com). Authors: Authors: (1) Michael Xieyang Liu, Google Research, Pittsburgh, PA, USA (lxieyang@google.com); (2) Frederick Liu, Google Research, Seattle, Washington, USA (frederickliu@google.com); (3) Alexander J. Fiannaca, Google Research, Seattle, Washington, USA (afiannaca@google.com); (4) Terry Koo, Google, Indiana, USA (terrykoo@google.com); (5) Lucas Dixon, Google Research, Paris, France (ldixon@google.com); (6) Michael Terry, Google Research, Cambridge, Massachusetts, USA (michaelterry@google.com); (7) Carrie J. Cai, Google Research, Mountain View, California, USA (cjcai@google.com).

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

Google

The Case for NL: More Intuitive and Expressive for Complex Constraints

Satisfying UI and Product Requirements and Improving User Experience, Trust, and Adoption

Structuring Technology FTW

Too Long; Didn't Read

Get Free Unlimited IP Geo Data With IPinfo Lite

How to Articulate Output Constraints to LLMS

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

How We Deployed an Online Survey to Users of an Internal Prompt-Based Prototyping Platform

"We Need Structured Output": Towards User-Centered Constraints on Large Language Model Output

How We Deployed an Online Survey to Users of an Internal Prompt-Based Prototyping Platform

Real-World Use Cases That Necessitate Output Constraints

The Benefits of Applying Constraints to LLM Outputs

How We Integrated Downstream Processes and Workflows

How We Deployed an Online Survey to Users of an Internal Prompt-Based Prototyping Platform

"We Need Structured Output": Towards User-Centered Constraints on Large Language Model Output

How We Deployed an Online Survey to Users of an Internal Prompt-Based Prototyping Platform

Real-World Use Cases That Necessitate Output Constraints

The Benefits of Applying Constraints to LLM Outputs

How We Integrated Downstream Processes and Workflows

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps