Authors: Qinshi Zhang Latisha Besariani Hendra Mohan Chi Zijian Ding Authors: Authors: Qinshi Zhang Latisha Besariani Hendra Mohan Chi Zijian Ding Qinshi Zhang Qinshi Zhang Latisha Besariani Hendra Latisha Besariani Hendra Mohan Chi Mohan Chi Zijian Ding Zijian Ding Table Of Links Table Of Links ABSTRACT ABSTRACT 1 INTRODUCTION 1 INTRODUCTION 1 INTRODUCTION 2 SYSTEM DESIGN 2 SYSTEM DESIGN 2 SYSTEM DESIGN 3 RESULTS, DISCUSSION AND REFERENCES 3 RESULTS, DISCUSSION AND REFERENCES 3 RESULTS, DISCUSSION AND REFERENCES ABSTRACT ABSTRACT The emergence of Generative AI is catalyzing a paradigm shift in user interfaces from command-based to intent-based outcome specification. In this paper, we explore abstract-to-detailed task transitions in the context of frontend code generation as a step towards intent-based user interfaces, aiming to bridge the gap between abstract user intentions and concrete implementations. We introduce Frontend Diffusion, an end-to-end LLM-powered tool that generates high-quality websites from user sketches. The system employs a three-stage task transition process: sketching, writing, and coding. We demonstrate the potential of task transitions to reduce human intervention and communication costs in complex tasks. Our work also opens avenues for exploring similar approaches in other domains, potentially extending to more complex, interdependent tasks such as video production. INTRODUCTION The development of Generative AI, particularly the capabilities of Large Language Models (LLMs) in interpreting and executing natural language, may be viewed as heralding the first new user interface paradigm shift in 60 years [8]. This shift moves from command-based interactions, typified by command line interfaces and graphical user interfaces, to intent-based outcome specification [8]. This emerging intent-based paradigm potentially enables users to communicate their intentions to machines without necessarily translating them into machine-comprehensible commands, whether through programming languages or graphical buttons. This shift may foster interfaces that support more abstract human expressions, especially for command-intensive tasks such as coding [2, 3]. Currently, the interfaces for command-intensive tasks continue to necessitate substantial human intervention, where individuals typically specify incremental steps while AI generates corresponding code, akin to agile programming [11]. However, ongoing advancements in Generative AI capabilities suggest the potential for developing a framework that may bridge the gap between intentlevel expression and command-level implementation, potentially enhancing output quality while reducing the need for extensive human intervention. Previous research has demonstrated that Generative AI, such as Large Language Models (LLMs), can complete fixed-scope content curation tasks based on human intent without further intervention or intent iteration. For example, LLMs have shown promise in text summarization tasks [6]. However, Generative AI require greater human intervention for tasks involving increasing amounts of information [4, 7]. It motivates us to develop more effective scaffolding paradigm for Generative AI to respond to human intent and complete tasks in an agent-like manner. Recent research has indicated the feasibility of bridging intent expression in abstract tasks to concrete implementation at a more granular level. Examples include the transition from sketching to writing [1] and from design to data analysis [5]. Building upon these findings, we propose exploring more extensive intent-tocommand transitions, such as progressing from sketching to writing (planning) and ultimately to coding (see Figure 2). Our choice of website frontend generation as a user interface coding task [10] is motivated by its similarity to sketching. In both cases, the code or sketch serves as a representation of visual elements [9]. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv available on arxiv
