How to Get Faster Responses With HTTP Streaming: AI For Web Devs

Welcome back to where we’re building web applications with AI tooling. this series Intro & Setup Your First AI Prompt Streaming Responses How Does AI Work Prompt Engineering AI-Generated Images Security & Reliability Deploying In , we got jokes into our application from . It worked, but the user experience suffered because we had to wait until the API completed the entire response before updating the client. the previous post AI-generated Qwik OpenAI API A better experience, as you’ll know if you’ve used any AI chat tools, is to respond as soon as each bit of text is generated. It becomes a sort of teletype effect. That’s what we’re going to build today using . HTTP streams https://www.youtube.com/watch?v=GkyHBwUA0EQ&embedable=true Prerequisites Before we get into streams, we need to explore something with a Qwik quirk related to HTTP requests. If we examine the current POST request being sent by the form, we can see that the returned payload isn’t just the plain text we returned from our action handler. Instead, it’s this sort of . serialized data This is the result of how the lazy loads assets and is necessary to properly handle the data as it comes back. Unfortunately, this prevents standard streaming responses. Qwik Optimizer So while and the components are super handy, we’ll have to do something else. routeAction$ Form To their credit, the Qwik team does provide a . However, it involves their function and . This would probably be the right approach if we’re talking strictly about Qwik, but this series is for everyone. I’ll avoid this implementation as it’s too specific to Qwik, and focus on broadly applicable concepts instead. well-documented approach for streaming responses server$ async generator functions Refactor Server Logic It sucks that we can’t use route actions because they’re great. So what can we use? Qwik City offers a few options. The best I found is . They provide enough access to primitive tools that we can accomplish what we need, and the concepts will apply to other contexts besides Qwik. middleware Middleware is essentially a set of functions that we can inject at various points within the request lifecycle of our route handler. We can define them by exporting named constants for the hooks we want to target ( , , , , ). onRequest onGet onPost onPut onDelete So, instead of relying on a route action, we can use a middleware that hooks into any POST request by exporting an middleware. In order to support streaming, we’ll want to return a standard object. We can do so by creating a Response object and passing it to the method. onPost Response requestEvent.send() Here’s a basic (non-streaming) example: /** @type {import('@builder.io/qwik-city').RequestHandler} */ export const onPost = (requestEvent) => { requestEvent.send(new Response('Hello Squirrel!')) } Before we tackle streaming, let’s get the same functionality from the old route action implemented with middleware. We can copy most of the code into the middleware, but we won’t have access to . onPost formData Fortunately, we can recreate that data from the method. We’ll also want to use to respond with the OpenAI data instead of a statement. requestEvent.parseBody() requestEvent.send() return /** @type {import('@builder.io/qwik-city').RequestHandler} */ export const onPost = async (requestEvent) => { const OPENAI_API_KEY = requestEvent.env.get('OPENAI_API_KEY') const formData = await requestEvent.parseBody()

const prompt = formData.prompt const body = { model: 'gpt-3.5-turbo', messages: [{ role: 'user', content: prompt }] }

const response = await fetch('https://api.openai.com/v1/chat/completions', { // ... fetch options }) const data = await response.json()

const responseBody = data.choices[0].message.content

requestEvent.send(new Response(responseBody)) } Refactor Client Logic Replacing the route actions has the unfortunate side effect of meaning we also can’t use the component anymore. We’ll have to use a regular element and recreate all the benefits we had before, including sending HTTP requests with , tracking the loading state, and accessing the results. HTML JavaScript Let’s refactor our client side to support those features again. We can break these requirements down to needing two things, a JavaScript solution for submitting forms and a reactive state for managing loading states and results. I’ve covered submitting HTML forms with JavaScript in depth several times in the past: Make Beautifully Resilient Apps With Progressive Enhancement How to Upload Files with JavaScript Building Super Powered HTML Forms with JavaScript So today, I’ll just share the snippet, which I put inside a file at the root of my project. utils.js This function accepts an then constructs a request based on the form attributes and returns the resulting : jsFormSubmit HTMLFormElement fetch Promise /** * @param {HTMLFormElement} form */ export function jsFormSubmit(form) { const url = new URL(form.action) const formData = new FormData(form) const searchParameters = new URLSearchParams(formData)

/** @type {Parameters [1]} */ const fetchOptions = { method: form.method }

if (form.method.toLowerCase() === 'post') { fetchOptions.body = form.enctype === 'multipart/form-data' ? formData : searchParameters } else { url.search = searchParameters }

return fetch(url, fetchOptions) } This generic function can be used to submit any HTML form, so it’s handy to use in a event handler. Sweet! submit As for the reactive data, Qwik provides two options, and . I prefer , which allows us to create an object whose properties are . Meaning changes to the object’s properties will automatically be reflected wherever they are referenced in the UI. useStore useSignal useStore reactive We can use to create a “state” object in our component to track the loading state of the HTTP request as well as the text response. useStore import { $, component$, useStore } from "@builder.io/qwik"; // other setup logic

export default component$(() => { const state = useStore({ isLoading: false, text: '', })

// other component logic }) Next, we can update the template. Since we can no longer use the object we had before, we can replace references from and to and , respectively (don’t ask me why I changed the property names 🤷♂️). I’ll also add a “submit” event handler to the form called , which we’ll look at shortly. action action.isRunning action.value state.isLoading state.text handleSbumit Prompt Tell me a joke

{state.isLoading ? 'One sec...' : 'Tell me'}

{state.text && ( {state.text}

)} Note that the does not explicitly provide an attribute. By default, an HTML form will submit data to the current URL, so we only need to set the to POST and submit this form to trigger the middleware we defined earlier. action method onPost Now, the last step to get this refactor working is defining . Just like we did in , we need to wrap an event handler inside Qwik’s function. handleSubmit the previous post $ Inside the event handler, we’ll want to clear out any previous data from , set to , then pass the form’s DOM node to our fancy function. state.text state.isLoading true jsFormSubmit This should submit the HTTP request for us. Once it comes back, we can update with the response body, and return to . state.text state.isLoading false const handleSubmit = $(async (event) => { state.text = '' state.isLoading = true

/** @type {HTMLFormElement} */ const form = event.target

const response = await jsFormSubmit(form) state.text = await response.text() state.isLoading = false }) OK! We should now have a client-side form that uses JavaScript to submit an HTTP request to the server while tracking the loading and response states, and updating the UI accordingly. That was a lot of work to get the same solution we had before but with fewer features. BUT the key benefit is we now have direct access to the platform primitives we need to support streaming. Enable Streaming on the Server Before we start streaming responses from OpenAI, I think it’s helpful to start with a very basic example to get a better grasp of streams. Streams allow us to send small chunks of data over time. So as an example, let’s print out some iconic David Bowie lyrics in tempo with the song, “ “. Space Oddity When we construct our Response object, instead of passing plain text, we’ll want to pass a stream. We’ll create the stream shortly, but here’s the idea: /** @type {import('@builder.io/qwik-city').RequestHandler} */ export const onPost = (requestEvent) => { requestEvent.send(new Response(stream)) } We’ll create a very rudimentary using the constructor and pass it an . This optional parameter can be an object with a method that’s called when the stream is constructed. ReadableStream ReadableStream optional parameter start The start method is responsible for the steam’s logic and has access to the stream , which is used to send data and close the stream. controller const stream = new ReadableStream({ start(controller) { // Stream logic goes here } }) OK, let’s plan out that logic. We’ll have an array of song lyrics and a function to ‘sing’ them (pass them to the stream). The function will take the first item in the array and pass that to the stream using the method. sing controller.enqueue() If it’s the last lyric in the list, we can close the stream with . Otherwise, the method can call itself again after a short pause. controller.close() sing const stream = new ReadableStream({ start(controller) { const lyrics = ['Ground', ' control', ' to major', ' Tom.'] function sing() { const lyric = lyrics.shift()

controller.enqueue(lyric)

if (lyrics.length < 1) { controller.close() } else { setTimeout(sing, 1000) } } sing() } }) So each second, for four seconds, this stream will send out the lyrics “Ground control to major Tom.” Slick! Because this stream will be used in the body of the Response, the connection will remain open for four seconds until the response completes. But the frontend will have access to each chunk of data as it arrives, rather than waiting the full four seconds. This doesn’t speed up the total response time (in some cases, streams can increase response times), but it does allow for a faster-perceived response, and that makes a better user experience. Here’s what my code looks like: /** @type {import('@builder.io/qwik-city').RequestHandler} */ export const onPost: RequestHandler = async (requestEvent) => { const stream = new ReadableStream({ start(controller) { const lyrics = ['Ground', ' control', ' to major', ' Tom.'] function sing() { const lyric = lyrics.shift()

controller.enqueue(lyric)

if (lyrics.length < 1) { controller.close() } else { setTimeout(sing, 1000) } } sing() } })

requestEvent.send(new Response(stream)) } Unfortunately, as it stands right now, the client will still be waiting four seconds before seeing the entire response, and that’s because we weren’t expecting a streamed response. Let’s fix that. Enable Streaming on the Client Even when dealing with streams, the default browser behavior when receiving a response is to wait for it to complete. In order to get the behavior we want, we’ll need to use client-side JavaScript to make the request and process the streaming body of the response. We’ve already tackled that first part inside our function. Let’s start processing that response body. handleSubmit We can access the from the response body’s method. This stream will have its own method that we can use to access the next chunk of data as well as the information if the response is done streaming or not. ReadableStream getReader() read() The only ‘gotcha’ is that the data in each chunk doesn’t come in as text; it comes in as a , which is “an array of 8-bit unsigned integers.” It’s basically the representation of the binary data, and you don’t really need to understand any deeper than that unless you want to sound very smart at a party (or boring). Uint8Array The important thing to understand is that on their own, these data chunks aren’t very useful. To get something we use, we’ll need to decode each chunk of data using a . can TextDecoder Ok, that’s a lot of theory. Let’s break down the logic and then look at some code. When we get the response back, we need to: Grab the reader from the response body using response.body.getReader() Setup a decoder using and a variable to track the streaming status. TextDecoder Process each chunk until the stream is complete, with a loop that does this: while Grab the next chunk’s data and stream status. Decode the data and use it to update our app’s . state.text Update the streaming status variable, terminating the loop when complete. Update the loading state of the app by setting to . state.isLoading false The new function should look something like this: handleSubmit const handleSubmit = $(async (event) => { state.text = '' state.isLoading = true

/** @type {HTMLFormElement} */ const form = event.target

const response = await jsFormSubmit(form) // Parse streaming body const reader = response.body.getReader() const decoder = new TextDecoder() let isStillStreaming = true

while(isStillStreaming) { const {value, done} = await reader.read() const chunkValue = decoder.decode(value) state.text += chunkValue

isStillStreaming = !done } state.isLoading = false }) Now, when I submit the form, I see something like: “Ground control to major Tom.” Hell yeah!!! OK, most of the work is down. Now, we just need to replace our demo stream with the OpenAI response. Stream OpenAI Response Looking back at our original implementation, the first thing we need to do is modify the request to OpenAI to let them know that we would like a streaming response. We can do that by setting the property in the payload to . stream fetch true const body = { model: 'gpt-3.5-turbo', messages: [{ role: 'user', content: prompt }], stream: true }

const response = await fetch('https://api.openai.com/v1/chat/completions', { method: 'post', headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${OPENAI_API_KEY}`, }, body: JSON.stringify(body) }) I used fetch and custom streams because, at the time of writing, the module on NPM did not properly support streaming responses. UPDATE 2023/11/15: openai That issue has been fixed, and I think a better solution would be to use that module and pipe their data through a to send it to the client. That version is not reflected here. TransformStream Next, we could pipe the response from OpenAI directly to the client, but we might not want to do that. The data they send doesn’t really align with what we want to send to the client because it looks like this (two chunks, one with data, and one representing the end of the stream): data: {"id":"chatcmpl-4bJZRnslkje3289REHFEH9ej2","object":"chat.completion.chunk","created":1690319476,"model":"gpt-3.5-turbo-0613","choiced":[{"index":0,"delta":{"content":"Because"},"finish_reason":"stop"}]} data: [DONE] Instead, what we’ll do is create our own stream, similar to the David Bowie lyrics, that will do some setup, enqueue chunks of data into the stream, and close the stream. Let’s start with an outline: const stream = new ReadableStream({ async start(controller) { // Any setup before streaming // Send chunks of data // Close stream } }) Since we’re dealing with a streaming fetch response from OpenAI, a lot of the work we need to do here can actually be copied from the client-side stream handling. This part should look familiar: const reader = response.body.getReader() const decoder = new TextDecoder() let isStillStreaming = true

while(isStillStreaming) { const {value, done} = await reader.read() const chunkValue = decoder.decode(value)

// Here's where things will be different

isStillStreaming = !done } This snippet was taken almost directly from the frontend stream processing example. The only difference is that we need to treat the data coming from OpenAI slightly differently. As we say, the chunks of data they send up will look something like” “. data: [JSON data or done] Another gotcha is that every once in a while, they’ll actually slip in TWO of these data strings in a single streaming chunk. So, here’s what I came up with for processing the data. Create a to grab the rest of the string after “ “. Regular Expression data: For the unlikely event there are more than one data strings, use a while loop to process every match in the string. If the current matches the closing condition (“ “) close the stream. [DONE] Otherwise, parse the data as JSON and enqueue the first piece of text from the list of options ( ). Fall back to an empty string if none is present. json.choices[0].delta.content Lastly, in order to move to the next match, if there is one, we can use . RegExp.exec() The logic is quite abstract without looking at code, so here’s what the whole stream looks like now: const stream = new ReadableStream({ async start(controller) { // Do work before streaming const reader = response.body.getReader() const decoder = new TextDecoder() let isStillStreaming = true

while(isStillStreaming) { const {value, done} = await reader.read() const chunkValue = decoder.decode(value)

/** * Captures any string after the text `data: ` * @see https://regex101.com/r/R4QgmZ/1 */ const regex = /data:\s*(.*)/g let match = regex.exec(chunkValue)

while (match !== null) { const payload = match[1] // Close stream if (payload === '[DONE]') { controller.close() break } else { try { const json = JSON.parse(payload) const text = json.choices[0].delta.content || ''

// Send chunk of data controller.enqueue(text) match = regex.exec(chunkValue) } catch (error) { const nextChunk = await reader.read() const nextChunkValue = decoder.decode(nextChunk.value) match = regex.exec(chunkValue + nextChunkValue) } } }

isStillStreaming = !done } } }) I discovered that OpenAI API sometimes returns the JSON payload across two streams. So the solution is to use a block around the , and in the case that it fails, reassign the variable to the current chunk value plus the next chunk value. The code above has the updated snippet. UPDATE 2023/11/15: try/catch JSON.parse match Review That should be everything we need to get streaming working. Hopefully, it all makes sense, and you got it working on your end. I think it’s a good idea to review the flow to make sure we’ve got it: The user submits the form, which gets intercepted and sent with JavaScript. This is necessary to process the stream when it returns. The request is received by the action handler which forwards the data to the OpenAI API along with the setting to return the response as a stream. The OpenAI response will be sent back as a stream of chunks, some of which contain JSON and the last one being “ “. [DONE] Instead of passing the stream to the action response, we create a new stream to use in the response. Inside this stream, we process each chunk of data from the OpenAI response and convert it to something more useful before enqueuing it for the action response stream. When the OpenAI stream closes, we also close our action stream. The JavaScript handler on the client side will also process each chunk of data as it comes in and update the UI accordingly. Conclusion The app is working. It’s pretty cool. We covered a lot of interesting things today. Streams are very powerful, but also challenging, and especially when working within Qwik, there are a couple of little gotchas. But because we focused on low-level fundamentals, these concepts should apply across any framework. As long as you have access to the platform and primitives like streams, requests, and response objects then this should work. That’s the beauty of fundamentals. I think we got a pretty decent application going now. The only problem is right now we’re using a generic text input and asking users to fill in the entire prompt themselves. In fact, they can put in whatever they want. We’ll want to fix that in a future post, but the next post is going to step away from code and focus on understanding how the AI tools actually work. I hope you’ve been enjoying this series and come back for the rest of it. Intro & Setup Your First AI Prompt Streaming Responses How Does AI Work Prompt Engineering AI-Generated Images Security & Reliability Deploying Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to , , and . share it sign up for my newsletter follow me on Twitter First published . here