Welcome back to this series on uploading files to the web. Upload files with HTML Upload files with JavaScript Receiving file uploads with Node.js (Nuxt.js) Optimizing storage costs with Object Storage Optimizing delivery with a CDN Securing file uploads with malware scans The previous posts covered uploading files using and . The steps required: HTML JavaScript Using an element with the file attribute to access the files. <input> type Constructing with either a element or with the Fetch . HTTP requests <form> API Setting the request method “ “. POST Setting the request header to . Content-Type multipart/form-data Today, we are going to the backend to receive those requests and access the binary data from those files. multipart/form-data https://www.youtube.com/watch?v=34VJ1SPhtfk&embedable=true Some Background Most of the concepts in this tutorial should broadly apply across frameworks, runtimes, and languages, but the code examples will be more specific. I’ll be working within a project that runs in a environment. Nuxt has some specific ways of which require calling a global function called . Nuxt.js Node.js defining API routes defineEventHandler /**
 * @see https://nuxt.com/docs/guide/directory-structure/server
 * @see https://nuxt.com/docs/guide/concepts/server-engine
 * @see https://github.com/unjs/h3
 */
export default defineEventHandler((event) => {
  return { ok: true };
}); The argument provides access to work directly with the underlying Node.js request object (a.k.a. ) through . event IncomingMessage event.node.req So, we can write our Node-specific code in an abstraction, like a function called , that receives this Node request object and does something with it. doSomethingWithNodeRequest export default defineEventHandler((event) => {
  const nodeRequestObject = event.node.req;

  doSomethingWithNodeRequest(event.node.req);

  return { ok: true };
});

/**
 * @param {import('http').IncomingMessage} req
 */
function doSomethingWithNodeRequest(req) {
  // Do not specific stuff here
} Working directly with Node in this way means the code and concepts should apply regardless of whatever higher-level framework you’re working with. Ultimately, finish things up working in Nuxt.js. Dealing With in Node.js multipart/form-data In this section, we’ll dive into some low-level concepts that are good to understand, but not strictly necessary. Feel free to skip this section if you are already familiar with chunks and streams and buffers in Node.js. Uploading a file requires sending a request. In these requests, the browser will split the data into little “ ” and send them through the connection, one chunk at a time. This is necessary because files can be too large to send in as one massive payload. multipart/form-data chunks Chunks of data being sent over time make up what’s called a “ “. Streams are kind of hard to understand the first time around, at least for me. They deserve a full article (or many) on their own, so I’ll share in case you want to learn more. stream web.dev’s excellent guide Basically, a stream is sort of like a conveyor belt of data, where each chunk can be processed as it comes in. In terms of an HTTP request, the backend will receive parts of the request, one bit at a time. Node.js provides us with an event handler API through the request object’s method, which allows us to listen to “data” events as they are streamed into the backend. on /**
 * @param {import('http').IncomingMessage} req
 */
function doSomethingWithNodeRequest(req) {
  req.on("data", (data) => {
    console.log(data);
  }
} For example, when I upload , then look at the server’s console, I’ll see some weird things that look like this: a photo of Nugget making a cute yawny face I used a screenshot here to prevent assistive technology from reading that gibberish out loud. Could you imagine? These two pieces of garbled nonsense are called “ ”, and they represent the two chunks of data that made up the request stream containing the cute photo of Nugget. buffers A buffer is a storage in physical memory used to temporarily store data while it is being transferred from one place to another. MDN Buffers are another weird, low-level concept I have to explain when talking about working files in JavaScript. JavaScript doesn’t work directly on binary data, so we get to learn about buffers. It’s also OK if these concepts still feel a little vague. Understanding everything completely is not the important part right now, and as you continue to learn about file transfers, you’ll gain a better knowledge of how it all works together. Working with one partial chunk of data is not super useful. What we can do instead is rewrite our function into something we can work with: Return a to make the async syntax easy to work with. Promise Provide an to store the chunks of data to use later on. Array Listen for the “data” event, and add the chunks to our collection as they arrive. Listen to the “end” event, and convert the chunks into something we can work with. Resolve the with the final request payload. Promise We should also remember to handle “error” events. /**
 * @param {import('http').IncomingMessage} req
 */
function doSomethingWithNodeRequest(req) {
  return new Promise((resolve, reject) => {
    /** @type {any[]} */
    const chunks = [];
    req.on('data', (data) => {
      chunks.push(data);
    });
    req.on('end', () => {
      const payload = Buffer.concat(chunks).toString()
      resolve(payload);
    });
    req.on('error', reject);
  });
} And every time that the request receives some data, it pushes that data into the array of chunks. So with that function set up, we can actually that returned until the request has finished receiving all the data from the request stream, and log the resolved value to the console. await Promise export default defineEventHandler((event) => {
  const nodeRequestObject = event.node.req;

  const body = await doSomethingWithNodeRequest(event.node.req);
  console.log(body)

  return { ok: true };
}); This is the request body. Isn’t it beautiful? I honestly don’t even know what a screen reader would do with if this was plain text. If you upload an image file, it’ll probably look like an alien has hacked your computer. Don’t worry, it hasn’t. That’s literally what the text contents of that file look like. You can even try opening up an image file in a basic text editor and see the same thing. If I upload a more basic example, like a file with some plain text in it, the body might look like this: .txt Content-Disposition: form-data; name="file"; filename="dear-nugget.txt"
Content-Type: text/plain

I love you!
------WebKitFormBoundary4Ay52hDeKB5x2vXP-- Notice that the request is broken up into different sections for each form field. The sections are separated by the “form boundary”, which the browser will inject by default. I’ll skip going into excess details, so if you want to read more, check out on MDN. Content-Disposition The important thing to know is that requests are much more complex than just key/value pairs. multipart/form-data Most server frameworks provide built-in tools to access the body of a request. So we’ve actually reinvented the wheel. For example, Nuxt provides a global function. So we could have accomplished the same thing without writing our own code: readBody export default defineEventHandler((event) => {
  const nodeRequestObject = event.node.req;

  const body = await readBody(event.node.req);
  console.log(body)

  return { ok: true };
}); This works fine for other content types, but for , it has issues. The entire body of the request is being read into memory as one giant string of text. This includes the information, the form boundaries, and the form fields and values. multipart/form-data Content-Disposition Never mind the fact that the files aren’t even being written to disk. The big issue here is if a very large file is uploaded, it could consume all the memory of the application and cause it to crash. The solution is, once again, working with streams. When our server receives a chunk of data from the request stream, instead of storing it in memory, we can pipe it to a different stream. Specifically, we can send it to a stream that writes data to the file system using . createWriteStream As the chunks come in from the request, that data gets written to the file system, then released from memory. That’s about as far down as I want to go into the low-level concepts. Let’s go back up to solving the problem without reinventing the wheel. Use a Library to Stream Data Onto Disk Probably my best advice for handling file uploads is to reach for a library that does all this work for you: Parse requests. multipart/form-data Separate the files from the other form fields. Stream the file data into the file system. Provide you with the form field data as well as useful data about the files. Today, I’m going to be using this library called . You can install it with , then import it into your project. formidable npm install formidable import formidable from 'formidable'; Formidable works directly with the Node request object, which we conveniently already grabbed from the Nuxt event (“Wow, what amazing foresight!!!” 🤩). So we can modify our function to use formidable instead. It should still return a promise because formidable uses callbacks, but promises are nicer to work with. doSomethingWithNodeRequest Otherwise, we can mostly replace the contents of the function with formidable. We’ll need to create a formidable instance, use it to parse the request object, and as long as there isn’t an error, we can resolve the promise with a single object that contains both the form fields and the files. /**
 * @param {import('http').IncomingMessage} req
 */
function doSomethingWithNodeRequest(req) {
  return new Promise((resolve, reject) => {
    /** @see https://github.com/node-formidable/formidable/ */
    const form = formidable({ multiples: true })
    form.parse(req, (error, fields, files) => {
      if (error) {
        reject(error);
        return;
      }
      resolve({ ...fields, ...files });
    });
  });
} This provides us with a handy function to parse using promises and access the request’s regular form fields, as well as information about the files that were written to disk using streams. multipart/form-data Now, we can examine the request body: export default defineEventHandler((event) => {
  const nodeRequestObject = event.node.req;

  const body = await doSomethingWithNodeRequest(event.node.req);
  console.log(body)

  return { ok: true };
}); We should see an object containing all the form fields and their values, but for each file input, we’ll see an object that represents the uploaded file, and not the file itself. This object contains all sorts of useful information including its path on disk, name, , and more. mimetype {
  file-input-name: PersistentFile {
    _events: [Object: null prototype] { error: [Function (anonymous)] },
    _eventsCount: 1,
    _maxListeners: undefined,
    lastModifiedDate: 2023-03-21T22:57:42.332Z,
    filepath: '/tmp/d53a9fd346fcc1122e6746600',
    newFilename: 'd53a9fd346fcc1122e6746600',
    originalFilename: 'file.txt',
    mimetype: 'text/plain',
    hashAlgorithm: false,
    size: 13,
    _writeStream: WriteStream {
      fd: null,
      path: '/tmp/d53a9fd346fcc1122e6746600',
      flags: 'w',
      mode: 438,
      start: undefined,
      pos: undefined,
      bytesWritten: 13,
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 1,
      _maxListeners: undefined,
      [Symbol(kFs)]: [Object],
      [Symbol(kIsPerformingIO)]: false,
      [Symbol(kCapture)]: false
    },
    hash: null,
    [Symbol(kCapture)]: false
  }
} You’ll also notice that the is a hashed value. This is to ensure that if two files are uploaded with the same name, you will not lose data. You can, of course, modify how files are written to disk. newFilename Note that in a standard application, it’s a good idea to store some of this information in a persistent place, like a database, so you can easily find all the files that have been uploaded. But that’s not the point of this post. Now, there’s one more thing I want to fix. I only want to process requests with formidable. Everything else can be handled by a built-in body parser like the one we saw above. multipart/form-data So, I’ll create a “body” variable first, then check the request headers, and assign the value of the body based on the “Content-Type”. I’ll also rename my function to to be more explicit about what it does. parseMultipartNodeRequest Here’s what the whole thing looks like (note that is another built-in Nuxt function): getRequestHeaders import formidable from 'formidable';

/**
 * @see https://nuxt.com/docs/guide/concepts/server-engine
 * @see https://github.com/unjs/h3
 */
export default defineEventHandler(async (event) => {
  let body;
  const headers = getRequestHeaders(event);

  if (headers['content-type']?.includes('multipart/form-data')) {
    body = await parseMultipartNodeRequest(event.node.req);
  } else {
    body = await readBody(event);
  }
  console.log(body);

  return { ok: true };
});

/**
 * @param {import('http').IncomingMessage} req
 */
function parseMultipartNodeRequest(req) {
  return new Promise((resolve, reject) => {
    /** @see https://github.com/node-formidable/formidable/ */
    const form = formidable({ multiples: true })
    form.parse(req, (error, fields, files) => {
      if (error) {
        reject(error);
        return;
      }
      resolve({ ...fields, ...files });
    });
  });
} This way, we have an API that is robust enough to accept , plain text, or URL-encoded requests. multipart/form-data 📯📯📯 Finishing Up There’s no emoji rave horn, so those will have to do. We covered kind of a lot, so let’s do a little recap. When we upload a file using a request, the browser will send the data one chunk at a time, using a stream. That’s because we can’t put the entire file in the request object at once. multipart/form-data In Node.js, we can listen to the request’s “data” event to work with each chunk of data as it arrives. This gives us access to the request stream. Although we could capture all of that data and store it in memory, that’s a bad idea because a large file upload could consume all the server’s memory, causing it to crash. Instead, we can pipe that stream somewhere else, so each chunk is received, processed, then released from memory. One option is to use to create a that can write to the file system. fs.createWriteStream WritableStream Instead of writing our own low-level parser, we should use a tool like formidable. But we need to confirm that the data is coming from a request. Otherwise, we can use a standard body parser. multipart/form-data We covered a lot of low-level concepts and landed on a high-level solution. Hopefully, it all made sense, and you found this useful. If you have any questions or if something was confusing, please go ahead and reach out to me. I’m always happy to help. I’m having a lot of fun working on this series, and I hope you are enjoying it as well. Stick around for the rest of it :D Upload files with HTML Upload files with JavaScript Receiving file uploads with Node.js (Nuxt.js) Optimizing storage costs with Object Storage Optimizing delivery with a CDN Securing file uploads with malware scans Thank you so much for reading. If you liked this article, and want to support me, the best ways to do so are to , , and . share it sign up for my newsletter follow me on Twitter Also published . here

Stream File Uploads to S3 Object Storage and Save Money

How to Upload Files With JavaScript

Read My Stories

Newsletter

Too Long; Didn't Read

Make resilience your competitive advantage

How to Upload File Uploads in Node & Nuxt

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

4 ways CSS :has() can make your HTML forms even better

10 Cool CI/CD Tools For Your Project

27 Stories To Learn About Ssl

20 Essential Backend Tools For Developers

3 Most Common Ways to Connect your Node and React Applications

4 ways CSS :has() can make your HTML forms even better

10 Cool CI/CD Tools For Your Project

27 Stories To Learn About Ssl

20 Essential Backend Tools For Developers

3 Most Common Ways to Connect your Node and React Applications

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps