How to Download Files Using Golang and gRPC

Sometimes we need to download files from some storage. Files have some metadata like name, size, and extension (maybe we need more metadata, but let’s keep it simple), also files should implement io.Reader interface for getting bytes of this content.

So file struct may look like this:

package file

import (
   "io"
)

func NewFile(name string, extension string, size int, r io.Reader) *File {
   return &File{Name: name, Extension: extension, Size: size, r: r}
}

type File struct {
   r         io.Reader
   Name      string
   Extension string
   Size      int
}

func (f *File) Read(p []byte) (n int, err error) {
   return f.r.Read(p)
}

I want to implement sending this struct from the server to the client using Golang/gRPC.

gRPC API

Before beginning to implement the server and the client we need to have API. Our API is quite simple:

syntax = 'proto3';

package api.v1;

option go_package = "github.com/andrey-berenda/go-filestorage/generated/v1;api";

service FileService {
  rpc Download(DownloadRequest) returns(stream DownloadResponse);
}

message DownloadRequest{
  string id = 1;
}

message DownloadResponse{
  bytes chunk = 1;
}

As we can see in our API there is no file metadata: metadata will be in headers because we need to send it only once and headers are the best candidate for achieving this goal.

Because of this we need to have Metadata() method and constructor from metadata in our struct:

package file

import (
   "io"
   "strconv"

   "google.golang.org/grpc/metadata"
)

var fileNameHeader = "file-name"
var fileTypeHeader = "file-type"
var fileSizeHeader = "file-size"

func (f *File) Metadata() metadata.MD {
   return metadata.New(map[string]string{
      fileNameHeader: f.Name,
      fileTypeHeader: f.Extension,
      fileSizeHeader: strconv.Itoa(f.Size),
   })
}

func NewFromMetadata(md metadata.MD, r io.Reader) *File {
   var name, fileType string
   var size int

   if names := md.Get(fileNameHeader); len(names) > 0 {
      name = names[0]
   }
   if types := md.Get(fileTypeHeader); len(types) > 0 {
      fileType = types[0]
   }
   if sizes := md.Get(fileSizeHeader); len(sizes) > 0 {
      size, _ = strconv.Atoi(sizes[0])
   }

   return &File{Name: name, Extension: fileType, Size: size, r: r}
}

Server implementation

The interface for the server looks like this:

type FileServiceServer interface {
   Download(req *pb.DownloadRequest, server pb.FileService_DownloadServer) error
}

At first, we need to check if the client sent the file id:

if req.GetId() == "" {
   return status.Error(codes.InvalidArgument, "id is required")
}

Then we get files from some storage:

f, ok := getFile(req.Id)
if !ok {
   return status.Error(codes.NotFound, "file is not found")
}

In my example, the storage is very simple and keeps everything in memory(only one file named “gopher”):

//go:embed static/gopher.png
var gopher []byte

func getFile(fileID string) (*file.File, bool) {
   if fileID != "gopher" {
      return nil, false
   }
   return file.NewFile("gopher", "png", len(gopher), bytes.NewReader(gopher)), true
}

After getting the file we send the headers:

err := server.SendHeader(f.Metadata())
if err != nil {
   return status.Error(codes.Internal, "error during sending header")
}

And send content:

   const chunkSize = 1024 * 3
   chunk := &pb.DownloadResponse{Chunk: make([]byte, chunkSize)}
   var n int

Loop:
   for {
      n, err = f.Read(chunk.Chunk)
      switch err {
      case nil:
      case io.EOF:
         break Loop
      default:
         return status.Errorf(codes.Internal, "io.ReadAll: %v", err)
      }
      chunk.Chunk = chunk.Chunk[:n]
      serverErr := server.Send(chunk)
      if serverErr != nil {
         return status.Errorf(codes.Internal, "server.Send: %v", serverErr)
      }
   }
   return nil

The server is implemented. The chunk size is not so big, I think we can increase it to 1MB or something like that, but keep the chunk size as a multiple of 3 (I will explain later why it is important).

All code of the server is here:

Client implementation

I want to implement the method for the Client:

type Client struct {
   client pb.FileServiceClient
}

func (c Client) GetFile(ctx context.Context, fileID string)(*file.File, error)

For reaching this we need to make a gRPC request to the server:

response, err := c.client.Download(
   ctx,
   &pb.DownloadRequest{Id: fileID},
)
if err != nil {
   return nil, fmt.Errorf("client.LoadFile: %w", err)
}

After that we need to get headers from the context:

md, err := response.Header()
if err != nil {
   return nil, fmt.Errorf("response.Header: %w", err)
}

And finally, create io.Pipe and copy all content from response to *io.PipeWriter:

r, w := io.Pipe()
f := file.NewFromMetadata(md, r)
go copyFromResponse(w, response)
return f, nil

Implementation of the copying is here:

func copyFromResponse(w *io.PipeWriter, res pb.FileService_DownloadClient) {
   message := new(pb.DownloadResponse)
   var err error
   for {
      err = res.RecvMsg(message)
      if err == io.EOF {
         _ = w.Close()
         break
      }
      if err != nil {
         _ = w.CloseWithError(err)
         break
      }
      if len(message.GetChunk()) > 0 {
         _, err = w.Write(message.Chunk)
         if err != nil {
            _ = res.CloseSend()
            break
         }
      }
      message.Chunk = message.Chunk[:0]
   }
}

The client is implemented. And here is all code of the client.

Bonus

When you send files using simple HTTP there is one advantage: you can get files by id using only bash without writing any code in Golang. Is there a way to do it when we use gRPC with streams?

The answer is “Yes”. To achieve this, we need to have grpcurl, jq, tr, base64. If the server is served on localhost:8000 you can get your “gopher” using this command:

grpcurl -plaintext -d '{"id":"gopher"}' localhost:8000 api.v1.FileService.Download  | jq .chunk | tr -d '"\n' | base64 -d > gopher.png

This command will work correctly only if each chunk has a bytes count multiple by 3. If the chunk size is not multiple by 3 we will get an error during decoding from string to bytes using base64.

Final Thoughts

In this article, I covered the topic of how to send “File” (see the struct at the beginning of the article) from server to client using Golang/gRPC. All code you can find in my GitHub repository.