Sometimes we need to download files from some storage. Files have some metadata like name, size, and extension (maybe we need more metadata, but let’s keep it simple), also files should implement interface for getting bytes of this content. io.Reader So file struct may look like this: package file

import (
   "io"
)

func NewFile(name string, extension string, size int, r io.Reader) *File {
   return &File{Name: name, Extension: extension, Size: size, r: r}
}

type File struct {
   r         io.Reader
   Name      string
   Extension string
   Size      int
}

func (f *File) Read(p []byte) (n int, err error) {
   return f.r.Read(p)
} I want to implement sending this struct from the server to the client using Golang/gRPC. gRPC API Before beginning to implement the server and the client we need to have API. Our API is quite simple: syntax = 'proto3';

package api.v1;

option go_package = "github.com/andrey-berenda/go-filestorage/generated/v1;api";

service FileService {
  rpc Download(DownloadRequest) returns(stream DownloadResponse);
}

message DownloadRequest{
  string id = 1;
}

message DownloadResponse{
  bytes chunk = 1;
} As we can see in our API there is no file metadata: metadata will be in headers because we need to send it only once and headers are the best candidate for achieving this goal. Because of this we need to have method and constructor from metadata in our struct: Metadata() package file

import (
   "io"
   "strconv"

   "google.golang.org/grpc/metadata"
)

var fileNameHeader = "file-name"
var fileTypeHeader = "file-type"
var fileSizeHeader = "file-size"

func (f *File) Metadata() metadata.MD {
   return metadata.New(map[string]string{
      fileNameHeader: f.Name,
      fileTypeHeader: f.Extension,
      fileSizeHeader: strconv.Itoa(f.Size),
   })
}

func NewFromMetadata(md metadata.MD, r io.Reader) *File {
   var name, fileType string
   var size int

   if names := md.Get(fileNameHeader); len(names) > 0 {
      name = names[0]
   }
   if types := md.Get(fileTypeHeader); len(types) > 0 {
      fileType = types[0]
   }
   if sizes := md.Get(fileSizeHeader); len(sizes) > 0 {
      size, _ = strconv.Atoi(sizes[0])
   }

   return &File{Name: name, Extension: fileType, Size: size, r: r}
} Server implementation The interface for the server looks like this: type FileServiceServer interface {
   Download(req *pb.DownloadRequest, server pb.FileService_DownloadServer) error
} At first, we need to check if the client sent the file id: if req.GetId() == "" {
   return status.Error(codes.InvalidArgument, "id is required")
} Then we get files from some storage: f, ok := getFile(req.Id)
if !ok {
   return status.Error(codes.NotFound, "file is not found")
} In my example, the storage is very simple and keeps everything in memory(only one file named “gopher”): //go:embed static/gopher.png
var gopher []byte

func getFile(fileID string) (*file.File, bool) {
   if fileID != "gopher" {
      return nil, false
   }
   return file.NewFile("gopher", "png", len(gopher), bytes.NewReader(gopher)), true
} After getting the file we send the headers: err := server.SendHeader(f.Metadata())
if err != nil {
   return status.Error(codes.Internal, "error during sending header")
} And send content: const chunkSize = 1024 * 3
   chunk := &pb.DownloadResponse{Chunk: make([]byte, chunkSize)}
   var n int Loop:
   for {
      n, err = f.Read(chunk.Chunk)
      switch err {
      case nil:
      case io.EOF:
         break Loop
      default:
         return status.Errorf(codes.Internal, "io.ReadAll: %v", err)
      }
      chunk.Chunk = chunk.Chunk[:n]
      serverErr := server.Send(chunk)
      if serverErr != nil {
         return status.Errorf(codes.Internal, "server.Send: %v", serverErr)
      }
   }
   return nil The server is implemented. The chunk size is not so big, I think we can increase it to 1MB or something like that, but keep the chunk size as a multiple of 3 (I will explain later why it is important). All code of the server is here : Client implementation I want to implement the method for the Client: type Client struct {
   client pb.FileServiceClient
}

func (c Client) GetFile(ctx context.Context, fileID string)(*file.File, error) For reaching this we need to make a gRPC request to the server: response, err := c.client.Download(
   ctx,
   &pb.DownloadRequest{Id: fileID},
)
if err != nil {
   return nil, fmt.Errorf("client.LoadFile: %w", err)
} After that we need to get headers from the context: md, err := response.Header()
if err != nil {
   return nil, fmt.Errorf("response.Header: %w", err)
} And finally, create and copy all content from response to io.Pipe : *io.PipeWriter r, w := io.Pipe()
f := file.NewFromMetadata(md, r)
go copyFromResponse(w, response)
return f, nil Implementation of the copying is here: func copyFromResponse(w *io.PipeWriter, res pb.FileService_DownloadClient) {
   message := new(pb.DownloadResponse)
   var err error
   for {
      err = res.RecvMsg(message)
      if err == io.EOF {
         _ = w.Close()
         break
      }
      if err != nil {
         _ = w.CloseWithError(err)
         break
      }
      if len(message.GetChunk()) > 0 {
         _, err = w.Write(message.Chunk)
         if err != nil {
            _ = res.CloseSend()
            break
         }
      }
      message.Chunk = message.Chunk[:0]
   }
} The client is implemented. And here is all code of the client . Bonus When you send files using simple HTTP there is one advantage: you can get files by id using only bash without writing any code in Golang. Is there a way to do it when we use gRPC with streams? The answer is “Yes”. To achieve this, we need to have , , , . If the server is served on you can get your “gopher” using this command: grpcurl jq tr base64 localhost:8000 grpcurl -plaintext -d '{"id":"gopher"}' localhost:8000 api.v1.FileService.Download  | jq .chunk | tr -d '"\n' | base64 -d > gopher.png This command will work correctly only if each chunk has a bytes count multiple by 3. If the chunk size is not multiple by 3 we will get an error during decoding from string to bytes using . base64 Final Thoughts In this article, I covered the topic of how to send “File” (see the struct at the beginning of the article) from server to client using Golang/gRPC. All code you can find in my . GitHub repository

Google

Read My Stories

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

How to Download Files Using Golang and gRPC

Too Long; Didn't Read

Company Mentioned

Andrei Berenda

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

How to Download Files Using Golang and gRPC

Too Long; Didn't Read

Company Mentioned

Andrei Berenda

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES