Leveraging Multithreading To Read Large Files Faster In Goby@mohammed-gadiwala
7,783 reads

Leveraging Multithreading To Read Large Files Faster In Go

tldt arrow
Read on Terminal Reader
Read this story w/o Javascript

Too Long; Didn't Read

The trick is to not load the whole file into memory and keep processing each word as we keep on moving the pointer of the file. With this, we can easily process the whole. file with a minimal amount of memory resources. The solution is we keep multiple pointers at different parts of the. file and each thread. reads chunks of. the file concurrently. We are using channels to unify all the words read by various threads into a single dictionary. The time required to process the 1GB file was halved by doing it in a serial manner.

Companies Mentioned

Mention Thumbnail
Mention Thumbnail
featured image - Leveraging Multithreading To Read Large Files Faster In Go
Mohammed Gadiwala HackerNoon profile picture

@mohammed-gadiwala

Mohammed Gadiwala

Backend engineer | Golang


Receive Stories from @mohammed-gadiwala

react to story with heart

RELATED STORIES

L O A D I N G
. . . comments & more!