Data Needs to Lose Weight

To start, the video above isn’t mine. I do not own it. All rights belong to HBO and everyone who helps make the fantastic show that is Silicon Valley. In the unlikely event HBO wants me to pull it, I will gladly do so. In the meantime, go watch Silicon Valley. It’s great.

This is going to be a slightly ‘techie’ post, but I’ll try to make it interesting for everyone.

Data and information is everywhere. Everything we do generates data, and with the rise of smart technologies in the lifestyle/fitness markets, this is only going to become more and more true. Everything from how we slept and if we woke up correctly, to what we ate for lunch and who we were with when we ate it, will be tracked. There is so much information that it is becoming difficult to track what is important and what is superfluous. We create data points and track things that don’t need to be tracked, simply because we can.

And this is just text and numerical data. With so much data, information, and media being created, we need to store everything in smaller and more accessible places.

A few nights ago while getting my zip-lining footage from my GoPro to my computer, I was faced with the tradeoff between storage/speed/ease of transfer, and quality. I could export my videos in 720p and use a still astounding 500Mb for every 5 minutes of video, or I could export in 4k and use over 1Gb for every clip. With the goal of sending this video to family, I was forced to choose 720p. In addition to this, I wanted to send over close to 100 pictures from Hawaii (I started with over 1000). As RAW files straight from the camera, each file was about 40Mbs. Even when edited and dropped down to the JPEG format, they were still 4Mbs each. That means for a 17 minute video and 100 pictures, I would have needed upwards of 3Gb of media space and transfer ability. With downgraded quality. Had I done the whole video in 4k and sent over larger pictures, I would have needed over 5Gbs.

Gmail now finally accepts up to 10Gbs of data per email, provided you send the media as a Google Drive link rather than attachments. Therefore technically, I could have sent everything over at the highest quality possible. But, the speed of transfer would have been ridiculously slow and in general, 5Gbs for a short video and a few pictures is unacceptable.

The video clip above is from HBO’s amazing show, Silicon Valley. The main character creates a lossless compression algorithm that can compress media to incredibly small file sizes. This means that media can be streamed or sent rapidly, and huge amounts of data and media can be stored with ease. Sadly, that algorithm does not exist. As of 2014, one of the best compression tools, 7z/7-Zip could only compress files to 46% of their original size. That means at best, my 3Gb need would be compressed to roughly 1Gb. Still way too big for traditional email, and not sustainable. If I take 100 pictures every weekend and produce a 7 minute YouTube video every week, I would need upwards of 400Gb of space every year. And that’s if I do everything perfectly. My 17 minute GoPro video was cut down from 40 minutes of video. I could feasibly burn through a Terabyte of data every year.

This is just me. I alone, may create over a Terabyte of data this year. According to IBM, we collectively create2.5 quintillion bytes of data every DAY. That’s 2,500,000,000,000,000,000 bytes every day. It’s pretty clear that we need to get better at storing and managing data. I struggle with my few gigabytes every week. How in the world are we going to manage quintillions of bytes of data every day, a number which is growing?

Compression. Someone needs to get on that, and fast. If data can be made to lose weight; made to fit into smaller places and move around more quickly, things will get easier for all of us. Imagine having the entire internet on a flash drive. It’s not that we haven’t made leaps in this field. I have flash drives that store a few hundred megabytes. These were made just a few years ago. Now, we can go out and get a flash drive that stores a few terabytes. But what’s next? Compression can make the 250Mb flash drive the same as the 2Tb flash drive, and the 2Tb flash drive an infinite storage pool. Massive hard drives are everywhere, but without better compression techniques, we will burn through our hard drives anyway.

Someone go make Pied Piper.

Originally published at www.indranilmitra.com.

Hacker Noon is how hackers start their afternoons. We’re a part of the @AMIfamily. We are now accepting submissions and happy to discuss advertising &sponsorship opportunities.

To learn more, read our about page, like/message us on Facebook, or simply, tweet/DM @HackerNoon.

If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!