A thread about Internet Archive's "Silent Killer" and why you should both donate to @internetarchive this month (archive.org/donate) and encourage others to do so. (Photos by Jamie Lyons)

The Archive has been saving copies of websites and hosting many different forms of media for a couple decades now. As part of what people think as The Wayback Machine (archive.org/web), crawlers and partner crawlers have been doing scans of the web for most of that time.

I started with Archive Team in 2009 and with Internet Archive in 2011. When I joined up with them, I brought along with me a host of activists and an urging to leverage the Wayback to make at least a snapshot of rapidly disappearing early websites that were getting shut down.

To be sure, Brewster and the gang were already deeply into website scanning to allow for historical record but when I joined, the entire site was roughly 5 petabytes (5,000 terabytes), built on 2 terabyte drives, mirrored. And I started uploading to it. Very intensely.

It usually takes the nimble team within the Internet Archive just a few iterations before they internalize the direction things are going and start moving in that direction. So deep crawls, along with Archive Team's specific crawl projects, became the norm.

I joined in 2011, and it's now 2019. The whole site was 5 petabytes when I joined. IT IS OVER SIXTY PETABYTES NOW. Just this year, we will add at least ten more petabytes of data. So in 2019, we will add two times as much data as we'd added from 1996-2011.

We've switched from 2 terabyte drives to 8 to 16 currently. This means we can fit more data in smaller spaces. These two server racks have 10.5 raw petabytes of data in them. Naturally, they're mirrored as well.

Every once in a while someone looks at our setups and goes "what about (cloud company, google drive, amazon glacier, this guy I know)" and there's a host of reasons we do things the way we do, but there's a bunch of factors like "terms of service", "transfer costs are huge", etc.

As for what we're hosting with that 60 petabytes - it includes over 40 million public media items, with 15,000 new items being added on an average DAY, heading out to over one million unique users every day, with the majority being Wayback Machine users. Not bad for a library.

Back to the disk usage. The Archive Team (as well as the generalized crawling by the Wayback group) encourages incredibly deep and complete scans for services that are going under, or who have gone under, and which we'll be feeling the loss for years to come.

What follows are some of our "Greatest Hits" and the disk space we're talking about. Once we use space for one of these projects, we keep the data around, so you, and researchers, and journalists, and activists, and families, can get to these places at any time, within seconds.

Xanga was/is a blogging site that announced they were converting to "2.0" and the websites that had been around for 15 years were going to go through a little rough patch, i.e. stored and inaccessible, unless the user took action. We backed it up. 10 terabytes.

Dogster and Catster were social media sites to allow folks to share stories and photos of their pets. A shutdown announcement put these at risk. We worked with the founder, @tedr (RIP), who'd left the company, to save as much as we could. 3.5 Terabytes.

There have been a number of video sharing sites that have gone under as the YouTube juggernaut won the battle. One of them was BLIP.TV, which had many unique videos that are otherwise lost. Archive Team backed a bunch of it up. 227 terabytes.

There's another sharing site called STAGE6 that was also taken down - it was hosting all sorts of videos in "DIVX" format. Ask your GenX relative about it - it's gone completely now. We grabbed it. 594 gigabytes. (1/2 terabyte).

Webshots was a photo sharing site from 1999 to 2012, when it was sold and converted to "Smile!", and in doing so, they deleted 13 years of user photos and galleries. We backed it up. 117 terabytes.

Now, let's talk about real disk space usage. When Google announced they were shutting down Google Plus, taking the lamented social service down, killing untold millions of posts, lists, archives and information, Internet Archive hosted the efforts to back it up. 1.1 Petabytes.

Tumblr's announced, fuzzy removal of "adult" blogs, with no clear indication of which these meant, meant a speculative grab was about the best we could do. It was still 92 terabytes of disk space, still spinning, still available instantly.

This is to say nothing of much smaller sites we've been grabbing when they announce they're coming down. A collection of UMICH student web pages from the 1990s. Verizon killed user pages from the last 20 years. WEBLOG-NL, IMDB Forums. G4TV Forums. Fileplanet.

In all, these many different projects are filling disk space at an enormous rate, petabytes, to become, in most cases, the only remaining location for the user data, historical record, cultural touchstone these websites have been, long after the companies that made them are dust.

If you watch to check some of what we have done, the collection of Archive Team grabs (the core WARC files the Wayback uses to pull out sites on request) is available here: archive.org/details/archiv… - and that's just Archive Team, not counting the hundreds of TB from crawls.

What I'm saying, then, is please recognize the enormous financial burden this has put on the organization, attempting to preserve a web that started out pretty quiet but has become a deafening roar of data, every growing and needing to be accessible 24/7, near-instantly.

Could we mitigate this cost? Sure, we could scale back, not do certain types of websites, halt crawls, end some of these all-consuming saving projects. But based on the letters, references and news articles we've seen, that would not be a good idea for the world at large.

If you were looking for an excuse to donate to the archive, which is a tax free donation here in the US, and with many different options available to you (archive.org/donate), then I'm going to say: Think of the disk space, our Silent Killer. Thanks.

Piece of cake.

Internet Archive's "Silent Killer"

Too Long; Didn't Read

@techtweeter

RELATED STORIES