In my previous , we discussed the future of the sharing economy and the exciting innovations that could be central to shaping it. One of the key technologies mentioned was the Interplanetary File System (IPFS). It’s a peer-to-peer (p2p) filesharing system that aims to fundamentally change the way information is distributed across & beyond the globe. IPFS consists of several innovations in communication protocols and distributed systems that have been combined to produce a file system like no other. So to understand the full breadth and depth of what IPFS is trying to achieve, it’s important to understand the tech breakthroughs that make it possible. post Communication Protocols & Distributed Systems For two people to exchange information, they need common sets of rules that define how & when information is transmitted. These rules are broadly known as communication protocols, but that’s quite a mouthful so we simply call it language. If you’ve ever been to a foreign country where you don’t speak the native tongue, you’ve probably experience a failure (or lack) of communication protocols. This was the case with computers; they were unable to communicate with each other and existed as isolated computational devices until the early 1980’s when the first communication protocols for computing were invented. “protocols are to communication what programming languages are to computations” In computers, communication protocols usually exist in bundles (called a protocol suite) of several layers. For example, the suite consists of 4 layers, each of which is responsible for specific functions. In addition to communication protocols, an important relationship to understand is the basic structure of the interconnections between the computers. This is known as the . Several exist, but the two types relevant to us are and . Internet protocol system architecture client-server peer-to-peer networks The internet is dominated by client-server relationships, which rely on the Internet Protocol suite. Of these, Hypertext Transfer Protocol ( ) is the basis for communication. HTTP Data is stored in centralized servers and accessed by location-based addressing. This makes it easier to distribute, manage, secure the data, and to scale the capacity of both servers and clients. However there are many weaknesses in the realms of security, privacy and efficiency: control of the server translates to control of the data. This means your data can be accessed, altered and removed by any party with control of the server; this could be an entity with legitimate authority over the server or a malicious hacker. In location-based addressing, data is identified by where it is located rather its content. This limitation means you have to go to all the way to specific location to access a piece of data, even if that same data is available somewhere closer. There is also no way of telling if the data has been altered, since the client only needs to know where it is, not what it is. But the client-server model and HTTP have served the internet pretty reliably for most of its history. This is because the HTTP web is highly effective for moving around small files like text and images. In the first two decades of the web, the size of the average web page has only increased from ~2 kilobytes to ~2 megabytes. Source HTTP is great for loading websites but it wasn’t designed for the transfer of large amounts data (like audio and video files). These constraints possibly enabled the emergence and mainstream success of alternative filesharing systems like Napster (music) and BitTorrent (movies and pretty much anything). Fast forward to 2018, where on-demand HD video streaming and big data are becoming ubiquitous; we are continuing the upward march of producing/consuming more and more data, along with developing more and more powerful computers to process them. Major advancements in cloud computing have helped sustain this transition, however the fundamental infrastructure for distributing all this data has remained largely the same. The InterPlanetary File System IPFS attempts to address the deficiencies of the client-server model and HTTP web through a novel p2p file sharing system. This system is a synthesis of several new and existing innovations. IPFS is an open-source project created by , an R&D lab for network protocols and former Y Combinator startup. Protocol Labs also develops complementary systems like and , which will be explained below. Hundreds of developers around the world contributed to the development of IPFS, so its orchestration has been a massive undertaking. Here are the main components: Protocol Labs IPLD Filecoin Distributed Hash Tables A is a data structure that stores information as key/value pairs. In distributed hash tables (DHT) the data is spread across a network of computers, and efficiently coordinated to enable efficient access and lookup between nodes. hash table The main advantages of DHTs are in decentralization, fault tolerance and scalability. Nodes do not require central coordination, the system can function reliably even when nodes fail or leave the network, and DHTs can scale to accommodate millions of nodes. Together these features result in a system that is generally more resilient than client-server structures. Block Exchanges The popular file sharing system Bittorrent is able to successfully coordinate the transfer of data between millions of nodes by relying on an innovative data exchange protocol, however it is limited to the torrent ecosystem. IPFS implements a generalized version of this protocol called BitSwap, which operates as a marketplace for any type of data. This marketplace is the basis for : a p2p storage marketplace built on IPFS. Filecoin Merkle DAG A merkle DAG is a blend of a and a Directed Acyclic Graph ( ). Merkle trees ensure that data blocks exchanged on p2p networks are correct, undamaged and unaltered. This verification is done by organizing data blocks using functions. This is simply a function that takes an input and calculates a unique alphanumeric string (hash) corresponding with that input. It is easy to check that an input will result in a given hash, but incredibly difficult to guess the input from a hash. Merkle Tree DAG cryptographic hash Don’t confuse these two. Source: http://www.merkletrees.org/ The individual blocks of data are called ‘leaf nodes’, which are hashed to form ‘non-leaf nodes’. These non leaf nodes can then be combined and hashed, until all the data blocks can be represent by a single root hash. Here’s an easier way to conceptualize it: A DAG is a way to model topological sequences of information that have no cycles. A simple example of a DAG is a family tree. A merkle DAG is basically a data structure where hashes are used to reference data blocks and objects in a DAG. This creates several useful features: all content on IPFS can be uniquely identified, since each data block has a unique hash. Plus the data is tamper-resistant because to alter it would change the hash, as shown below: Source: ethereum.org The central tenet of IPFS is modeling all data on a generalized merkle DAG. The significance of this security feature is hard to overstate. For comparison with client-server systems, see how this guy . hacked 40 websites in 7 minutes Version Control Systems Another powerful feature of the Merkle DAG structure is that it allows you to build a distributed version control system ( ). The most popular example of this is Github, which allows developers to easily collaborate on projects simultaneously. Files on Github are stored and versioned using a merkle DAG. It allows users to independently duplicate and edit multiple versions of a file, store these versions and later merge edits with the original file. VCS IPFS uses a similar model for data objects: as long as objects corresponding to the original data, and any new versions are accessible, the entire file history can be retrieved. Given that data blocks are stored locally across the network and can be cached indefinitely, this means that IPFS objects can be stored permanently. Additionally, IPFS does not rely on access to Internet protocols. Data can be distributed in , which are simply networks built on another network. These features are notable, because they are core elements in a censorship-resistant web. It could be a useful tool in promoting free speech to counter the prevalence of around the world, but we should also be cognizant of the potential for abuse by bad actors. overlay networks internet censorship Self-certifying File System The last essential component of IPFS we’ll cover is the Self-certifying File System ( ). It is a distributed file system that doesn’t require special permissions for data exchange. It is “self-certifying” because data served to a client is authenticated by the file name (which is signed by the server). The result? You can securely access remote content with the transparency of local storage. SFS IPFS builds on this concept to create the InterPlanetary Name Space (IPNS). It is an SFS that uses to self-certify objects published by users of the network. We mentioned earlier that all objects on IPFS can be uniquely identified, but this also extends to nodes. Each node on the network has a set of public keys, private keys and a node ID which is the hash of its public key. Nodes can therefore use their private keys to ‘sign’ any data objects they publish, and the authenticity of this data can be verified using the sender’s public key. public-key cryptography Here’s a quick recap of the key IPFS components: With the Distributed Hash Table, nodes can store & share data without central coordination IPNS allows exchanged data to be instantly pre-authenticated and verified using public key cryptography. The Merkle DAG enables uniquely identified, tamper-resistant and permanently stored data You can access past versions of edited data via the Version Control System Simple conceptual framework So why is all this important? IPFS provides high throughput, low latency, data distribution. It is also decentralized and secure. This opens up several interesting and exciting use cases. It can be used to deliver content to websites, globally store files with automatic versioning & backups, facilitate secure filesharing and encrypted communication. Below are a few interesting projects being built on IPFS: , Akasha a Next-Generation social network , Balance3 a triple-entry accounting platform , BlockFreight an open network for global freight , Digix a platform for tokenizing physical gold , Infura an infrastructure provider for DApps , Livepeer a decentralized live-video streaming platform , Origin a peer-to-peer marketplace for the sharing economy , uPort a self-sovereign identity system The variety of these applications demonstrate the versatility of IPFS for several different use cases. It is also being used as a complementary file system for public blockchains and other p2p applications. At the time of writing it can cost several dollars to store a kilobyte of data in an Ethereum smart contract. This is a major constraint and there is currently massive growth in new decentralized apps (DApps) being launched. IPFS is interoperable with smart contracts and data, so it can add reliable, low-cost storage capacity to the ethereum ecosystem. The attempt to make Ethereum blockchain data natively accessible on IPFS is a separate protocol known as (InterPlanetary Linked Data). blockchain IPLD Challenges Despite the impressive performance of IPFS, a few issues are yet to be fully resolved. Firstly, the content addressing on IPNS is currently not very user-friendly. Your typical IPNS link looks like this: ipfs.io/ipns/QmeQe5FTgMs8PNspzTQ3LRz1iMhdq9K34TQnsCP2jqt8wV/ These links can be shortened to simpler names using a Domain Name System (DNS), but this introduces an external point-of-failure for content distribution. The content can however be still accessible through the original IPNS address. Some users also report that IPNS can be slow at resolving domain names, with delays of up to a few seconds. It’s not clear exactly what the root of this issue is. : Update On 03/26/2018, IPNS released an upgrade with an experimental feature to speed up publishing/resolving. Details available here On IPFS there is also little incentive for nodes to maintain long term backups of data on the network. Nodes can choose to clear cached data to save space, meaning theoretically files can end up ‘disappearing’ over time if there are no remaining nodes hosting the data. At current adoption levels this isn’t a significant issue but in the long term, backing up large amounts of data requires strong economic incentives. Storage Markets Filecoin is a separate protocol designed to add economic incentives to file storage on IPFS, and foster a distributed storage market that rivals enterprise cloud storage (like Amazon S3, etc). Instead of centralized infrastructure with fixed pricing, IPFS + FileCoin offers storage on a global network of local providers who have the freedom to set prices based on supply and demand. Instead of a consensus algorithm like , Filecoin uses to ensure security and reliability. So anyone can join the network, offer unused hard drive space on their computing device, and get rewarded in Filecoin tokens for data storage and retrieval services. Proof-of-Work Bitcoin Proof-of-Storage The network is being developed on Ethereum, so smart contract integration could produce advanced features like escrow, insurance, etc in the storage marketplace. In theory this economic model should develop a highly competitive free market with potentially lower costs than large-scale providers. But FileCoin has not been launched yet, so it will be interesting to observe how these concepts play out in reality. Getting Started IPFS is a highly ambitious endeavor, and obviously the precise mechanics of how the system functions are far more complex than what has been described in this guide. We’ll leave those details for the cryptographers and computer scientists to enjoy. You don’t have to be an expert to IPFS, so if any of the advantages or use cases seem useful or appealing to you, download IPFS & get started . If you’ve got gigabytes or terabytes of unused storage and would like to make good use of that idling capacity, you can to be an early Filecoin miner when the network launches. You can also if you’re interested in being an early storage user. use here sign up sign up Using IPFS is quite remarkable and understanding the technical wizardry that makes it possible is even more exciting. If successful, IPFS and its complementary protocols could provide resilient infrastructure for the next generation of the web. The web that was promised to be distributed, secure, & transparent. Sources Comer, Douglas. . Pearson, 2015. Computer networks and internets goo.gl/AbM8st IPFS whitepaper (draft 3) http://goo.gl/zwKD9J Filecoin whitepaper https://goo.gl/RrxcvX
