Ten terrible attempts to make the Inter Planetary File System human-friendly

This article is part 6 of the Blockchain train journal, start reading here: Catching the Blockchain Train . IPFS human-friendly naming; is it possible? Did you see the traffic-magnet-title I came up with? It is a bit over the top; we’re just looking at how the addressing of content can be made a bit easier than . ipfs cat /ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV As mentioned in the previous post, adding some mutable “label” to point to (immutable) IPFS content is sometimes desired. An example is the content of this blog: the original address (hash) for the home page is different from the current (ah no, already expired!) home page . /ipfs/QmcPx9ZQboyHw8T7Afe4DbWFcJYocef5Pe4H3u7eK1osnQ/ /ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/ Mutable links are essential, but not enough for us humans. They are hard to read, let alone remember. What we’ll do in this episode is to explore different ways to: resolve this problem of adding mutable links, and to make the links more human-friendly, while trying to stay decentralized where possible. We’ll see the following methods in action: The IPFS gateway IPNS DNS TXT IPNS Records Browser extensions Namecoin Blockstack ID Ethereum Name Service Proquint Pronounceable Identifiers Name shorteners Filecoin To make IPFS more human-friendly - in the context of a decentralized IPFS powered website - there is currently no other way than sacrificing a part of decentralization using HTTP-to-IPFS gateways and old school DNS. Naming systems in the blockchain sphere (Blockstack, Namecoin, EthNames) are possible candidates to improve the UX while keeping it 100% decentralized, but at this point of time none of these work out of the box. Filecoin might be the way to go, but that is still far away. TL;DR The IPFS gateway The most obvious step to make content available on the old skool internet with a regular browser is to make use of the IPFS gateway. These gateways speak HTTP and access content through their IPFS node. Any gateway will do, but the gateway where content is pinned is of course faster. Examples of accessing the previous (before this post) home page of this blog: http://decentralized.blog:8080/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/ https://gateway.ipfs.io/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/ (if you have a local IPFS node running) http://localhost:8080/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV/ So while that works, it has a couple of problems: when the content of the home page changes these URL’s still point to old content the hash is not very human-friendly (a.k.a. ugly) it introduced an element of centralization (sort of, there is no dependence on a particular single gateway) IPNS To solve the first of these three problems we can, of course, use IPNS. Any time the content changes we publish the latest content hash like: ipfs name publish QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeVPublished to QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN: /ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV Now we can access the home page through any gateway but with the mutable IPNS link: http://decentralized.blog:8080/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN/ Much better, but still ugly… Note: IPNS is still a bit shaky and forgets published names after about 12 hours. You might want to run a cron job to republish every 8 hours or so. DNS TXT IPNS Records Now, let’s get rid of the hash. The white paper mentions the use of a DNS TXT record. This is what Wikipedia has to say about it: A TXT record (short for text record) is a type of resource record in the Domain Name System (DNS) used to provide the ability to associate some arbitrary and unformatted text with a host or other name, such as human readable information about a server, network, data center, and other accounting information. So the owner of a domain can associate any data to this domain by adding a TXT record. It is like a key-value store where only the owner of the domain can write, and the world can read. This is used by IPNS to look up the path if you insert a valid domain as the path. /ipns/ This is how IPNS normally resolves: # Publishipfs name publish QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeVPublished to QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN: /ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV # Normally IPFS resolves the peerID hash$ ipfs name resolve /ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV To be able to resolve a domain we need to add a to the TXT record for that domain. dnslink After doing so for the domain name of this blog (decentralized.blog) pointing to the link /ipfs $ dig txt decentralized.blog..;; ANSWER SECTION:decentralized.blog. 900 IN TXT "dnslink=/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV" we can resolve by domain name $ ipfs name resolve decentralized.blog/ipfs/QmcJTRZuGVdqUoNS1414G2mim3mt39RU14JsTmbh4KJYeV which allows us to use a much nicer address: /ipns/decentralized.blog This works, but every time I change the content, I have to change the DNS TXT record and wait for propagation. The cool thing is that I can make the DNS TXT record point to a link, and simply do a when content has changed: /ipns ipfs name publish $ dig TXT decentralized.blog..;; ANSWER SECTION:decentralized.blog. 900 IN TXT "dnslink=/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN" # points to /ipns path now Note, I should have done this in the first place, but I was led on the wrong track by this example in the white paper: # this DNS TXT record ipfs.benet.ai. TXT “ipfs=XLF2ipQ4jD3U …” # behaves as symlink ln -s /ipns/XLF2ipQ4jD3U /ipns/fs.benet.ai That is probably a typo, although the ipfs.io domain also points to an immutable path for some reason: /ipfs $ dig TXT ipfs.io..;; ANSWER SECTION:ipfs.io. 60 IN TXT "dnslink=/ipfs/QmPCawMTd7csXKf7QVr2B1QRDZxdPeWxtE4EpkDRYtJWty" More about how IPFS resolves paths . here So now we have a human-friendly URL but at the cost of adding more centralization (the DNS). Browser extensions I was hoping there was a browser extension that allowed me to browse a web of and paths, but the extensions that do exist are not made for that purpose. /ipfs /ipns Because it is still interesting, I’ll mention it anyway. The most up to date extension is . There are browser extensions available for and . ipfs-companion Firefox Chrome One of the cool things it does is helping to distribute data by redirecting requests to the public gateway to your local node. Another exciting project is the that used to support IPFS, but not anymore. It is a browser for distributed content based on . Beaker browser Dat Namecoin What is Namecoin? is a very interesting protocol based on blockchain technology. I won’t go into it here in much depth, because I want to start exploring the mother of all blockchains first (i.e., Bitcoin, but you knew that). Namecoin I just realized, this is episode 6 of , and we haven’t seen a single blockchain yet! And worst, we are not going into that here either… Please bear with me, and we will get there eventually. Catching the Blockchain Train So, Namecoin. This is what they say about themselves on the website: Namecoin is an experimental open-source technology which improves decentralization, security, censorship resistance, privacy, and speed of certain components of the Internet infrastructure such as DNS and identities. (For the technically minded, Namecoin is a key/value pair registration and transfer system based on the Bitcoin technology.) Bitcoin frees money — Namecoin frees DNS, identities, and other technologies. The part about DNS might be able to help us here. Namecoin facilitates the TLD, which has its own . .bit portal Getting a .bit domain To get started we need to get hold of a .bit name. I rented one a while ago at an exchange, but I now that there is a better way: using a Namecoin client. If you go the exchange way, realize that you don’t have the private keys to control the domain (change value in the KV pair), so you don’t own it. see To get your own .bit domain name, follow the steps at . Register and Configure .bit Domains You’ll see that you need a bit of namecoin (NMC) to get your .bit domain, but mining is not really an option (it is done alongside Bitcoin mining and hard), but you can also buy some. Here is a list of exchanges where you can buy NMC, , but I’d recommend if you already own some bitcoin or altcoin. You need 0.02 NMC which is about . You can use this to see if your desired name is still available. Namecoin Markets ShapeShift nothing namecoin explorer How a .bit lookup works .bit domains have a unix-path equivalent like this: . These paths are the key of a record stored in the Namecoin's blockchain. So to see what the value is you have to have access to a Namecoin node, and ask for the value of the record with key . d/somedotbitname d/somedotbitname That value can be anything, but in practice, it is either a reference to nameservers (standard DNS) or directly to an IP address, or to a address. Others add their email and say the .bit address is for sale. Bitmessage We need access to a namecoin node to query it, but there are online REST API’s (introducing a central point of failure) we can use, like : webbtc # Bitmessage address value$ curl {"name": "d/mysite","value": "BM-2cUyrUNq91XqdPSKvcHSytuED9nnTazf5r","txid": "78e212c47b41b3ae80413a7064c7bee044f8f2bc27362e9c37d5474b68bfd9e3","address": "NFE3ED8C3BGRXuzjBRD3YGJV6jrmbS5h3Q","expires_in": 23292} http://namecoin.webbtc.com/name/d/mysite.json # IP address value$ curl {"name": "d/dappersoftware","value": "{\"ip\":\"207.111.216.146\",\"map\":{\"*\":{\"ip\":\"207.111.216.146\"}}}","txid": "da72c6b0cf85c94d7b2d4f2f8534d1cdafa47ca94fd00489bf205f00c3fc1cb2","address": "N2hrunEcP3PNxdKvihMKCoEnvLUcCcBhKw","expires_in": 4344} http://namecoin.webbtc.com/name/d/dappersoftware.json How browsing .bit websites works Obviously the normal DNS system doesn’t know what to do with a domain, so in a browser that won't resolve to anything. The Namecoin team provides a layer on top op the namecoin node to be able to browse domains: . I'll leave it at that since we are diverging from our goal... .bit .bit ncdns How do we resolve a .bit domain to an IPFS hash? What are we trying to do again? So much exciting stuff is going on that we almost got lost in the shapeshiftbitmessagewebbtcncdnschain. Focus! The goal of this exploration is to be able to pass around a human-friendly address like to bring up an IPFS-powered website. That would be possible if the IPNS system knew how to read the value of in the Namecoin blockchain and interpret it and resolve it to an IPFS address. dweb.bit dweb.bit It’s just like we saw with the DNS TXT record trick, but now the IPFS node would need to have access to the Namecoin blockchain (or a centralized public API like webbtc we saw before). So: needs to resolve to . The value field of a .bit record can be anything, so that is easy to solve. The other part would be that the IPNS resolver implementation needs to recognize .bit domains and access the Namecoin network somehow. This was not implemented as can be easily shown: /ipns/dweb.bit /ipfs/ $ ipfs name resolve dweb.bitError: Could not resolve name. The IPFS-Namecoin integration is discussed . Personally, I would start by providing a central .bit lookup service on ipfs.io, and extend that later with a Namecoin resolver implementation in the ipfs code. It doesn’t look like it has priority now in the IPFS project though. here In any case, I’m ready: points to . dweb.bit { "ipns": "/ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN" } Blockstack ID Another contestant for the naming layer that is mentioned in Juan Benet’s presentations is Blockstack. What is Blockstack? This is what Wikipedia has to say about it: Blockstack is the first implementation of a decentralized DNS system on top of the Bitcoin blockchain. It combines DNS functionality with public key infrastructure and is primarily meant to be used by new blockchain applications. That sounds very similar to what Namecoin does, but it uses the Bitcoin blockchain instead of its own fork. In the past, the Namecoin blockchain, but the idea is that it should be . it used blockchain agnostic On the homepage of Blockstack a bigger goal is stated: A New Internet for Decentralized Apps Blockstack is a new decentralized internet where users own their data and apps run locally. A browser portal is all that’s needed to get started. But for now, let’s limit our focus to the decentralized DNS system. How to register a Blockstack ID The naming system is called Blockstack ID, and the TLD is . You can get your own .id at or use the terminal as described here: . .id onename Blockstack CLI Let’s try that: # First install blockstack with pip (Python2 only, booh!) # Create a wallet; not mentioned in the documentation, but required to do first$ blockstack setup # Check price for the desired name, the shorter the name, the more expensive$ blockstack price dweb.id{"name_price": {"btc": 0.016,"satoshis": 1600000},"preorder_tx_fee": {"btc": 0.00219904,"satoshis": 219904},"register_tx_fee": {"btc": 0.0020583,"satoshis": 205830},"total_estimated_cost": {"btc": 0.02213352,"satoshis": 2213352},"total_tx_fees": 613352,"update_tx_fee": {"btc": 0.00187618,"satoshis": 187618}} # Get a bitcoin address to pay the fee to$ blockstack deposit{"address": "3BmAHjCLgELuRdg3jM1MMgm5twXPWWxr7s","message": "Send bitcoins to the address specified."} # Start the API server; also not documented, but the error on register is helpful$ blockstack api start{"status": true} # Now register a name$ blockstack register dweb.idCalculating total registration costs for dweb.id...Registering dweb.id will cost about 0.02213352 BTC.Use `blockstack price dweb.id` for a cost breakdown The entire process takes 48 confirmations, or about 5 hours.You need to have Internet access during this time period, sothis program can send the right transactions at the righttimes. Continue? (y/N): y{"message": "Name queued for registration. The process takes several hours. You can check the status with `blockstack info`.","success": true,"transaction_hash": "c4fc237f4b13d9925f8180fbe9e131cd0a1ee54d9b8c7455c4b2e2464fbe1315"} # Check the status$ blockstack info{"cli_version": "0.14.4.2","consensus_hash": "47f015a3959a80e7ce9c5fb425833860","last_block_processed": 482064,"last_block_seen": 482070,"queues": {"preorder": [{"confirmations": 0,"name": "dweb.id","tx_hash": "c4fc237f4b13d9925f8180fbe9e131cd0a1ee54d9b8c7455c4b2e2464fbe1315"}]},"server_alive": true,"server_host": "node.blockstack.org","server_port": 6264,"server_version": "0.14.4.0"} This is a Bitcoin transaction which can be inspected here . c4fc237f4b13d9925f8180fbe9e131cd0a1ee54d9b8c7455c4b2e2464fbe1315 About 5 hours later the registration was completed: # Check if we own the dweb.id name now$ blockstack names{"addresses": [{"address": "3PR8Js7pLt5Bagk5MZHffXoR43EpQrJHxT","names_owned": ["pors.id","dweb.id"]}],"names_owned": ["error"]}# Hmm, sort of I guess :) All in all, it seems more like an identity thing, but the domains can also be used as a DNS record (BNS in this case). Blockstack also introduced namespaces, the bit after the dot (now id), which is obviously expensive. .id How a .id lookup works There is a command for lookups in the client: $ blockstack lookup timblee.id{"profile": {"@type": "Person","account": [{"@type": "Account","identifier": "timbl","proofType": "http","proofUrl": "https://gist.github.com/timbl/04e8ac7c81cd2dee2f51a5e8c672188d","service": "github"},{"@type": "Account","identifier": "timberners_lee","proofType": "http","proofUrl": "https://twitter.com/timberners_lee/status/740677355950080001","service": "twitter"}],"image": [{"@type": "ImageObject","contentUrl": "https://s3.amazonaws.com/97p/lUU.jpeg","name": "cover"}]},"zonefile": "$ORIGIN timblee.id\n$TTL 3600\n_http._tcp URI 10 1 \"https://blockstack.s3.amazonaws.com/timblee.id\"\n"} The zonefile bit is the interesting part in our use case; we can store routing information in there. Also, there is an API (it seems a bit shaky, but it works for some accounts): . For the API doesn’t work, but the does return data for this account. https://core.blockstack.org/v2/users/werner pors.id blockstack explorer So how can we use all this? How do we resolve a .id domain to an IPFS hash? As said, the zone file entry seems to be the best spot to store routing information. This can be done with the CLI client with the command. But to what value do we need to set it? That all depends on what the Blockstack and IPFS teams have agreed on. blockstack update A bit of Googling and digging in the Blockstack and IPFS forums didn’t give me any answers and my hope was set on this duo-presentation , but not a word about integration there (at least not the addressing bit). Muneeb Ali & Juan Benet: Blockstack IPFS “CTO Briefing” at CONSTRUCT 2017 There is some work about the integration between Blockstack and IPFS as a . But that’s not what we are looking for here. data storage layer So just as with Namecoin, there is no way to use Blockstack as the name layer for IPFS. But again, I want to be ready when they are so I updated my zonefile: $ echo '{"ipns":"ipns/QmRf4ERGvYpVo6HRa2VueZT8pWi8YvyLS3rW6ad2y83tdN"}' > new_zone_file.txt$ blockstack update dweb.id new_zone_file.txt{"message": "Name queued for update. The process takes ~1 hour. You can check the status with `blockstack info`.","success": true,"transaction_hash": "026c739761ec6ed60119b4d80da2ccf81f274d558f53f377d6862a6020d767b8","zonefile_hash": "4db474dd1dda7502c6152d248b7a24302f6e104a"} See the resulting Blockstackified zone file . here I will revisit Blockstack in depth in a future post, as soon as we got a better understanding of blockchain technology in general. Ethereum Name Service We start to see a trend here: very cool blockchain powered name systems, but no support in IPFS. Let’s see how ENS fares. What is ENS? From the documentation: ENS is the Ethereum Name Service, a distributed, open, and extensible naming system based on the Ethereum blockchain. ENS can be used to resolve a wide variety of resources. The initial standard for ENS defines resolution for Ethereum addresses, but the system is extensible by design, allowing more resource types to be resolved in future without the core components of ENS requiring upgrades. We haven’t looked at yet, but that will, of course, be the case in an upcoming post (multiple posts most likely). It is next to Bitcoin the most important blockchain around. Ethereum In short: Ethereum is building a decentralized virtual machine where you can run applications. From the Ethereum website: Ethereum is a decentralized platform that runs smart contracts: applications that run exactly as programmed without any possibility of downtime, censorship, fraud or third party interference. These apps run on a custom built blockchain, an enormously powerful shared global infrastructure that can move value around and represent the ownership of property. This enables developers to create markets, store registries of debts or promises, move funds in accordance with instructions given long in the past (like a will or a futures contract) and many other things that have not been invented yet, all without a middle man or counterparty risk. As said, more on that later, for now we just look at the naming system. To understand the ENS basics, I recommend to: read and ENS documentation introduction watch , slides . ENS Ethereum Domain Name System here The currently available TLD is . Let's try to get one... .eth How to register a .eth name The registration for a .eth name is set up as an auction. The details are not really relevant, but you can read more about it here: . registrar.ens.domains This is also the officail app, but it didn’t work for me for some reason. This one works much nicer: . It operates in combination with an Ethereum browser or this browser extension: . myetherwallet metamask You need some Ether (the Ethereum currency) and a bit patience, and you will be the owner of a address. .eth Progress can be followed here . etherscan.io How a .eth lookup works There is a that allows you to hook into the ENS system. This code needs to run on the Ethereum virtual machine. Currently, there seems no other way to interact with it. javascript library How do we resolve a .eth domain to an IPFS hash? Well, we don’t. Although IPFS is mentioned once in the ENS documentation (in the intro) there is no code that makes integration possible. Intermission: now what? As we have seen so far, there is no (easy) way to use blockchain-based names to address IPFS content. We want to be able to pass around , , and addresses and know it will resolve in a trusted way to the correct piece of content. I.e., the content that the owner of such an address wants it to point to. .eth .id .bit One way forward that will certainly happen for Ethereum and Blockstack is that IPFS will be used as one of the supported file systems. In that case, the user lives in the Ethereum or Blockstack world and will be able to use or addresses. .eth .id The alternative, where IPFS itself resolves these addresses is harder because IPFS nodes need trusted access to nodes of the other network. For example, this relies on having access to a Namecoin node on the same machine where the IPFS node runs. This is obviously not feasible (who wants to run a node for every naming system that comes out next to its IPFS node?). proof of concept A proposal to solve this that I like is . I have added my own 2 cents to it, let’s see where it leads. But till there is a solution, we still have a couple of contestants, starting with… pluggable ipns resolvers Proquint Pronounceable Identifiers Another round another chance! From the white paper: There have always been schemes to encode binary into pronounceable words. IPNS supports Proquint. Thus: # this proquint phrase /ipns/dahih-dolij-sozuk-vosah-luvar-fuluh # will resolve to corresponding /ipns/KhAwNprxYVxKqpDZ I’m not sure if this is an improvement, especially because the definition of “pronounceable” is stretched a bit here. As an aside, this remembers me of my first open source contribution back in 2000. There was no github yet, so the source code is gone, but someone put it in a 11 years later (I found it when I Googled for my own name, ahem). To make sure this never gets lost I added it to IPFS. Sort-of-pronounceable password generator pastebin historic piece of code Back to Proquint! This is actually implemented in IPFS. So let’s give it a try: $ ipfs name resolve -r /ipns/dahih-dolij-sozuk-vosah-luvar-fuluhEU�o�C.��M�# yuk So yeah that works, but is not something valid. /ipns/KhAwNprxYVxKqpDZ You can replay it with these online tools: Proquint online tool Base58 encoder/decoder online To encode a valid IPNS peerId is a bit more work to figure out. I’m not interested in that now and also lazy, so I’ll leave that to anyone who likes to :) dive into that Another solution mentioned in the white paper is… Name shorteners From the white paper: Services are bound to spring up that will provide name shortening as a service, offering up their namespaces to users. This is similar to what we see today with DNS and Web URLs: # User can get a link from /ipns/shorten.er/foobar # To her own namespace /ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm So this is based on the ipns feature it can resolve domains via a DNS TXT record. Resolving goes like this: A request for is resolved to . /ipns/shorten.er/foobar /ipns/ /foobar This file to the peerID address of , like: /ipns/ /foobar points foobar /ipns/XLF2ipQ4jD3UdeX5xp1KBgeHRhemUtaA8Vm Now, this pointing is not as easy as it sounds. This was probably the intended use of what is described in the white paper as Peer Links: As encouraged by SFS, users can link other users’ Objects directly into their own Objects (namespace, home, etc). This has the benefit of also creating a web of trust (and supports the old Certificate Authority model): # Alice links to bob Bob ipfs link / /friends/bob / # Eve links to Alice ipfs link / # Eve also has access to Bob / /foo.com Unfortunately, the command is not implemented yet which probably explains why no-one implemented an IPFS name shortener. ipfs link There is a way to do something similar to peer links with , but it is not suitable to build a name resolver. See here for my search for a solution: . ipfs files Are Peer Links supported? Again no success here, but there might be hope for the future… Filecoin As mentioned in a previous post, Filecoin will be the cryptocurrency to help incentivize the file storage marketplace. This means that there will be a blockchain available in the IPFS network. The primary function of this blockchain is to pay and get paid for storage, but since it’s there, it can also be used for other functionality. Like identity or a name system. In other words, just like Namecoin, Blockstack and ENS provide blockchain enabled identity/naming, Filecoin can do the same. We’ll have to see how this all plays out in the next months (years?) but this seems like an attractive option since the IPFS nodes and the Filecoin nodes will most likely be one and the same or at least tightly integrated. The doesn’t mention anything about naming, but it is still very early to say anything definite about it. Filecoin white paper This is an fascinating introduction although he gets into a hate speech for a couple of minutes about the ICO (which I think is not completely undeserved). Filecoin | White Paper Breakdown and Token Sale Analysis OK, more than enough about friendly naming I’d say! Time to move on to the next step in our journey to decentralizing this blog (more soon in the next episode…). Originally published at decentralized.blog .