The pioneer’s guide to GX — decentralized dependency management on IPFS

Decentralized package management is an exciting area of tool development for today’s developers. Decentralized package managers have the potential to remove centralized points of failure from the interconnected network of modern software libraries. They give development teams a straight-forward way to ensure their internal networks of code are always available, while at the same time ensuring redundancy across the entire Internet. Decentralized package managers can make our code, one of modern society’s most valuable outputs, more resistant to tampering, censorship, and manipulation. That said, decentralized package management is still incomplete. There are many issues to sort out when publishing our code into networks built around an immutability of digital information where unlimited copies of that information may exist at any time. Without central authorities, they need new solutions for discovery, information removal (e.g. content not owned by the publisher), disputes, and potential bad actors. These issues are shared not just by decentralized package managers, but by decentralized networks in general, and so some solutions have already been proposed, and many others are actively being developed. That said, before using any decentralized package manager, it’s important you understand your risks and responsibilities. I’ve included a more complete discussion of the promise (and some of the pitfalls) we face when decentralizing our source code and network dependencies. If you aren’t familiar with the topic, I highly recommend you read it before working through this tutorial. You can find that here, Decentralized code distribution for the future of open source. So with that, here’s a brief introduction to using the decentralized package manager (specifically the implementation, ) to distribute your code and manage your code dependencies over IPFS. One last thing before we start, here is what the GX developers say in their own , GX Golang gx-go readme gx is . It’s not perfect yet, but it’s proven dependable enough for managing dependencies in and ready for pioneering developers and early users to try out and explore. Alpha Quality go-ipfs An introduction to GX and gx-go The GX library core functionality that is used by both the gx-go and package managers. The basic principles that they all share are, gx-js Code versions you publish are referenced by an immutable hash of the code contained. Dependencies you link into your code link to those immutable hashes both to find and retrieve the code and to ensure you are using the version you expect. The immutable hashes are the addresses where the published code should be available on . IPFS What does all that mean? Well, to start, instead of relying on something like semantic versioning as a global reference to a specific release of code, it uses the fingerprint (the hash) of the code itself. GX also enables a secondary mapping of the hash to a semantic version, this is mostly to improve human interaction with code (e.g. readability). These hashes become part of system used in IPFS. the content addressing If you aren’t familiar with IPFS, there are a few great overviews to explain what the , , and of GX. The thing to understand is that it is a set of technologies and protocols that enable any user to distribute media or data over a decentralized network of computers. In the case of a GX user, their code. As long as someone is distributing the media you are looking for, you can use IPFS to retrieve that data over the network. system is how it works using it outside For GX, this distribution model means that if you want to publish your code over the system, you need to start by making it available on the IPFS network. While you might be the first to host that data, once other people start using it, it will likely become available from other nodes pretty quickly. Textile and other GX users often mirror the code they use, adding redundancy and speed to the network. In the for example, we have a team IPFS node where we pin all the dependencies we rely on from GX. Textile Photos Go library Okay, that’s the basic mechanics. Let’s try using it. Installing GX and gx-go This easy step is required for both of the next sections (distribution and dependency management). Get GX and gx-go running on your system, simply run the following two commands. go get -u github.com/whyrusleeping/gxgo get -u github.com/whyrusleeping/gx-go As long as you added your Go root and correctly to your the above should make the GX command line tools available. If the previous line doesn’t make sense to you, there with Go that could be useful before continuing here. /bin path are a few tutorials on getting started Sharing code on GX We more or less stumbled upon GX because it’s the package manager used in . But using it got us really thinking about all the benefits we mentioned above. GX was originally written to support , but there is also , and the original code is ready to be forked for package managers in any language. It’s extensible enough that it got us thinking about what the future of an mobile app store on IPFS could look like… We’ll save that for another day though. In the following blog post, let’s walk through the basics of using gx-go in your own projects. IPFS related projects gx-go gx-js GX Setting up GX in your Go project From inside the folder of your Go project, you’ll simply run the following in your terminal, gx init --lang=go Running the above is going to add a file to your folder containing a few necessary bits to work with GX. If your project already has a file for some other reason: the important bits will be added. Here is an overview of what information GX stores in your file. package.json package.json package.json Your will list the GX version you are using in this project: package.json "gxVersion": "0.12.1", Also from the , GX adds the import rewrite paths. This will specify how to rewrite import statements in a user’s code (as opposed to showing them non-human readable hashes all the time). Here is what mine looks like, automatically detected from my Go path. package.json "gx": {"dvcsimport": "github.com/textileio/gx-demo"} Next, the contains the helper command that will later tell GX how to handle new releases for you. package.json "releaseCmd": "git commit -a -m \"gx publish $VERSION\"" That’s it. Now, your code is ready to start publishing on GX. Let’s try publishing the version version of your code on GX. To do that, we can just commit our code using GX, which will run the script above. releaseCmd gx release 0.0.1-dev Error: ipfs daemon isn't running Now… you may have hit the first gotcha. Like I said in the introduciton, GX uses IPFS to distribute your code, which means you need to have an IPFS daemon running on your system. I’m not going to walk through the here or the to get the daemon up and running. But once you’ve done that, you should be able to run again and get an output like, installation steps one-liner you need gx release 0.0.1-dev gx release 0.0.1-dev package gx-demo published with hash: QmZFb51F1rJcHy9UUWWgPvwJAMorMgdzw2a52y2xgjVZJu Now, anyone should be able to access your code through GX or directly through IPFS (or even a gateway). You may have noticed that when your ran a new file was created, . This file contains a mapping of semantic version number to the IPFS hash. You should include this folder in your git history as it can link the code to the releases at any point in the future. Unlike a hash, a semantic version isn’t guaranteed to be immutable. Even in GX, you could republish the same semantic version with a totally new set of code, and therefor a new hash. gx release .gx/lastpubver Publishing a new release on GX Okay, so you’ve gone through and changed your code, now you are ready to release a new version. You now have a couple of options. GX doesn’t have many rules here so technically you could republish the same version number again just by running, . But more likely you are going to want to release the next version, so just run, gx release 0.0.1-dev gx release 0.0.1 package gx-demo published with hash: QmcUaXVES69qZEhV8tdUbhYtVBkbGU7b9h7bpvAo6xnvC9[master 6f1a98f] gx publish 0.0.12 files changed, 2 insertions(+), 2 deletions(-) You’ll notice that your has been updated and you can now share your new hash with the web. .gx/lastpubver Installing dependencies from GX If you want to write source code that is resistant to centralized services going offline, changing their roadmap, or being blocked, then you may want to use GX for managing your dependencies. You don’t necessarily have to publish your project on GX in order to use GX this way and not every library is available on IPFS, but hopefully more and more will get there in the months to come. So let’s take a look. The first thing you can do is import a new dependency. You’ll need to have a target hash for GX to be able to do anything. Okay, so here is an project we could use . It’s a straight-forward library that can convert in and out of in Go. So, if you know what you’re looking for you’ll notice that the repo doesn’t contain any file. So we have a couple options: open source https://github.com/mr-tron/base58 strings base58 .gx/pubver We can use the steps in the above section to add GX to the project and get the hash that way. After that, we can use and share this dependency and we can help out the network by pinning the source on our IPFS node. We can convince the owner to do it themselves. Making it easy on us, someone has already taken the first route with this project. Forking the original base58 project and publishing the fork with GX version here, . So now, we can just use the published hash to attempt and install the import. https://github.com/gxed/base58 gx import QmWFAMPqsEyUX7gDUsRVmMWz59FxSpJ1b2v6bJ1yYzo7jY update imports of github.com/mr-tron/base58 to the newly imported package? [y/N] , you’ll prompt GX to rewrite the imports of the library to use the GX provided library. So in this case would look more like, .It is recommended that you your projects to GX with their imports rewritten. Instead, let that be dealt with by GX downstream. By answering **yes** to the prompt import github.com/mr-tron/base58 import gx/ipfs/QmWFAMPqsEyUX7gDUsRVmMWz59FxSpJ1b2v6bJ1yYzo7jY/go-base58-fast/base58 not publish GX will use your file to track these dependencies and handle future rewrites / un-rewrites. You can find the list of your GX managed dependencies in the section, which looks like, package.json gxDependencies "gxDependencies": [{"author": "mr-tron","hash": "QmWFAMPqsEyUX7gDUsRVmMWz59FxSpJ1b2v6bJ1yYzo7jY","name": "go-base58-fast","version": "0.1.1"}] Start using GX today! That’s it, just a few commands and you are up and running… but… oh wait. You might hit some snags around project availability. At , we collaborate on code and each can update the dependencies. This means, if I add a dependency and am the first IPFS peer to pin the source code, any of my collaborators might hit snags when they try and install it and the code isn’t available. GX does its best, but sometimes still has issues. To solve this on our team, we’ve been pinning all our GX dependencies to our shared IPFS server. Textile By pinning the code our work depends on we are also making the redundancy and availability of the network better. You should join the effort :) If you are interested in more of our development work, check out our library or our library on GitHub. If you are interested in other thoughts on the IPFS network, check our our . And of course, if you just want to have IPFS running on your phone, storing and preserving your photos, or enabling peer-to-peer sharing, . textile-go textile-mobile plans to add the next 1,000,000 peers to the network get on the list to try out our mobile app