Libraries.io February Progress Update

Continuing from last month with the regular progress updates on Libraries.io since securing a grant from both the Sloan Foundation and the Ford Foundation, here’s what we got up to in February 2017.

BitBucket and GitLab support

The biggest feature to ship was support for finding and parsing dependency information from source code repositories hosted on GitLab and Bitbucket.

Previously Libraries.io could only pull in information about GitHub repositories and it’s taken a few months to untangle the hard coded GitHub logic throughout the application.

We’re slowly spidering both services and so far have around around 300,000 repositories on GitLab and Bitbucket, this helps give a much better picture of the whole open source ecosystem rather than just the one on GitHub.

The large refactoring to get that shipped also means in the future we can add support for other hosts like SourceForge, Launchpad and self-hosted GitLab repositories much more easily.

Other Notable Features

During the great repository refactoring, we also made switched the approach to parsing dependency information we were taking.

Previously we had a separate node.js web service call Librarian, which the background workers interacted with over http, but now we’ve switched to using the ruby library Bibliothecary for parsing dependencies, which runs directly in the background workers.

With this we reduced the amount of different web services we needed to run, simplified a whole lot of code and everything is in one language, this also means that Libraries.io is now sharing it’s parser with Dependency CI, so an improvement to the library benefits both services.

We’ve started to codify the infrastructure and server configuration that Libraries.io with Ansible, it’s all being done in the open on the new infrastructure repository: https://github.com/librariesio/infrastructure

I also rolled out a new small service to replace our usage of the npm rss feed, which was pretty terrible, instead npm-update-stream directly follows the changes feed and produces a list of recently updated project names for watcher to consume every 60 seconds.

https://docs.libraries.io/

Ben has been working to improve Libraries.io’s documentation throughout February. He re-wrote much of the old, developer-centred documentation to make Libraries.io a more welcoming project for designers, copywriters, and users alike. You can check it out at docs.libraries.io

A big part of this has involved re-writing our contributor’s handbook and the templates we use for issues and pull requests, which we’ve rolled out to all the repositories in the Libraries.io organisation.

In line with our improvements to documentation we’ve also improved our ticketing. Inquisitive, would-be contributors can now find issues to pick up by size, skill or type and can easily find issue to help others with. We even have a label for first-time contributors, explanations of each label can be found on the documentation repo.

And finally, in return for financial support we made a number of commitments to the Alfred P. Sloan Foundation and, as of January, the Ford Foundation. You can now see those applications and our commitments to these institutions in our new supporters repo.

A full list of all changes right across the Libraries.io org on GitHub is available in this gist, generated by: https://github.com/librariesio/org-pulse

Unseen Open Source Infrastructure

At the start of February I wrote a post about Exploring Unseen Open Source Infrastructure, along with an experimental page showing the most depended upon repositories with very few stars.

The blog post went down well and has helped kicked off a number of conversations around how we can use Libraries.io data to pinpoint important projects that might need more support.

Statistics

42,594 new libraries and 265,766 new versions found, bringing the total to 2,154,734 libraries indexed.

701 commits, 33 pull requests and 220 issues opened across all Libraries.io repositories on GitHub: https://github.com/librariesio

Libraries.io has also discovered over 20 million open source repositories from GitHub, GitLab and Bitbucket.

Contributors

This month we had a quite a few new contributors whom we would like to thank:

• Casey Strouse
• Jon Moss
• Timothée Mazzucotelli
• Sam Clift
• Carol Willing

Plans for March

The first week of March we’ll be continuing our documentation push, getting involved in Docathon, Ben has written more about that on his Medium.

After that I’ll be focusing on beefing up the postgres and elasticsearch infrastructure and refactoring the organisation/user mapping to be less GitHub specific and bring in support for GitLab and Bitbucket Users and Teams.

As always, follow us on Twitter at @teabass, @benjam and @librariesio for more updates.