InfluxDB: From Open Source Time Series Database to Millions in Revenue

At Flagsmith, we are working on building a commercial open source feature flagging product. As we work through the struggles of surviving and thriving as a small software company, we connect with fellow founders to see how they built their open source projects into successful businesses. Most recently, we interviewed Paul Dix, co-founder and CTO of InfluxDB on “The Craft of Open Source” Podcast.

The company was originally founded in 2012 by Paul, as a closed source SaaS product called “Errplane”. The focus was to help companies with real-time metrics and monitoring, but by 2013, it was clear that the product wasn’t taking off. Determined to find a market, Paul and the team realized that the time-series infrastructure they had built could be useful for other businesses. 

In September 2013, after not having the initial traction with Errplane they wanted, they decided to use the infrastructure to build a stand-alone time-series database engine. At the time, very few commercial companies and open source projects were focusing on time-series databases. The most notable open source project of the time was Graphite, which had become an orphaned project, but still retained a big following.

The fact that this project still had engagement, but wasn't being driven forward in a focused manner, told Paul and the team that there was space in the open source community for a new time-series data base project.

On the closed source commercial side of the market, there were a couple of time series commercial offerings at that time, but only in the financial data space - Onetick and KDB. However, these products were designed for a small number of time series that moved at high velocity, high frequency trading for example, which had a small data series that was updated 100,000’s times per second.

Other applications such as sensor or server-monitoring data, demanded the opposite - with potentially hundreds of millions or even billions of unique time-series (especially for big companies such as FB or Netflix), that moved at a slower rate. Commercially, nobody was catering for this specific market back in 2013.

As they looked at their own customer base, Paul discovered that many of their existing Errplane customers were just using the application as a time-series solution, building on top of the API using the generic dash boarding solution to handle their time series. The product was beginning to take shape along with the open source opportunity...

Most companies that were trying to “roll their own stack” were using Graphite, which is a limited “round-robin” database, only designed for regular time series. In other words, it  takes samples at fixed intervals - e.g. sampling once per minute, and not in real-time.

This meant it could only be used for metric measurement. As Graphite was not designed for large scale applications, Paul saw an opening for event data time series - logs, exceptions, individual requests through an API, etc. which all use time series data that Graphite couldn’t handle. Anything above a quantity of 100,000 data series would fall down with Graphite, making scaling difficult.

At the time, most database engines were focused on NoSQL, such as MongoDB, which became the number one NoSQL engine. Paul watched MongoDB closely and learned from what they did, in particular how they nailed the developer experience.

They made it super-easy for developers to build their application quickly and become more productive. So, Paul realized it was important to make InfluxDB super-easy to use, helping developers to work faster, and build a time series app as quickly as possible.

In other words, Paul started without a clear monetization plan, but was confident that there would be some way to make money further down the road. The important thing was to make it open source as that was the only way developers would buy into it.

InfluxDB started as a greenfield project, brand new code, but built from the infrastructure of the SaaS product. To make it user friendly, Paul implemented a query language that looks and feels like SQL, but with their own parser and executor, etc. InfluxDB was ready after 5 weeks of coding and was released using MIT-licensed open source libraries. The goal was to create an open source distributed DB with no external dependencies. In other words, you install and you’re good to go.

Back in 2016, the series A round of funding raised 8.1 million. It quickly got to a point where the project needed series B funding. Normally you need to show proof of some significant revenue to get series B, but InfluxDB had none at the time. Paul realized that one way to monetize, was to offer high availability and scale-out clustering. This had the potential to make money in a closed source way. 

In the podcast, Paul goes into detail about how they monitored popularity and community growth using a variety of metrics, and a "phone-home" feature which gave them some data on how it was being used.  

Of the first 10,000 servers by Jan 2016, 99% were individual servers, not clusters. This wasn’t surprising. All the traction InfluxDB had gained so far was focused on what you could do on a single server only. Paul recognized that they could make a closed source version of the product that would offer a service for high availability and scale-out clustering. 

In March 2016, Paul wrote a blog post announcing that all future versions of OS influxdb would be single server only, and there would be a commercial version for clustering. This made the front page of HackerNews and generated some criticism - people accused InfluxDB of doing a “bait and switch”. 

On the plus side, there was immediate interest, with people contacting and asking how they could buy the commercial version. 

In the podcast, Paul discusses how their philosophy was to make their open source completely open, licensed through MIT for example, and make the commercial products completely closed source and the reasons for this.

He also states his dislike of source-available code, which doesn’t allow for a clean implementation. Source-available is only good for a freemium product, similar to old shareware games where you’d get a free version that you could upgrade to a paid version. But that is not open source. Paul believes that true open source means that anybody can take the code and build a business from it.

Therefore, InfluxDB maintained their open source single server engine, but the clustered version is a paid product. Paul recalls how this was a gamble at first as they weren’t sure if there would be enough interest in buying the product, and they risked losing the support of our OS community as a result.

Fast forward to now, and the business is built around the paid clustered version. They now have two offerings - an on premises, site-installed version and a cloud-based service hosted on AWS. The cloud version allows you to pick a size, and they provision the new cluster for you, install the software, monitoring systems, and run them. 

By Jan 2018, they were generating consistent revenue, so they raised series C funding which equated to around $30 million. 2018 was even better, so in Jan 2019 they raised series D funding of around $60 million. This was all based on commercial traction from the single paid product. The OS community has grown massively too - from 10,000 servers in 2016 to more than 400,000 single servers today.

Paul has recently announced a new project that he’s building called InfluxDB Iox, which is a new core time series engine, written entirely in Rust.

Iox is an in-memory columnar store with object store optimised for time series, but it’s also a system for defining replication rules, and data partitioning rules within a set of servers. The idea is to create an open source engine that continues doing what Influx is good at, but also adds best-in-class performance for larger scale analytic queries.

The commercial aspect of this new project is to run it as a cloud-based solution. The cloud based version requires extra software to coordinate and operate a fleet of servers. This software is closed source and will be on a paid licence. 

Instead of running open source code, then upgrading to the paid version if you want high availability or scale-out capability, which is a big ask, Iox takes a different approach where the commercial product is complementary, rather than an entire replacement of the open source version. In other words, if you switch to the cloud version, you’re making the running and management of the servers easier and you’re not turning anything off in the process.   

Paul sees Iox a big win both technically and architecturally. The separate cloud-based software idea makes the whole thing much more flexible for the operator, allowing them to apply a number of different cluster topologies and operational environments.

In the podcast Paul goes into detail on how Iox works and how everyone in the ecosystem can benefit from it. His ultimate goal is for Iox to crack the top ten in database engine rankings. 

The 2.0 cloud product is designed to give much more control over the DB and operations. When moving from InfluxDB 1.0 to IOX 2.0 there will be no customer downtime or migration issues.

For open source users, Paul plans to make IOX available as a backend to be used with InfluxDB next year. 

Paul believes that Iox won’t split the community, as 1.0 and 2.0 are complementary. Most people use InfluxDB around the edges, not as a core database. Iox brings in API compatibility, so people who use it around the edges at the moment, will be able to use it in the same way, and people who want to create a new core database can use it too.

If you want to find out more about open source InfluxDB and the new Iox project, then listen to the podcast which goes in-depth into the projects, and on the topic of open source in general.