We develop . It’s an open source visual programming IDE available for Windows, macOS, Linux, and browser. From the very beginning, we used to build, test, and pack release distros on Linux and macOS. And recently we switched to . I’m happy with the change. Why? Read on. XOD Travis CI Circle CI Performance problems For the last year and a half, XOD has grown significantly in code. And if a year ago the full CI-cycle took 5 minutes or so, recently it went over 25 minutes in machine time. That’s for work-in-progress integration. When we release a new version, the duration can go over 90 minutes. Put here additional infrastructure problems of Travis itself (more on this below) and merging 3–4 pull requests per day is the maximal pace we can afford. You can think we have a too massive project that should be split into multiple repos. Well, it a , but it doesn’t stop a developer to entirely build and test XOD locally within few minutes on his machine. is big monorepo Mac provisioning time 25-minute builds is something we could accept for a while. Few opportunities to parallelize builds still left. But there’s an issue we can’t affect anyhow on Travis. It’s Mac provisioning. It literally can take hours. That’s no secret. See “Backlog macOS Builds for Open Source projects” on their . Hundreds (!) of builds are stacked in the queue waiting for a machine. status page The plot shows queue length. The groups of five hills are workdays of a week. What’s making things worse, I can’t pay to get some priority-pool or something. Two months ago or so they force us to move from commercial plan to the free one just because our GitHub repo is public. To be fair Travis devs are trying to solve the problem for the last few months. They place some Mac limits per project, etc. But it doesn’t help. I can’t understand why they “simply” can’t double Mac-cluster capacity and take money if you don’t want to sit for ages in the queue. The last straw As I have said, when we make a release CI builds the distros. The is quite complex per se and involve few hours of careful work. The last thing you want to see is a build failure when you know everything compile fine because you rehearsed it the day before. release process should Finally, we just hang. We were not able to release 0.16.0 because functional tests on Travis failed randomly for no apparent reason. Each rebuild cycle took 20 minutes (plus 30–90 minutes for provisioning) and fail again in another “random” point. A pretty common scenario. The build has not even started after an hour of wait. Well, there a flaky functional test problem in our suite. We rely on WebdDriverIO to run the application and simulate user actions. It seems to have a performance-dependent bug which doesn’t appear on local machines and Circle CI as well. We’re going to rewrite the test suite using Puppeteer and headless Chrome, but it will take time, and we want to keep things running until it’s done. was The show-stop caused me to search for a replacement immediately. Circle CI as an alternative The choice was pretty easy. I found no CI SaaS which supports Macs other than Travis and Circle. I gave Circle CI a try. Surprisingly, the migration was quite easy and took less than 24 hours. We were able to release next day. The switch was simple not because there’re some “import” tools or the configuration structure is very similar, no, they differ much. It’s because Circle CI recently launched their brand new 2.0 platform with . excellent docs The 2.0 concept is so simple and powerful that I was able to replicate 90% of our CI process with caching, parallelization, GitHub integration, etc in less than a day. Having few week experience with CircleCI I can compare these two services side by side. CircleCI vs TravisCI Performance Linux regular build: Travis ~12m, Circle ~9mLinux release build: Travis ~15m, Circle ~9m (no change due parallelization)Linux provisioning: few seconds on Travis and Circle macOS regular build: Travis ~21–25m (not stable), Circle ~9mmacOS release build: Travis ~24–45m (not stable), Circle ~9mmacOS provisioning: Travis weekend — few seconds, Travis workday — from 5m to few hours, Circle — few seconds Branch build wall clock time: Travis — from 30m to few hours, Circle — 9m Yes, Circle builds XOD in 9 minutes, always, no matter what is the day today and where The Sun is. Travis builds it in 30m if you’re fortunate. The total time is 09:35. Parallelization helps a lot. That’s all you should know if you’re going to build on Mac. CircleCI Winner: Machines and infrastructure Travis and Circle seem to use similar Linux-machine configurations. But CircleCI truly supports Docker containers. I can use own docker image to run the build. So I can have everything preinstalled, configured, and tweaked to save few minutes at the build start. On Travis I have to `apt-get update && apt-get install` over and over again. Macs are provided by for Travis. Not sure about their config, but they’re 2× slower to perform the same tasks comparing to their Linux counterparts. I don’t know where CircleCI takes their Macs, but they are equivalent to Linux machines in performance. Mac Stadium Neither of services provides Windows machines. That would be a benefit for either one. In that case, it would be the only CI SaaS on the market with all desktop OS’es support. huge : CircleCI Winner Web interface Travis UI looks old-school. Circle is more modern in aesthetics. But, from the usability point of view, Travis is more clear. They have less visual noise, and more explicit accents done by color and animation. If something is running, it is spinning. If something errored, it is presented in vivid red on a basically gray-scale page. CircleCI constantly tries to switch me to another GitHub organization which is very confusing and looks like I lost all history. The dashboard (i.e. the start page) is quite meaningless in CircleCI if you use their modern features and recommended workflow. Buttons to restart a build not always do what you expect from them and the buttons themselves are not always set in intuitive places. Can you spot the button to retry the failed build?… Yep, there is no such one on the build details page. You have to go back to build list and there will be “Restart” below failed builds. But (!) it will not recover your build anyway 😬 On the other side, CircleCI has desktop notifications. That was the first time when I accepted “Show desktop notifications” from Chrome. They’re very handy for CI. Both services have room for many UI/UX improvements. : Travis Winner Build log Don’t know what the technical problems are, but on Travis, the log gets damaged from time to time or fails to render at all. The records themselves are pushed to a continuous roll with hard-to-find task boundaries. On CircleCI I did not found any log problems yet. And the output is clearly separated in collapsible sections with timing information. : CircleCI Winner Configuration flexibility Both services use YAML configuration that you store right in the repo. But architecture is quite different. Travis defines a pretty rigid pipeline. Your project installs, builds, and deploys. You can deploy here, and there, period. “ — But, but… we want to deploy on Google Cloud Storage. — No way, make it a part of the build step.” To achieve parallelism on Travis, you should use the Matrix build feature that exists for other purposes. There you assign different sets of environment variables, and then perform a little Bash voodoo to skip some steps of the build and complete other. CircleCI 2.0 has a workflow concept that’s simple and powerful. It lets you perform jobs fan-out, fan-in, sequences, and all this concerning the branch and tag of Git commit. Super cool. Both services lack some features. E.g., neither can observe only a part of repo, so I can’t skip lengthy tests if I work on docs. CircleCI configs tend to have much code duplication because I can’t reuse of job steps across macOS and Linux. The cache system is pretty basic: in both cases I can’t cache programmatically derived paths, only hard-coded directories. a part : CircleCI Winner Pricing and open source support Travis charges for workers, i.e. for concurrent jobs. For $129/mo you get 2 concurrent jobs. That’s all, pretty simple. Circle is a bit more complicated. It charges for Linux and macOS parts separately. You get one Linux worker for free and each additional costs you $50/mo. To use macOS you have to pay $49/mo for 2 workers, but it is additionally limited to 500 minutes of CPU time, so if you have a bigger project, be ready to give $129/mo for 5 worker Macs and 1800 minutes of CPU time. XOD is an project and both services support open source movement (they market their services pretty well, I guess). What does an OSS project get from these services? open source Travis lets you build on Linux and macOS for free. $0 for 5 workers. If you’re open source want to pay for better capacity, you can’t. and CircleCI gives 4 Linux workers for free and 2-worker/500-minutes free macOS. Free Macs available only after a talk with CCI team. Additional resources you can buy on a regular basis. : Draw Winner To me the choice is pretty obvious. CircleCI 2.0 seems like a next-generation system comparing with Travis. But time flows, products evolve. If you’re going to choose a CI SaaS, give both a try first. Each one offers free 14-day trial. I wish both services a good luck.
