Benchmarking Xcode Availability Across Hosted CICD Services

Full Disclosure: I work at Bitrise, the folks that gathered this benchmark data and the folks that came out as the fastest: Take that into account when reading, but note that the sources of this data have been quoted so you can validate its accuracy for yourself.... and now on to our regularly scheduled programming:

Xcode versions are the drumbeat that iOS teams all around the world march to. Rapid access to Xcode versions ensures that apps are submitted, and updated on time, while Xcode betas help prepare for new features, functionality, and – dreaded – deprecations.

With WWDC 2021 around the corner, enticing teams with the shiny new things the Apple folks thought up for us, this topic is more relevant than ever.

The research method

Over a 12 month period, a sample of Xcode GMs, RCs, and betas was generated based on data aggregated by https://xcodereleases.com/ - For this sample of 15 Xcode releases, we pulled in publicly available information on Xcode releases on Bitrise, GitHub Actions, CircleCI, and Travis.

Whenever a particular Xcode version was skipped by a CI provider, the next version was used instead to determine availability.

The data

The following table shows the tracked releases and their availability on the different platforms:

To ensure we're comparing apples to apples, the dates listed are based on CET

Speed of Xcode availability on Bitrise

Over the last 12 months, on average, it took 3.4 days to have a new Xcode version available for developers on Bitrise. This includes GM’s and betas.

Even though this 3-4 days average puts Bitrise ahead 3 days ahead of its second-fastest competitor, it falls short of their aim to have new Xcode versions available within 48 hours. Mostly, this is due to a stretch of slower releases in Q4 2020, and Q1 2021 resulting from increased focus and ongoing infrastructure improvement. Regardless, they have expressed a renewed commitment to hit <48 hours availability, where possible.

Bitrise Xcode updates based on stack update notices posted to their Discourse space here

Xcode support on CircleCI

New Xcode versions are usually available on CircleCI within a one-week window, but their average time to availability came in at 7.2 days – a little over a week. This is mostly caused by some slip-ups related to a number of Xcode beta versions that took several weeks to make it onto CircleCI’s macOS stacks.

Note that, at the time of writing, no Xcode 12.5 beta or GM is available on CircleCI’s macOS stacks, so we’ve left this latest version out of our calculations.

CircleCI Xcode updates based on stack update notices posted to their Discourse space here

Xcode support on GitHub Actions

New Xcode versions are generally available on GitHub Actions a little over a week after launch, but their average time to availability came in at 6.5 days – a little under a week. Their average got pulled up by a handful of versions that were supported in 2 or 3 days. Like most CI’s, they also managed to support the Xcode 11.5 GM in a day.

Note that, at the time of writing, the Xcode 12.5 beta made available briefly on GitHub Actions is now only available as a “private preview”. There, it can be accessed to those organizations that used GHA’s macOS 11.0 image while it was in public preview mode.

GitHub Actions Xcode updates based on the commit history for their MacOS 10.15 virtual environments doc here and their MacOS 10.14 doc here

Xcode support on Travis CI

New Xcode support on Travis CI seems erratic, with some versions being supported very quickly (looking at you, Xcode 12.2 Beta 1), but their average time to availability came in at 12.6 days – close to two weeks, and slower than anyone else benchmarked in this article. Time to availability regularly ran over a week, and 12.4 took a full month to support.

Note that Travis CI is one of the few cloud CICD platforms that currently offers Xcode 12.5 builds, but they seem to have skipped the betas and took a little over 2 weeks to make the RC available.

Travis CI Xcode updates based on stack update notices posted to their changelog here

Xcode support on App Center

Even though there are still quite a few mobile teams building there today, we’ve not included App Center Xcode updates in this roundup: For the entirety of the measured period, App Center macOS builds ran on GitHub Action macOS images. Their performance can therefore be assessed as being identical to what we’ve recorded for GHA.

Xcode support on Jenkins

🤷‍♀️ probably captures the sentiment here best: On Jenkins, you or someone on your team will be responsible for ensuring new Xcode versions are available.

How long that’ll take is dependent on a number of factors, including the priorities of the team, or the stability of your Jenkins environment: If it’s already kept together by luck, positive thoughts, and firefighting, those engineers responsible for maintaining it will often be hesitant to rush out an Xcode update… However eager mobile developers might be to get their hands on it.

Previously published at https://blog.bitrise.io/post/hosted-ci-xcode-support