Hosted CI Xcode availability benchmark: Does Bitrise still support new versions fastest?

The results are in: Over the last year, Bitrise provided developers with new Xcode versions for their iOS builds faster than any other CICD platform. Check out the details in this Xcode support for CICD benchmark roundup.

The results are in: Over the last year, Bitrise provided developers with new Xcode versions for their iOS builds faster than any other CICD platform. You found Xcode 12.5 here first, and the numbers show we’re significantly faster than the alternatives.

Want to build on Xcode 12.5 today? Sign up to Bitrise or reach out to us to discuss your needs.

Already on Bitrise? To select Xcode 12.5, you just have to open your app on 14, go to the Workflow tab (Workflow Editor), and on the Stack tab select the Xcode 12.5.x option and your next build will start on the Xcode 12.5.x stack.

Speed of Xcode availability on Bitrise

Over the last 12 months, on average, it took 3.4 days to have a new Xcode version available for developers on Bitrise. This includes GM’s and betas, a selection of which is posted in a table with the release dates and availability on Bitrise, GitHub Actions, CircleCI, and Travis CI.

With an average of 3.4 days between release and availability, Xcode updates on Bitrise are 3 days faster than the second fastest competitor

Even though this 3.4 day average puts us a full 3 days ahead of our second fastest competitor, it falls short of our aim to have new Xcode versions available within 48 hours. Mostly, this is due to a stretch of slower releases in Q4 2020, and Q1 2021 resulting from increased focus on our upcoming high-performance Gen2 infrastructure. Regardless of the reason – it highlights the need for us to keep ourselves and each other accountable for hitting this target.

Xcode support on CircleCI

New Xcode versions are usually available on CircleCI within a one week window, but their average time to availability came in at 7.2 days – a little over a week. This is mostly caused by some slip-ups related to a number of Xcode beta versions that took several weeks to make it onto CircleCI’s macOS stacks.

Note that, at time of writing, no Xcode 12.5 beta or GM is available on CircleCI’s macOS stacks, so we’ve left this latest version out of our calculations.

CircleCI Xcode updates based on stack update notices posted to their Discourse space here

Xcode support on GitHub Actions

New Xcode versions are generally available on GitHub Actions a little over a week after launch, but their average time to availability came in at 6.5 days – a little under a week. Their average got pulled up by a handful of versions that were supported in 2 or 3 days. Like most CI’s, they also managed to support the Xcode 11.5 GM in a day.

Note that, at time of writing, the Xcode 12.5 beta made available briefly on GitHub Actions is now only available as a “private preview”. There, it can be accessed to those organizations that used GHA’s macOS 11.0 image while it was in public preview mode.

GitHub Actions Xcode updates based on the commit history for their MacOS 10.15 virtual environments doc here and their MacOS 10.14 doc here.

Xcode support on Travis CI

New Xcode support on Travis CI seems erratic, with some versions being supported very quickly (looking at you, Xcode 12.2 Beta 1), but their average time to availability came in at 12.6 days – close to two weeks, and slower than anyone else benchmarked in this article. Time to availability regularly ran over a week, and 12.4 took a full month to support.

Note that Travis CI is one of the few cloud CICD platforms that currently offers Xcode 12.5 builds, but they seem to have skipped the betas and took a little over 2 weeks to make the RC available.

Travis CI Xcode updates based on stack update notices posted to their changelog here.

Xcode support on App Center

Even though there are still quite a few mobile teams building there today, we’ve not included App Center Xcode updates in this roundup: For the entirety of the measured period, App Center macOS builds ran on GitHub Action macOS images. Their performance can therefore be assessed as being identical to what we’ve recorded for GHA.

Xcode support on Jenkins

🤷‍♀️ probably captures the sentiment here best: On Jenkins, you or someone on your team will be responsible for ensuring new Xcode versions are available.

How long that’ll take is dependent on a number of factors, including the priorities of the team, or the stability of your Jenkins environment: If it’s already kept together by luck, positive thoughts and firefighting, those engineers responsible for maintaining it will often be hesitant to rush out an Xcode update… However eager mobile developers might be to get their hands on it.

Why we’re committed to having Xcode versions first 

Xcode versions are the drumbeat that iOS teams all around the world march to. Rapid access to Xcode versions ensure that apps are submitted, and updated on time, while Xcode betas help prepare for new features, functionality, and – dreaded – deprecations.

With WWDC 2021 around the corner, enticing teams with the shiny new things the Apple folks thought up for us, this topic is more relevant than ever.

Wondering what iOS community members like Kaya Thomas, Donny Wals, Anastasiia Voitova, Antoine van der Lee, Ellen Shapiro and Carola Nitz expect, fear and hope from WWDC 2021? Check out this playlist of interviews hosted by Vincent Pradeilles, answering those questions and more. Also, sign up for our post-WWDC webinar where some of the interviewees will be joined by Paul “Hacking with Swift” Hudson for some early WWDC reactions.

In keeping with our mission to support you in building amazing mobile apps, faster, we always keep a close eye on the latest tech stacks. Whenever updates are required, we have a dedicated team that springs into action, ensuring you and your team have access to the latest SDKs for iOS, iPad, and watchOS, but also OS updates like Big Sur. 

Where our target of no more than 48 hours between Xcode release and availability might seem arbitrary, having a target matters, and hitting it matters even more. Especially to ensure that everyone in Bitrise understands that having the right tools and support at the right time, is what separates a mobile CICD platform from something generic.

Although at times we had our work cut out for us, we’re more committed than ever to provide you with Xcode updates as soon as possible 🚀

Overview of Xcode support on different CI/CD platforms in the past 12 months:

a table of Xcode releases versus their availability on different CICD system
To ensure we compare ‘Apples to Apples’ (pun most definitely intended), the listed dates are always based on CET. When a version was skipped or not made available, we’ve opted to use the date of releases for the next version as it’s indicative of the availability of necessary features and functionality to iOS devs

For more information about available Stacks and the Stack Update Policy on Bitrise, visit our DevCenter page. For updates about other releases, you can follow us on Twitter. If you have any questions, reach out to the team via our contact form.

No items found.
The Mobile DevOps Newsletter

Explore more topics

App Development

Learn how to optimize your mobile app deployment processes for iOS, Android, Flutter, ReactNative, and more

Bitrise & Community

Check out the latest from Bitrise and the community. Learn about the upcoming mobile events, employee spotlights, women in tech, and more

Mobile App Releases

Learn how to release faster, better apps on the App Store, Google Play Store, Huawei AppGallery, and other app stores

Mobile DevOps

Learn Mobile DevOps best practices such as DevOps for iOS, Android, and industry-specific DevOps tips for mobile engineers

Mobile Testing & Security

Learn how to optimize mobile testing and security — from automated security checks to robust mobile testing and more.

Product Updates

Check out the latest product updates from Bitrise — Build Insights updates, product news, and more.

The Mobile DevOps Newsletter

Join 1000s of your peers. Sign up to receive Mobile DevOps tips, news, and best practice guides once every two weeks.