Speeding up iOS builds by 'fanning out' tests on Bitrise

📝 UPDATE:

This blog post describes a legacy solution. To run builds in parallel, learn more about Build Pipelines.

‍

Speed and quality can go hand in hand, but sometimes you need a bit of tinkering to get both. Try this Bitrise setup to split tests up, run them in parallel and speed up your iOS builds.

TL;DR

Just want to try it out for yourself? Find the code and assets used to make this post here:

iOS Test App

bitrise.yml

Fan Out Calculator (How many "fan outs" are likely optimal for your use case?)

Fan Out for Android

Before we get started

• This article assumes you have a Bitrise account. If not, register here (the standard free and Developer plans have a single concurrency, so create an organization if you'd like to try out our Org Standard plan with two concurrencies for free)
• To run workflows in parallel, you'll also need multiple concurrencies. If you'd like to upgrade your plan to have more of those (or want a trial, demo or anything to help explain to your CTO why this is useful) reach out to the team here
• This is "advanced Bitrise": By following the example, you should be able to achieve (significantly) faster build times for specific cases, but the results will vary (they might be worse, the same or even better than mine) and you'll end up with a more complex workflow. If you're willing and able to add some complexity to increase performance, though, read on:

Automated testing on iOS Simulators can be a slow process, especially when working with a large number of test cases. Apple provides a way to run your tests in parallel, but it can result in non-linear speed improvements and increased stability issues.

Tests sometimes fail!

Whether we like it or not, tests fail! Sometimes they fail due to code changes, but sometimes they fail due to simulator crashes. Apple automatically provides the ability to run iOS Simulators in parallel, but depending on your build machine, the number you can run at one time is limited.

iOS Simulators Crashes

To find out just how often iOS Simulators crash when running in parallel, we took the open source Firefox iOS application and ran the same iOS Simulator tests repeatedly with varying amounts of parallel simulators.

We tested this on Bitrise's Standard & Elite Mac OSX Machines

• Standard - 2 vCPU @ 2.70GHz 4 GB RAM
• Elite - 4 vCPU @ 3.5GHz 8 GB RAM

Total Tests Runs

Average Build Length

Speed Vs 1 Simulator

Failure Rate

Conclusion

When you increase the number of parallel simulators, you trade speed for stability. This can lead to random builds that fail, but then pass when run again.

Solution

You can speed up and increase the stability of your tests by splitting them up and running them on multiple virtual machines in parallel.

In this post, we will show how to build an app once, run the tests in parallel, and collect the results to display in a test report.

Running Parallel Test Builds!

You can start parallel builds and wait for them to complete using the Bitrise Start Build & Bitrise Wait for Build steps (read some documentation on that in the DevCenter). This allows you to run more simulators at once.

This example shows how to:

1. Trigger X test workflows from a Primary workflow and wait for them to complete
2. Gather the test results from all X fan out workflows via the Bitrise API using NodeJS
3. Generate a single Test Report for all X test results using bash

Workflows Explained

Primary Workflow

The primary workflow triggers the fan out builds using the Bitrise Start Build & Bitrise Wait for Build steps.
In order to not have to rebuild the app multiple times, we can leverage the Cache Push & Cache Pull steps, or other steps like the S3 File Uploader step.

Fan Out Workflows

The test_x workflows triggered by the primary build will run in parallel on the Derived Data created in the Primary build. Each of these workflows can target a different XCode Test Plans or XCode Targets. For simplicity, in this example we will use the same XCode Target for each fan out.

No Fan Out Workflow

The no_fan_out workflow runs the same number of tests in serial for performance comparison.

Collecting build artifacts

Using the Bitrise API we can get the build artifacts back into the primary build for processing. We can do this in a script step easily but for convenience I created a custom step to do it for you.

Custom Bitrise step to automatically retrieve all artifacts created by the fan out builds (work in progress)

Splitting test into chunks

To avoid the need to make changes to your XCode Project you automatically generate XCode Test Plans with equal chunks of tests for use in you fan out builds. This will mean there is no need to make changes to your existing tests in XCode.

In order to automatically split your tests into testable chunks I created a step for that. The step will automatically create X Test Plans from a given XCode Target:

https://github.com/DamienBitrise/bitrise-test-plan-sharder (work in progress)

Here is a version of the sample app with a bitrise.yml you can use to try out auto generated test plans:

https://github.com/DamienBitrise/bitrise-fan-out-auto-generated-test-plans (also a work in progress)

Creating Test Reports

By collecting the test xcresults from the fan out builds we can convert them to JUnit XML for display in the Test Reports Add-on.

Converting xcresults → JUnit XML

We can do this conversion using a tool called Trainer from Fastlane

Results

Although this is a trivial example with very few tests, it shows how you can apply fan out builds to your own workflows.

No Fan Out 6 Test Runs 6.8 mins

Here you can see that the tests happened one after another and the total build time was 6.8mins in this simple example. Depending on how many tests you perform your results may vary.

Fan Out with 6 fan outs 5.0 mins

Here you can see that the tests happened in parallel and the total build time was 5.0mins in this simple example. Depending on how many tests you perform your results may vary.

What else could you use Fan Out builds for?

• UI Testing
• Unit Testing
• Code Coverage
• Linting
• Building App Variants
• App Store Uploads
• and more…

Any sequential action that could be split up to run in parallel, can be split out over workflows to cut down on the time spent waiting for a build to finish tremendously. Have you tried the fan out setup and want to share your experiences?

Find us on Twitter, drop by our LinkedIN or Facebook pages, or simply start a topic on Reddit - We're always happy to jump in on the conversation.

Happy building! 🚀