Xcode is excellent and got better over the years if it comes down to tests. Last WWDC 2019 brought us Test Plans, and earlier, we got features like code coverage and parallel testing.
Guest post by Antoine van der Lee, Lead iOS Engineer at WeTransfer.
As a Lead iOS Engineer at WeTransfer, Antoine’s work is focused on code architecture and team processes. He's passionate about contributing to the iOS community where you might know him from his weekly blog posts on his personal blog called SwiftLee. He particularly enjoys speaking on best practices for structuring code architecture in a way that creates sustainability, as well as open sourcing frameworks and how iOS developers can be more successful in their work.
A thing that is not so easy to fix by Apple are flaky tests. It's something we'll all run into someday when writing tests in general as it's not something specific to Xcode or Apple products. Many other languages have the same issues, and it cannot be enjoyable while writing new tests.
You've finished your feature, opened your pull request, and you like to merge it in after your colleague's approval until you realize that your tests are only failing on CI. One of the causes could be flaky tests, so let's dive in, see what they are, and how you can work towards solving this issue.
What are flaky tests?
A flaky test is a test that sometimes succeeds and sometimes fails as a result of using different setups. One setup might be faster or better performing than another setup. Therefore, it's quite common to run into flaky tests when working with a CI tool like Bitrise, as it's likely using a different configuration compared to your local machine.
Flaky tests can occur on CI tools but also within a development team. If your colleague is using a different machine with a different configuration, it could suddenly be that tests succeed on your machine while not on your colleague's.
How to identify flaky tests
If a test fails, it's not necessarily a flaky test. You have to run it a few times to see whether it's consistently failing or not. If the test fails every time, its cause is something else.
Once a test is only failing on CI, you need to verify that its cause is not any other issue. It could be related to your CI setup, for example. You can take a look at Bitrise's known Xcode issues to see whether its cause is something else.
Common causes that lead to inconsistent tests
Some might argue that CI tools are the cause of flaky tests, but most of the time, this is not the case. It could be caused by any known Xcode issues or simply because your local configuration is mismatching the one from Bitrise.
A common cause is the fact that CI is starting the tests for your app from a clean configuration with erased content on the simulator, and an empty Xcode build folder. Therefore, it's good to run the tests in Xcode on your local machine after:
- Cleaning the build folder using Product ➞ Clean Build Folder.
- Resetting the simulator using Hardware ➞ Erase All Content and Settings...
If the result is still different from CI, you might want to clean your derived data folder. Cleaning can be done quickly by using Fastlane from the terminal:
fastlane run clear_derived_data
If you don't have Fastlane installed, you can also find the derived data path in the Xcode preferences under Locations.
It's time to access the virtual machine if your tests are still succeeding locally while failing on CI.
Using a virtual machine to solve failing tests
Once you've tried everything to reproduce the same result on your local machine but you still don't manage to succeed, it's time to open up the virtual machine and fix the failing tests there.
Bitrise has excellent support for remote access via SSH or a screen share app. It allows you to do anything in your Xcode project until 10 minutes after the triggered build has finished. Note that this could be quite annoying as you could be in the middle of a debugging session while your connection gets lost. However, it's better than having to figure out the issue from your local configuration without being able to reproduce it!
At WeTransfer, we've been fixing a lot of flaky tests in our suite by making use of the following technique. Combined with making use of the Thread Sanitizer, which is covered later, we've managed to make our test suite a lot more stable on both CI and local machines.
Connecting to your build with remote access
It's best to reference Bitrise's excellent documentation about Remote Access. It explains the details of connecting to a virtual machine.
Opening your project on the virtual machine
Once connected to the virtual machine, it's time to open up the project folder. You can do this by opening the terminal and typing open git as the project path is /Users/vagrant/git/.
Before opening the project, verify within logs of the running build that all setup completes, and the build has started running the tests. It could be that dependencies still need to be installed or that it's not finished with pulling the project.
Run the tests and start debugging
After you've opened up Xcode, you can start running the tests on the virtual machine. Adding a breakpoint for test failures is recommended so you can quickly get into debugging mode for the failing test:
Adding this breakpoint is especially valuable as flaky tests often only occur when running the full suite of tests. Once your test fails and the breakpoint hits, you can start debugging and find the fix for the flaky test. It might be useful to add some logs in your code so you can identify the cause of the issue.
Common Swift related causes for flaky tests
There are often two different types of flaky tests.
If a flaky test is only occurring when running the full suite of tests, it's often related to other not correctly cleaned up tests. Tests are not cleaned up correctly as a result of retain cycles in combination with not making use of the tearDown method to clean up test-related instances. These are also the ones that are hard to find. You can read more about finding memory leaks by making use of the Memory Graph Debugger in the Apple Documentation.
If your test is also failing when you're not running the full suite of tests, its cause is often a race condition. For this, you can make use of the Thread Sanitizer that you can also enable for tests. You can read more about that in my blog post Thread Sanitizer explained: Data Races in Swift.
Conclusion
That's it! Hopefully, you're now able to solve flaky tests more easily by making use of remote access with Bitrise.
Feel free to reach out to me on Twitter if you have any feedback or questions. You can follow my personal blog SwiftLee for more workflow and Swift related blog posts.
Thanks!
