Scale up your iOS automated UI tests

It was a pleasure to participate in the iOS Singapore Conference 2022! My talk was about how we can scale up our iOS automated UI tests using fastlane and fastlane plugin. I put together this article as a recap — read on to learn more!

For a quick summary, here are the key takeaways of my talk:

Source: https://twitter.com/iosconfsg/status/1484058934541713410

Automated UI Testing in iOS

Mobile test automation is becoming a vital part of the Mobile DevOps cycle. Without it, we can’t deliver mobile apps to our customers. In the ever-growing complexity of technology, automated mobile testing is a solution to produce high-quality, robust, and reliable apps, and achieve cost optimization — even under massive competitive pressure. 

Writing UI tests is not just ‘nice to have’ anymore. In different companies, the responsibility of mobile test automation is shared between mobile developers and test engineers.

What are the iOS test frameworks?

Nowadays, there are different frameworks to write UI tests for iOS apps such as XCUITest, EarlGrey, and Appium. 

Let’s see some of the main differences between them:

• Appium: is an open-source framework that supports many programming languages and testing tools. It is possible to automate UI testing for Android and iOS apps of all kinds – native, web, and hybrid. Appium uses the XCTest UI testing feature to automate iOS test cases.
  

• EarlGrey: is a native iOS automation framework developed by Google. EarlGrey integrates into the app’s code to speed up testing execution. Test engineers can write tests both in Swift and Objective-C directly in XCode and execute test runs from there.
  

• XCUITest: is a UI testing framework developed by Apple. XCTest runs the unit, performance, and UI tests written in Swift or Objective-C. One of the advantages of using XCUITest is the collaboration between developers. 
  

Once iOS developers and test engineers use the same language to write code, collaboration becomes easier, which helps them better understand the app’s functionality and leads to higher quality apps. Moreover, it’s already pre-built into XCode. Basically, there is no need to install or download anything else if you already have XCode.

Let’s assume that we are using XCTest and XCUITest in our team and we built and designed a lot of test cases to cover our app functionality, but after that, we figured out that we have the following challenges or problems when running these test cases:

• Our business is growing and we are adding more features every one or two weeks. 
• The team is scaling and becoming a large engineering team.
• We have hundreds/thousands of UI tests that need to run most of them in our CI server.
• The build time is affected on the CI server because UI tests are usually slow and take a long time. 
• Our UI tests are flaky because our CI server is not suitable for our mobile app and the simulators 
  

Most organizations or teams struggle with being able to run multiple tests in parallel, which is another important challenge. 

After we addressed most of the issues, the question now is:

How to scale up the iOS UI tests?

There are different solutions to scale up our UI tests on iOS, for example: 

• Xcode Parallel Test Execution 
• fastlane & mutliscan plugin
• Firebase Test Lab and Flank
• Bluepill
  

This talk focused on one of these solutions, which is the fastlane and fastlane plugin. 

What’s fastlane?

fastlane is an open-source platform that lets you automate every aspect of your development and release workflows. It’s the easiest way to build and release mobile apps. 

Fastlane's automation tasks can save you hours, especially when you push a new release to the store or to beta testing.

It offers easy setup, runs on your machine, and integrates with all major CI systems, such as Bitrise, and more.

Getting started with fastlane

To install and configure fastlane on your local machine you can use Homebrew with macOS or Bundler. But before that, you should install and configure Ruby on your machine. 

Then you can initialize fastlane into your iOS app by command:

After that, you will have a fastlane folder with a Fastfile in it. You can start adding lanes for different tasks like running unit tests, UI tests, checking code style, deploying your app, and more. 

Run UI tests in parallel

Now we need to install a fastlane plugin to be able to run our iOS UI tests, inside the project and in the command line add the following command: 

After the installation is finished, we can open the Pluginfile. We will find our plugin listed there like this:

After that, in the Fastfile, we can add the multiscan. Multi_scan is one of the most popular actions of the test_center plugin to run your tests in parallel with multiple retries. They will finish faster and only the truly failing tests will be reported as failing.

In addition to the parameters supported by scan, multiscan also supports the following parameters:

• Try_count: The number of times to retry running tests via scan
• Batch_count: The number of test batches to run through the scan. Can be combined with :try_count.
• Batches: The explicit batches (an Array of Arrays of test identifiers) to run either serially, or each batch on a simulator in parallel if parallel_testrun_count is given.
• Retry_test_runner_failures: Set to true If you want to treat build failures during testing, like 'Test runner exited before starting test execution', as 'all tests failed'.
• Collate_reports: Whether or not to collate the reports generated by multiple retries, batches, and parallel test runs.
  

After we configured the lane inside the Fastfile, we can now run the UI tests from the command line using the following command: 

Fastlane will run the build and you will notice that there are three clone devices, which means we split our test suite into subsets and run them on three different devices at once.

You will notice that tests will run on headless devices and at the end, all the devices will be deleted.

The test results will be saved in the test_output folder inside the fastlane folder and you can find two different reports (html and JUnit).

fastlane and Bitrise

Bitrise helps you build and operate better mobile apps, faster. Through Bitrise, developers automate the processes required for integrating, testing, and deploying your mobile applications easily and frequently. 

At Bitrise, we are helping you build CI/CD pipelines and customize your workflows by using our out-of-the-box Steps, which save you time and money.

Bitrise has over 330 Steps for all mobile platforms and all of them are open-source. A Bitrise Step is a specific task: for example, Swiftlint to Run Swiftlint on the project and fastlane Step to run your fastlane lane. 

Adding the Step and the lane name to the workflow is what we need to do to run UI tests in fastlane.That’s it: you can use different fastlane steps for each lane in your Fastfile such as Lint, Test, or Deploy. 

That’s it: you can use different fastlane steps for each lane in your Fastfile such as Lint, Test, or Deploy.

And we can customize the Test Report Add-on to show the test results by using the following shell script: 

After that, we will find the test results like here:

Additionally, we can also optimize the builds by running parallel builds or fan-out the tests in different sub-workflows. You only need to create sub-workflows and run them at the same time with the Bitrise Start Build Step and configure the workflows’s names with it.

Finally, If you missed the iOS SG Conf 2022, you can watch our session here. 

Conclusion

• Scaling up the UI tests is required when our business or app is scaling and needs to run hundreds of tests before releasing our mobile apps.
• Test Automation is a vital part of Mobile DevOps.
• CI/CD is the coordination process.
• Testing is a shared responsibility and a whole team approach.

Resources

• https://github.com/moatazeldebsy/UpcomingMovies
• Test Center Plugin
• Fan-out parallel builds and tests on Bitrise
• Hands-on tips for decreasing the duration of iOS UI tests via fastlane and Bitrise
• GitHub - Flank/flank: Massively parallel Android and iOS test runner for Firebase Test Lab
• Bluepill is a reliable iOS testing tool that runs UI tests using multiple simulators on a single machine