Continuous testing for mobile apps at scale with Sauce Labs and Bitrise

Today, mobile apps are becoming more complex, modules are becoming more diverse, and mobile teams are growing and becoming large engineering teams to serve millions of users around the world. 

Gergely, the author of Building Mobile Apps at Scale: 39 Engineering Challenges book said that there are different challenges due to large engineering teams, the nature of mobile apps, app complexity, and cross-platform and non-functional aspects.

And one of these challenges is Device and OS fragmentation.

Device and OS fragmentation

Device model and OS fragmentation is an everyday problem for both iOS and Android apps. Device fragmentation and hardware-related bugs are familiar pain points on Android. It is crucial to execute mobile application testing more often due to OS version upgrades and the launch of new devices.

Mobile device and OS fragmentation refer to the diverse range of hardware and software configurations used on mobile devices. This can include different device manufacturers, models, screen sizes, connection types, power consumption/battery life, and resolutions, as well as different versions of the same OS.

Fragmentation can make it challenging for developers to create and maintain applications that work across all devices and OS versions. For instance, an app that works perfectly on one device might have performance issues or display problems on another device with a different configuration.

Teams need to test their mobile apps early, frequently on different devices and OS versions to increase the test coverage and make sure that our mobile apps behave correctly as expected on different devices. 

As a result, the continuous testing concept was born.

Continuous testing

Continuous testing is the practice of testing software as part of the development process, rather than waiting until the end of the development cycle to perform testing. This can be especially important for mobile apps, which are often used by a large number of users and need to be of high quality.

There are a number of ways to perform continuous testing for mobile apps at scale:

Automated testing: Automated testing involves using software to perform tests on the app without the need for manual intervention. This can be especially useful for running large numbers of tests quickly and consistently.

Cloud-based testing: Cloud-based testing allows you to run tests on a wide variety of devices and operating systems without the need to physically own or maintain the devices. This can help you ensure that your app works correctly on a variety of different devices.

Continuous integration: Continuous integration involves automatically building and testing code changes as they are made, rather than waiting for a large batch of changes to be made before testing. This can help catch issues early on in the development process.

By implementing these practices, you can help ensure that your mobile app is of high quality and performs well on a variety of different devices and operating systems.

Sauce Labs

Sauce Labs is a cloud-based testing platform that allows you to run automated and manual tests on a wide variety of virtual and real mobile devices. It provides access to a range of tools and features to help you test the functionality, performance, and compatibility of your web and mobile applications.

Sauce Labs helps mobile engineers to:

• Leverage a wide range of programming languages, including Java, Javascript, Python, and Ruby.
• Use different types of mobile test automation frameworks such as Espresso, XCUITest, and Appium.
• Test applications on a variety of different operating systems and devices, including Android, iOS, Windows, and macOS.
• Monitor and debug tests in real time.
• Integrate with popular Continuous Integration tools such as Bitrise, Jenkins, GitHub Actions, and more. 
• Work in a user-friendly interface for managing tests.
• Perform continuous testing and ensure the quality of applications for developers and QA teams. It reduces the time and effort required to test applications manually and can help catch issues early on in the development process.

Using saucectl CLI

saucectl CLI orchestrates the relationship between your tests in your framework, and the rich parallelization, test history filtering, and analytics of Sauce Labs. saucectl performs the underlying business logic to access the tests in your existing framework, runs them (either in the Sauce Labs Cloud or locally in a Docker image), then securely transmits the test assets to the Sauce Labs platform, where you can review, share, and evaluate your test outcomes at scale.

The following provides a high-level overview of the saucectl CLI architecture:

1. The saucectl command sends the test payload (mobile app, test app, or project files) to the app storage.
2. Saucectl calsl the appropriate device cloud in accordance with the capabilities of the underlying test framework (e.g., XCUITest or Espresso).
3. You request a device from the device cloud, and the cloud retrieves and runs the test payload from the app storage.

The integration between Sauce Labs and Bitrise 

Bitrise is a Mobile DevOps platform that helps mobile developers deliver secure mobile apps faster with good quality. You can integrate any 3rd party tools with the Bitrise Mobile DevOps platform if you want to customize your workflows by, for instance, adding your mobile tests from Sauce Labs. 

Sounds interesting, let’s get started!

The pre-requests 

The following requirements are required in order to run your UI tests with Sauce Labs and Bitrise:

• A Sauce Labs account (Log in or sign up for a free trial license)
• Your Sauce Labs Username and Access Key
• Bitrise free account
• Android or iOS application configured on Bitrise, if you don’t have one you can follow the instructions here.
• Your Android and test app files for real devices and/or emulators.
• Your iOS and test app files for real devices.
• A config.yml file (in the <root>/.sauce directory) that is preconfigured to run the tests, which is also included in the repo.

If you’d like to run your tests from your local machine, you can install and use saucectl CLI and this will be the same approach on Bitrise. More information can be found here.

saucectl can execute both Espresso and XCUITest tests, so the setup steps are the same regardless of which framework you are using.

📝NOTE: Espresso requires saucectl version 0.36.0 or later and XCUITest requires saucectl version 0.44.0 or later.

XCUITest with iOS

Here’s an XCUITest and iOS sample config.yml file

apiVersion: v1alpha
kind: xcuitest
sauce:
  region: eu-central-1
  concurrency: 1
  metadata:
    tags:
      - e2e
      - release team
      - other tag
    build: $BITRISE_BUILD_NUMBER

xcuitest:
  app: $BITRISE_IPA_PATH
  testApp: ./Debug-iphoneos/DailyCheckUITests-Runner.app

suites:
  - name: "Example 1 - Run all tests"
    devices:
      # Run on whatever iPhone device is available
      - name: "iPhone.*"
        platformVersion: "15"

artifacts:
  download:
    when: always
    match:
      - "junit.xml"
    directory: ./artifacts/

You can find more details about the configuration for XCUITest here. 

Let’s assume that we have the following iOS workflow to build the iOS and test apps.

As a first step, you need to add the Sauce Labs credentials as secret environment variables. 

Then, add a step to unzip the test bundle because saucectl should receive it as the app as mentioned in the above example. 

#!/usr/bin/env bash
set -e
set -o pipefail
set -x

unzip $BITRISE_TEST_BUNDLE_ZIP_PATH

‍

After that, we need to add another Script step to install, authorize and run the saucectl CLI

#!/usr/bin/env bash
set -e
set -o pipefail
# debug log
set -x

npm install -g saucectl
saucectl -v

saucectl configure --username $SAUCE_USERNAME --accessKey $SAUCE_ACCESS_KEY

saucectl run --config .sauce/config.yml

Finally, we will add an Export test results to the Test Reports add-on Step to download the test results from Sauce Labs and send them to the Test Report add-on. 

The final workflow will be like the following image:

‍📝NOTE: Run automated tests on Sauce iOS simulators is not supported at this time.

Run the workflow and check the build and test results as displayed in the following images. 

Open the Test Report Add-on from the Artifacts tab to double-check that the test results are downloaded and displayed correctly in the report.

Open Sauce Labs Dashboard and check the detailed test results. 

💡TIP: If you’d like to create an IPA for your iOS app instead of .app you can use these steps to do that.

Espresso with Android

The Here’s Espresso and Android config.yml file

apiVersion: v1alpha
kind: espresso
defaults: {}
showConsoleLog: false
sauce:
  region: eu-central-1
  concurrency: 2
espresso:
  app: $BITRISE_APK_PATH
  testApp: $BITRISE_TEST_APK_PATH
suites:
- name: espresso - Google Pixel .* - Android GoogleAPI Emulator
  devices:
  - name: Google Pixel .*
  emulators:
  - name: Android GoogleAPI Emulator
    platformVersions:
    - "12.0"
artifacts:
  download:
    match:
    - '*'
    when: always
    directory: artifacts

Let’s assume that we have the following Android workflow to build the Android and test APKs.

We will do the same steps we did with the iOS Workflows add Sauce Labs credentials as secrets and add a Script step to install, authorize and run the tests.

The final workflow will be like the following image:

You can find more details about the configuration for Espresso here. 

Run the workflow and check the build and test results as displayed in the following images.

‍

You can also run your tests on Android real devices by adding the following section to config.yml:

devices:
   - id: Samsung_Galaxy_S9_free
     platformVersion: "10"

The Here’s Espresso and Android config.yml file

apiVersion: v1alpha
kind: espresso
defaults: {}
showConsoleLog: false
sauce:
  region: eu-central-1
  concurrency: 2
espresso:
  app: $BITRISE_APK_PATH
  testApp: $BITRISE_TEST_APK_PATH
suites:
- name: Espresso Android UI tests
  emulators:
  - name: Android GoogleAPI Emulator
    platformVersions:
    - "12.0"
  devices:
   - id: Samsung_Galaxy_S9_free
     platformVersion: "10"
artifacts:
  download:
    match:
    - '*'
    when: always
    directory: artifacts

Run the workflow and check the build and test results.

💡TIP: It is possible to use the screenshot capture functionality of Espresso only on real devices to take screenshots of errors to support the visual verification of your debugging process, more details can be found here. 

Congratulations you have successfully integrated Sauce Labs with Bitrise!

What’s next?

Maybe in your company, you need to use Sauce Connect to run the local tests on apps located behind a firewall. In this case, you can use the Sauce Connect Proxy tunnel in minutes using the steps here.

In case you don't want to wait for Sauce Labs' results, you can use the -- async option with sauceCTL like the following command:

saucectl run --config .sauce/config.yml -- async 

Additionally, if you’d like to use concurrencies to sets the maximum number of suites to execute at the same time or set the number of times to retry a failed suite, or change the report format all of these options can be configured in the config.yml file.

After integrating Sauce Labs with Bitrise, you can now run the Mobile UI tests on every code change, pull request, tag, or during the nightly builds easily and frequently. 

Thank you for reading and happy testing!