Xamarin CI with Bitrise Tutorial Part 2 - Testing

This tutorial helps you understand how to test your apps on Bitrise, from running basic NUnit tests, through UI Testing and Calabash, to using the Xamarin Test Cloud.

We recently released support for Xamarin iOS and Android apps on Bitrise. If you are just about moving to Bitrise with your Xamarin apps, check out the first part of this tutorial about how to get started.

This tutorial helps you understand how to test your apps on Bitrise, from running basic NUnit tests, through UI Testing and Calabash, to using the Xamarin Test Cloud.

Last part of the series is coming up next week about how and where to deploy your Xamarin applications using Bitrise.

Restore packages

Before starting to build and test your apps, you will have to restore your packages your project is depending on. To help you achieve this, we implemented a NuGet restore and a Xamarin Components restore step.

There is not much to do here, Bitrise already recognized your solution file's path during your app's setup and saved it to an environment variable. This $BITRISE_PROJECT_PATH variable will be the default input value for these steps.

Make sure to place every package restore step after the Git clone repository step and before any kind of build steps!

Custom NuGet packages

If you have custom NuGet packages to restore, add a generic Script step to your workflow and paste this line of bash into it.

Make sure to place this step before NuGet restore.

NUnit tests

If your project contains NUnit tests, the only thing you have to do is to add the NUnit Runner step to your app's workflow.

Its default inputs are pre-set, but can be modified. The paths to your solution file, your configuration and the platform of your NUnit tests to run. You are able to add extra options through the NUnit command options.

You can always check out what inputs, outputs and logic a Bitrise step has. There's a button pointing to the source of the selected step in the workflow editor of your apps. Check out all the steps we have on our integrations page, or their implementation at the Bitrise StepLib repository.

UI tests

For adding UI tests, you will have to modify your project a bit. It's a temporary solution until we figure out a better way. Let us know if you have a suggestion!

Use the NUnit runner step for running .NET UI tests.

iOS

In case of an iOS app, you'll have to expose the app's bundle ID and the simulator's UDID in your app configuration. We are going to export the simulator UDID for you as environment variable, so it should look like the following.

If you finished updating your project, just add the Xamarin iOS Test step to your workflow to make it part of your build process.

Android

If you'd like to UI test Android apps, you'll have to update your configuration code to contain the emulator and bundle IDs.

It's strongly recommended to add some extra waiting time to deal with Android emulator launching issues.

Now you just have to add the Xamarin Android Test step to your workflow.

Calabash

If you'd like to run your native tests with Calabash on Bitrise, look for the Calabash steps when adding a new step to your workflow. You'll be able to run the tests on our VMs, and/or in the Xamarin Test Cloud.

iOS

For running iOS UI tests with Calabash, add the Calabash iOS UI test step to your workflow. Specify the OS version of the Simulator you'd like to run your tests on, then you are ready to roll.

Android

To run your Android UI tests, you have to add the Calabash Android UI test step and specify the path to your apk. You can find it relatively via $BITRISE_DEPLOY_DIR.

Xamarin Test Cloud

Using the Test Cloud lets you run your tests on real devices. You can initiate XTC tests form Bitrise easily. You just have to pull in the Xamarin Test Cloud for iOS or Xamarin Test Cloud for Android step to get started. The following information is necessary to fill in:

  • User email - Your Xamarin user's email address
  • API key - Your Xamarin Test Cloud API key
  • Device selection id - The device id you would like to use

It's possible to pass other parameters to the XTC steps, please check out the step's configuration in your app's workflow editor for more details.

Submit Calabash tests to Xamarin Test Cloud

For submitting your native tests with Calabash to the XTC, just add the proper steps Submit Calabash iOS tests to XTC or Submit Calabash Android tests to XTC to your workflow. There you will have to provide your email address, your API key and the desired device id as described above.

Test results

You are going to have generated output variables after running any of the listed test steps, BITRISE_XAMARIN_TEST_RESULT, and BITRISE_XAMARIN_TEST_FULL_RESULTS_TEXT. The first one will simply tell you whether the tests succeeded, or not. The full text results will contain all the tests' logs. You are able to use these values in steps followed by your test steps, like Send a Slack message for example.

Anything else?

You can always share your questions, problems, compliments, or virtual hugs with us on Twitter, by email, or on our public Slack.

If you are attending the Xamarin EVOLVE conference in April, make sure to come by to our booth for a chat!

See you in part three about deploying Xamarin apps from Bitrise and happy building in the meantime! 🤖

No items found.
The Mobile DevOps Newsletter

Explore more topics

App Development

Learn how to optimize your mobile app deployment processes for iOS, Android, Flutter, ReactNative, and more

Bitrise & Community

Check out the latest from Bitrise and the community. Learn about the upcoming mobile events, employee spotlights, women in tech, and more

Mobile App Releases

Learn how to release faster, better apps on the App Store, Google Play Store, Huawei AppGallery, and other app stores

Mobile DevOps

Learn Mobile DevOps best practices such as DevOps for iOS, Android, and industry-specific DevOps tips for mobile engineers

Mobile Testing & Security

Learn how to optimize mobile testing and security — from automated security checks to robust mobile testing and more.

Product Updates

Check out the latest product updates from Bitrise — Build Insights updates, product news, and more.

The Mobile DevOps Newsletter

Join 1000s of your peers. Sign up to receive Mobile DevOps tips, news, and best practice guides once every two weeks.