.png)
A mobile testing pyramid is used by mobile product organizations to plan manual and automated mobile tests. The guiding principles of mobile testing pyramids help establish a strategy for unit testing, integration testing, and end-to-end mobile testing.
At the Mobile DevOps Summit, Laurent Sigal, the Co-founder & CTO of Waldo, hosted a session about how to master the mobile testing pyramid. Let’s recap the session.
What is the mobile testing pyramid?
The mobile testing pyramid is a set of guiding principles that help mobile developers strategize testing structures and coverage for mobile apps. It breaks testing into three layers — forming a pyramid. The size of each layer refers to the volume of tests.
At the bottom, on the largest layer of the pyramid, is unit tests. In other words, mobile testing teams should use this as their foundation to unit test most of the codebase. Unit tests help teams evaluate specific parts of the architecture to see the implementation-level details. It helps ensure that blocks are behaving as expected. These tests are easy to run and often run automatically. Although it’s usually the most cost-effective, it can a limited scope.
Above unit tests, there are integration tests for service-level testing. It helps mobile testing teams ensure that each input gives the expected output. Each service-level test is considered a black box — you don’t need to know what’s happening inside; you are simply testing and focusing on the public interface and the integration between the services or components. These tests take slightly longer than unit tests and are harder to put in place, but have a wider scope. Integration tests are often run automatically.
At the top of the pyramid is end-to-end tests. This is for user-interface level testing — to test what the users see when they use your app. This helps build release confidence in what you are deploying. These tests are the most complex and expensive to put together and execute because end-to-end tests often need to be run manually, though E2E tests can be automated in a stable testing environment.
How to properly set up your testing environment to enable end-to-end testing
When you’re automating your testing environment, it’s important to distinguish between automating your app and automating your testing. In order to build automated, scalable end-to-end testing, there are three core principles to follow:
- Automate your app processes
When setting up your app’s infrastructure, you need to consider how you set up your testing environment. You should be able to easily access any part of your app in a simple, repeatable way. In other words, you need to know how your app behaves, even before you open it, and you also need to have a way to get into specific modes of your app in order to test them.
For example, your app has different users that behave differently. As an admin and mobile engineer, you want to see what your users see, and be able to replicate their environment and experience. So, if you want to test the experience of a user who doesn’t log in for 10 days, you need to test a user account for someone who meets those parameters. And thus, need to be able to easily replicate that account and environment for testing.
This is the most important step when creating your E2E tests. Spend most of your time automating your app, as it yields the most return for your efforts. Automating your app infrastructure ensures testers get a repeatable way to test the diverse set of features that you care about. This also makes your manual testers happy.
- Automate your tests
When planning your mobile testing pyramid, remember that proper end-to-end testing automation is based on the user experience, not the code. As Laurent advises, it’s critical to consider the end user's experience when creating your E2E test plan. End-to-end tests should mirror what the users see as closely as possible and not get caught up in the implementation details.
A good rule of thumb to abide by is to set up a testing environment where tests wouldn’t be impacted if you decided to change all the IDs in your app, regardless of the automation you decide to implement.
However, you do need to consider what is happening to your code, and you want to have loosely coupled tests and development. You don’t want to rely on implementation details for testing. There needs to be a one-way communication that allows the test to know what branch and version to test without poking into the internals of the code.
💡 TIP: If you automate your tests before you are able to automate your app, you will likely run into problems. Don’t skip to testing automation before you can predict how your app will behave.
“You really have two separate problems — the automation of your app and the automation of your tests. Making this distinction really helps to put a good testing strategy into place for mobile because it allows you to see that they require very specific needs on both sides,” says Laurent.
- Build a proper E2E testing strategy
End-to-end testing is not meant to be comprehensive tests that cover your entire codebase. That’s why the mobile testing pyramid and testing strategy exist with E2E testing at the top of the pyramid. Only use end-to-end testing to test the essence of your app, as Laurent advises. End-to-end testing is the most expensive type of mobile testing, so it’s important to pre-plan when it comes to E2E test strategies.
Once your mobile testing becomes more sophisticated and robust, you can further develop your E2E testing strategy to include end-to-end testing methods such as:
- Pre-production testing
- Nightly builds
- Smoke testing
Mastering the mobile testing pyramid helps mobile product organizations create better apps, build better user experiences, and launch functional features. A proper mobile testing pyramid helps teams establish test coverage of the code base while balancing costs and time.
