In engineering teams, many test automation projects fail — not because of a lack of interest or commitment from the team but due to a lack of proper planning. Many teams jump into test automation without understanding and deciding on their ‘why.’ The result is that these teams end up with failed automation projects where they were unprepared for the issues that arose and unsatisfied with their return on investment.
To set yourself up for success with test automation, you need to nail down a test automation strategy. A test automation strategy is a set of initial considerations and decisions you and your team will need to make to properly inform your overall approach to automated testing and help you set realistic expectations.
In a previous episode of our podcast, Mobile DevOps is a thing!, Angie Jones, creator of Test Automation University and former test automation engineer at Twitter, shared her insights on four crucial factors to consider when creating a test automation strategy.
1. What is the goal of test automation?
Before you implement any form of test automation, Angie advises that teams clarify what their test automation goals are. What are you trying to accomplish with test automation?
Ideally, this goal should be both direction- and metric-oriented so it can be tracked. For example, your goal can be:
- Direction: To facilitate the release of software without any critical bugs to create a better user experience for our end users
- Metric: To achieve 99.8% annual app uptime
It’s necessary that every key stakeholder (engineers, product managers, testers, and so on) discuss and commit to this goal. This will help them understand what’s needed to accomplish the goal and what role they can play to ensure its realization. Test automation is more successful when there is collaboration between different team members.
Once you’ve all agreed on and defined your test automation goal, it will drive every other test automation–related decision you’ll make, such as what tools to use, which test to automate, and who to involve.
2. Who should be in charge of the test automation initiative?
“Make no mistake, test automation is a full software development project in and out of itself. It takes a considerable amount of time and skill to pull off. If you treat it as something that people can work on when they have the time, it will most likely fail.” – Angie Jones
You’ll need someone to take ownership of your test automation processes. Most companies assign these tasks to either their developers or existing manual testers, but this presents some challenges. Angie says the best option is to hire dedicated automation engineers.
Embarking on a test automation project means you’ll need to have someone who will:
- Write the initial scripts to automate the test suite
- Fix, or be responsible for knowing when and why a test fails
- Update the test script when test scenario behavior changes
- Add new features to the automation suite as required
Let’s say you assign these tasks to your developers. But, they’re already busy writing code for new features or fixing bugs. Will they be able to devote enough time to the test automation initiatives? Yes, your developers know how to code, but how familiar are they with test automation?
On the other hand, let’s say you assign the tasks to your existing manual testers because they’re better at testing anyway. Do they also have test automation skills? For either scenario, you also need to take into account the learning curve, time commitment, and willingness to do the work.
The truth is, unless you’re able to relieve your developers or testers of their current tasks so they can have time to focus on the automation initiative, they may not be the best option.
3. What test scenarios should you automate?
Teams often use test automation for scenarios that validate the overall stability and functionality of features. However, not every feature should be or needs to be automated because test automation comes with a cost. More tests mean more maintenance, more noise, and more time commitment to tune and debug them.
There is no hard-and-fast rule for automating tests for apps, as each project is different. However, the tests you automate should be in line with your automation goal and should be for features of your app which are most critical and will give a high ROI.
In her talk ‘Which tests should we automate?’, Angie shared a framework on how to discover which features are critical for your app to help you decide what to prioritize.
For each scenario or feature, she recommends pondering the following metrics:
- What is your gut feeling about this feature or scenario? Do you think it should be automated? (G)
- What is the risk associated with the feature or scenario? What is its frequency of use by customers, and if broken, what is its impact on customers? (R)
- What is the value of this scenario? Does it provide new info, and how quickly would it be fixed if it breaks? (V)
- Is automating this scenario or feature cost efficient? How quickly can it be scripted, and how easy will it be to script it? (C)
- What is the volume of historical failures for this test or related areas? (H)
Again, all key stakeholders should participate in this activity. For example, based on the metrics highlighted above, here is a snapshot from her slides of what a hypothetical test consideration sheet for Twitter profile features would look like:
Each metric is graded on a scale of 0–20, which will eventually be added to get a total score. A score of 67–100 means you should automate, 34–66 means you should possibly automate if you can afford to, and 0–33 means you should forfeit automating that feature.
4. Which test automation tools should you use?
Not all test automation tools are created equal. Angie mentioned that one of the factors that should guide the choice of test automation tools for your mobile app project is who will be using them.
If the test automation will be championed by non-programmers, codeless test automation tools for mobile apps that will allow your testers to record and then execute their scenarios may be a better option. Otherwise, if it will be led by developers or automation engineers, a coded solution or a custom test framework may be a better option because it provides more control and flexibility.
Another factor to consider is the platforms you build for (i.e., Android or iOS). If you build separate native apps for both operating systems, you’ll need a test automation suite that can run on both iOS and Android. You may also want to choose a tool or library, such as Appium, that will allow you to write the test automation once and have it run against multiple mobile OS to save time.
Other factors to consider are:
- Your budget
- The right balance between the ease of use and advanced features
- Customer support
- Availability of training resources, good documentation, and community
- Ease of Integration with your CI/CD and project management software
Supercharge your test automation with Bitrise
Continuous integration is as crucial to test automation as beams are to bridges. Along with a suitable test automation tool, you’ll also need the right CI/CD tool — one built specifically for mobile development — to be efficient with your test automation efforts. That’s where Bitrise can help.
Bitrise offers seamless integration with most of the popular test automation tools and frameworks for mobile, such as Perfecto, TestProject, and Applitools, to mention a few. It also features over 350 steps critical to mobile development workflows to help you move faster and be more productive.
See how Bitrise has helped teams just like yours build better continuous delivery pipelines fueled by continuous integration and continuous testing. Request a demo today to learn how Bitrise can help you build better mobile apps.
.jpg)
.jpg)
