What is software testing?
Whether it is business applications (such as banking) or consumer products (such as automobiles), software plays an integral role in everyday life. Most people have had an experience with software that did not work as expected. Software that does not work correctly can lead to many problems, including loss of money, time, or business reputation, and even injury or death.
Software testing is a way to assess the quality of the software and reduce the risk of software failure in operation. A common misperception of testing is that it only consists of running tests, i.e., executing the software and checking the results.
Software testing is a process that includes many different activities; test execution (including checking of results) is only one of these activities. The test process also includes activities such as test planning, analyzing, designing, and implementing tests, reporting test progress and results, and evaluating the quality of a test object.
Some testing does involve the execution of the component or system being tested; such testing is called dynamic testing. Other testing does not involve the execution of the component or system being tested; such testing is called static testing. So, testing also includes reviewing work products such as requirements, user stories, and source code.Â
Another common misperception of testing is that it focuses entirely on the verification of requirements, user stories, or other specifications. While testing does involve checking whether the system meets specified requirements, it also involves validation, which is checking whether the system will meet user and other stakeholder needs in its operational environment(s). Test activities are organized and carried out differently in different lifecycles.
The objectives of software testing
For any given project, the objectives of testing may include:
- To prevent defects by evaluating work products such as requirements, user stories, design, and code.
- To verify whether all specified requirements have been fulfilled.
- To check whether the test object is complete and validate if it works as the users and other stakeholders expect.
- To build confidence in the level of quality of the test object.
- To find defects and failures thus reducing the level of risk of inadequate software quality.
- To provide sufficient information to stakeholders to allow them to make informed decisions, especially regarding the level of quality of the test object.
- To comply with contractual, legal, or regulatory requirements or standards, and/or to verify the test object’s compliance with such requirements or standards.
Is quality assurance the same as testing?
While people often use the phrase quality assurance (or just QA) to refer to testing, quality assurance and testing are not the same, but they are related. A larger concept, quality management, ties them together.
Quality management includes all activities that direct and control an organization about quality. Among other activities, quality management includes both quality assurance and quality control. Quality assurance is typically focused on adherence to proper processes to provide confidence that the appropriate levels of quality will be achieved. When processes are carried out properly, the work products created by those processes are generally of higher quality, which contributes to defect prevention. In addition, the use of root cause analysis to detect and remove the causes of defects, along with the proper application of the findings of retrospective meetings to improve processes, is important for effective quality assurance. Quality control involves various activities, including test activities, that support the achievement of appropriate levels of quality. Test activities are part of the overall software development or maintenance process. Since quality assurance is concerned with the proper execution of the entire process, quality assurance supports proper testing.
Software development and software testing
- For every development activity, there is a corresponding test activity.
- Each test level has test objectives specific to that level.
- Testers participate in discussions to define and refine requirements and design and are involved in reviewing work products (e.g., requirements, design, user stories, etc.) as soon as drafts are available, test activities should start in the early stages of the lifecycle, adhering to the testing principle of early testing.
Test levels
There are different testing levels for each development activity:
- Unit testing: examines functionality and defects in parts of the application that are accessible and can be tested independently (program modules, objects, classes, functions, etc.).
- Integration/Component testing: designed to test the connection between components as well as interaction with various system components (operating system, equipment, or communication between different systems).
- System testing: checks the system's overall functional and non-functional requirements.
- Acceptance/E2E: a formal testing procedure that ensures the system meets the specifications.
Types of tests
Functional testing
Is a system that involves tests that evaluate functions that the system should perform. Functional requirements may be described in work products such as business requirements specifications, epics, user stories, use cases, or functional specifications, or they may be undocumented. The functions are “what” the system should do.
Non-functional testing
Non-functional testing of a system evaluates characteristics of systems and software such as usability, performance efficiency, or security. Refer to ISO standard (ISO/IEC 25010) for a classification of software product quality characteristics. Non-functional testing is the testing of “how well” the system behaves.
Concepts to consider
User acceptance testing (UAT)
User acceptance testing of the system is typically focused on validating the fitness for use of the system by intended users in a real or simulated operational environment. The main objective is to build confidence that the users can use the system to meet their needs, fulfill requirements, and perform business processes with minimum difficulty, cost, and risk.
Alpha and beta testing
Alpha and beta testing are typically used by developers of commercial off-the-shelf (COTS) software who want to get feedback from potential or existing users, customers, and/or operators before the software product is put on the market.
Alpha testing is performed at the developing organization’s site, not by the development team, but by potential or existing customers, and/or operators or an independent test team.
Beta testing is performed by potential or existing customers, and/or operators at their locations. Beta testing may come after alpha testing or may occur without any preceding alpha testing having occurred.Â
One objective of alpha and beta testing is building confidence among potential or existing customers, and/or operators that they can use the system under everyday conditions, and in the operational environment(s) to achieve their objectives with minimum difficulty, cost, and risk.Â
Another objective may be the detection of defects related to the conditions and environment(s) in which the system will be used, especially when those conditions and environment(s) are difficult to replicate by the development team.
Exploratory testing
In exploratory testing, informal tests are designed, executed, logged, and evaluated dynamically during test execution. The test results are used to learn more about the component or system, and to create tests for the areas that may need more testing.
Regression testing
Regression testing is a type of software testing that verifies an application continues to work as intended after any code revisions, updates, or optimizations. As the application continues to evolve by adding new features, the team must perform regression testing to evaluate that the existing features work as expected and that there are no bugs introduced with the new feature(s).
This subset selection results from a combination of several logical criteria as follows:
- The cases are derived from identifying the potentially affected feature(s) due to the change(s).
- Business-critical cases.
- The most commonly used paths.
Overview of software testing jobs, their characteristics, and their roles
Now let me explain what I see as the differences between these roles:
1- Quality Assurance Engineer (QAE)
Description:
- Is familiar with the bug tracking, ticketing, and testing process. Introducing the standards of quality
- Usually does not have technical experience but a few of them has the basics of software engineering and programming and maybe do some SQL statements.
- Focuses more on manual testing.Â
- Has experience the test management tools.
- Has an eye for detail and proficiency in technical documentation with a solid ability to generate valuable test scenarios from agile requirements.
- Has experience developing test strategies for complex projects, test plans, and test cases with out-of-the-box thinking to create edge case scenarios.Â
- Defines QA activities and principles for your testing activities based on best practices and guidelines such as; Test planning, Risk Assessment, Test case management, etc.
- Usually works with the PM, TPM, and product designers.
Focus:
Preventing bugs through different manual processes.
2- Quality Control Engineer (QCE)
Description:
- Makes sure that the product corresponds to the requirements and specs before it is related
- Is familiar with operations such as infrastructure, servers, platforms, etc.
- Does test automation at several levels including API, UI, and Mobile
- Is the best candidates for the test automation role because they have the required skills.Â
- Works closely with developers, automation engineers, product owners, and product designers on sharing best practices and improvements within the squad and the QA team.
- Understands the project execution plan and coordinate Quality Control procedures at appropriate stages of the work and in line with the project schedule.
- Needs to carry out all product-related testing.
Focus:
​​Verification that the app or the software working as expected.Â
3- ​​Test Automation Engineer (TAE)
Description:Â
- Detects and solves the software errors via test automation scripts.
- Has skills in Security, performance testing, and integration in the CI/CD.Â
- Scales the UI tests, stabilizes the test suites, and understands the business to be able to convert the manual tests to automated test cases.
- Has good programming, code review, and development skills.
- Designs, develops, and maintains the mobile test automation frameworks.
- Contributes to the increase in test automation coverage for iOS and Android
- Usually works closely with the development teams and engineering family.
- Coachs and mentors other team members with innovative ideas to enhance our test processes.
Focus:
Detection of the bugs with every code change at the early stages.Â
4- Software Engineer in Test (SDET)
Description:
- Has strong hands-on skills and must be able to do advanced automated tasks.Â
- Is well-versed in software development.
- Is able to create orchestration platforms.Â
- Works as a test infrastructure engineer to provide custom solutions to the TAEs to be able to scale the tests and increase the test coverage.
- Knows about CI/CD, DevOps, Cloud Computing, and monitoring tools/services.
- Maintains configurations and workflows in the CI/CD pipeline.
- Has excellent problem solving and algorithmic thinking skills
- Has a strong foundation in test automation design patterns, best practices, and approaches.
- Has strong troubleshooting and debugging skills.
- Works closely with the DevOps, and the infrastructure teams.
Focus:
Developing and helping test engineers and developers with out-of-the-box solutions.
Note
Sometimes, in this market, there can be confusion between job titles. So when you find that a QA Engineer is also a test automation engineer, or that a QC Engineer is responsible for both manual and automated testing, pay attention to the job description and responsibilities.
7 frequently asked questions about Bitrise and the test engineers
In this article, you learned about the differences between SDET, QAE, QCE, and TAE. For more information, here are the seven most frequently asked question about test engineers and Bitrise, as well as some additional resources to reference.
Q1: As a test automation engineer, is Bitrise support different mobile test frameworks?
Answer 1: Yes, Bitrise supports you to run different mobile test frameworks:
- Espresso for Android
- XCUITest for iOS
- Appium for both (Cross-Platform)
- Detox for React Native
- Flutter Test
Related Resources
- What is unit testing in mobile development?
- Getting started with testing Jetpack Compose
- React Native E2E UI testing with Detox and Bitrise
- Android UI testing with AWS Device Farm and Bitrise
- Getting started with Appium & Bitrise for Android apps
Q2: Automated tests always need to be run on different devices virtual (Emulators and Simulators) or physical devices, is Bitrise support us to use all of this device management?
Answer 2: Yes, actually bitrise has different options to be able to run the automated tests:
- Execute tests on Bitrise virtual machines through emulators and simulators.
- Use most of the cloud testing tools provided by 3rd parties. For instance, we support:‍
- SauceLabs‍
- BrowserStack‍
- TestingBot‍
- Genymotion‍
- AWS Device Farm‍
- Kobiton‍
- Perfecto‍
- Sofy.ai‍
- Waldo
Related Resources
Q3: If we need to run automated security tests or checks as part of our DevOps cycle, is Bitrise integrated with any security testing tools?
Answer 3: Yes, we have different security testing tools including both static and dynamic tools to assist your team in applying security testing early on as a shift-left testing approach, as well as identifying app vulnerabilities.
We are supporting:
Related Resources
- DevSecOps: injecting security into the mobile CI/CD pipeline | Bitrise
- Mobile app security made easy with AppSweep's Verified Step | Bitrise
- Guarding your mobile apps: Introducing the DexProtector Step | Bitrise
- Data Theorem's Verified Step has arrived at the Step Library | Bitrise
- Detect app vulnerabilities with the Oversecured Step | Bitrise
Q4: One of the important things we need are the test reports to be able to show the output of the automated test suite and the coverage or the test statuses, is there any way on Bitrise to do this?
Answer 4: Yes, Bitrise has an add-on called Test Report which is configured with our Test Steps. It supports the following steps:
But also if you need to integrate any other testing step you can customize the Test Report to show your test results
Q5: In Bitrise, is there a tool or option for monitoring test performance, so that we can know which tests are stuck or slowing down our CI workflows?
Answer 5: Yes, Bitrise Build Insights allows you to view build performance, build stability, and credit consumption. Additionally, Build Insights Pro allows you to:
- Measure and react to changes in performance more quickly.
- Monitor, predict, and optimize usage throughout the year.
- Spot those anomalous tests so you can drill down on exact issues.
- Address flakiness and stop wasting build time.
- Compare up-to-date performance snapshots with historical trends to quickly spot and resolve issues.
Related Resources
Q6: Is there an Integration Step to run Appium Server on Bitrise?
Answer 6: Yes, you can use the AVD Manager to spin up the Android Emulator on Bitrise VMs, while the iOS Simulators will start automatically once the tests start running.
Related Resources
Q7: I'm a QA Engineer and test our mobile apps manually before each release; how will Bitrise help me?
Answer 7: You have different alternatives, such as Bitrise, which can help you release your mobile apps faster and more frequently for alpha and beta testing, to be able to download and install the app on your device and begin testing immediately. We can also help you upload the mobile apps to 3rd party testing tools, such as SauceLabs, BrowserStack, and Genymotion, to install and verify the apps through live and interaction testing.
Related Resources
- Browserstack App Live Upload - Integrations
- Upload application to Kobiton Step
- (Android) Deploy to Google Play (Internal, Alpha, Beta, Production) - Bitrise Docs
- Device testing for iOS - Bitrise Docs
