Let's talk about everyone's favorite part of mobile development: watching your tests fail just before you head home on a Friday.
A test fails, you dig through logs across several different tools, identify what went wrong (maybe), fix it (hopefully), and then push again, only to discover it was a flaky test all along… Meanwhile, your entire team is blocked, the release train is derailed, and that "quick fix" just ate two hours of context-switching between GitHub, your CI dashboard, and that one Slack thread where someone mentioned seeing this failure before.
Here's the uncomfortable truth: 32% of developers waste up to 10 hours a week fixing bugs instead of shipping features. Another 22% lose up to 20 hours, according to recent research. For mobile teams, where a single flaky test can block an entire release, those hours add up fast.
The overhauled Bitrise testing experience changes that.
Five ways we're making debugging failed tests less painful
Here's what's shipping:
- PR comments bring test results directly to your code review so you can understand and triage build failures without ever switching out of your repo
- Automatic test retries distinguish between real failures and flaky tests without manual intervention
- Unified test reporting consolidates absolutely everything you need to diagnose test failures, including artifacts, into a single place
- Flaky test detection automatically identifies and flags problematic tests across builds
- Test quarantine (Enterprise plans only) lets you isolate flaky tests until they can be fixed
Let's walk through how these work together.
So your PR just failed
You pushed code. Bitrise ran your tests. Something failed.
But instead of the usual fire drill, here's what happens now:
The feedback comes to you (not the other way around)
You don't need to leave your repository to figure out what broke. Test failure information and details are posted directly into your pull request comments. Zero context switching needed.
A new PR comment from Bitrise now clearly shows:
- Build summary with pass/fail status
- Specific step failures with error snippets
- Failed test details with links to full logs
- Flaky test detection with rerun information
Here's what it looks like in practice:
✅ Green build
Simple one-line comment with a link to the build.
Additional details available when expanded.
Failure with build summary (expanded)
You can select exactly what gets posted in the comment from your Bitrise project settings.
Each item is enabled by default, but you can customise based on what your team needs.
💡 Pro tip: Single comment mode keeps your PR tidy by updating one comment with the latest build results. Need to track what failed across multiple runs? Switch to multiple comment mode for a full history.
Setting this up takes at most 30 seconds: toggle PR comments in your project settings, select which information to include, choose your comment mode, and you’re done. This feature integrates with GitHub, GitLab, Bitbucket.
PR comments allow your entire team to triage failures without breaking their flow. Junior devs can see at a glance if they actually broke something or just triggered a known flaky test; senior devs doing code review get the full context without tab-switching.
But sometimes you need to dig deeper. One click on that link in the comment and...
Everything you need, nothing you don't: the new test report
The new testing tab brings together all your test results, logs, and artifacts in one view.
Failed tests front and center
The moment you land, you see what matters: failed tests.
- Full failure output including exactly where the test broke
- Execution time and retry attempts
and, drumroll please…
👉 Connected artifacts 👈
We’re really excited about this one. Every screenshot, video, and log file is now connected directly to the test that generated it, so you can stop playing detective work and see exactly what broke, instantly.
The straightforward availability of test outputs and artifacts linked to the test case that generated them, including Firebase device tests, takes out all the pain of diagnosing the failed tests. Everything you need is at your fingertips.
One place for everything
One tab. One view. One consolidated place to find out everything that happened in your tests. Note: The old test add-on has been deprecated. If you're looking for it, don't worry, all its functionality (and more) now lives in this new Testing tab.
Automatic flaky test detection
When tests fail then pass on retry, they're automatically flagged as flaky.
How auto-retry works in practice
When a test fails, Bitrise can automatically retry it without rerunning your entire suite. No more waiting to rerun the whole build because one UI test timed out. Auto-retry is available in five key Bitrise steps:
- Xcode Test for iOS (ID: xcode-test)
- Xcode Test without building (ID: xcode-test-without-building)
- Android Unit Test (ID: android-unit-test) (via Gradle plugin)
- Note: For Android Unit Tests, retries are configured in your Gradle plugin, not in Bitrise. Bitrise will detect them as retries and include it in your test reports.
- iOS Device Testing (ID: virtual-device-testing-for-ios)
- Virtual Device Testing for Android (ID: virtual-device-testing-for-android)
Each step has slightly different configuration options, but the principle is the same: retry what failed, identify what's flaky, keep your build green where possible.
For example, say one of your UI tests occasionally fails because an animation hasn't completed. Configure the test step to retry up to three times. If it passes on the second attempt, your PR stays green, but the test gets marked as flaky in the report. You're unblocked, but you still know what needs fixing.
This solves two problems:
- No more reflexive full re-runs: Teams waste hours re-running entire builds "just to check"
- Flaky tests can't hide anymore: They're automatically surfaced, even when they eventually pass
Stop guessing which tests are flaky
Switch to the Flaky Tests subtab and see every inconsistent test from a given build. Flaky tests are flagged right in the test list on the left, you can spot them at a glance before diving into any details. Click through to to see:
- Which attempts failed vs. passed
- Failure patterns (timing out? Only on certain devices?)
- Historical flakiness rate across builds
When flaky tests are detected, they're automatically listed in the $BITRISE_FLAKY_TEST_CASES environment variable.
You can use $BITRISE_FLAKY_TEST_CASES to:
- Fail builds on flaky tests: check if the variable is non-empty and fail the build if required
- Send alerts: notify your team when flaky tests are detected
- Create GitHub issues: raise a work request to fix the test
- Log failure in analytics: track and measure build performance in external apps
Click here for example snippets for these cases.
Test quarantine: turn off the water while you fix the leak
Enterprise customers can now quarantine problematic tests while keeping everything else running.
For example: you’ve identified a flaky test that's failing intermittently on your release branch. You know it needs fixing, but you also need to ship today. With test quarantine, once you’ve confirmed it’s harmless, you can surgically remove that specific test from your builds while keeping everything else running.
From the Flaky tab in your test report (or from Insights), click "Add to quarantine" next to any problematic test. Now you can scope the quarantine precisely:
- Quarantine everywhere: leave all fields empty, the test is skipped on all branches and workflows
- Target specific branches: only quarantine on release-* branches while keeping it active on main
- Target specific workflows: skip it in your nightly-tests pipeline but keep it in your PR checks
- Combine both: quarantine only when it's the release workflow AND the release-* branch
Say you have an Xcode UI test that times out on older simulators only on your release branch. You can now quarantine it specifically for this branch. Your releases keep shipping, the test still runs everywhere else, and you can debug without pressure.
Bitrise injects quarantined tests into the $BITRISE_QUARANTINED_TESTS_JSON environment variable. The supported steps (Xcode Test, Android Unit Test, iOS Device Testing, Virtual Device Testing) automatically read this and skip those tests.
Managing quarantined tests is simple:
- Access the Test Quarantine dashboard from your CI workspace
- See all quarantined tests with who quarantined them and why
- Add comments with ticket links so future-you remembers the context
- Remove tests from quarantine once they're fixed
Quarantined tests don't consume build minutes on automatic reruns. They're completely skipped, saving both time and money while you work on a proper fix.
Setting all this up (it's surprisingly simple)
- PR comments: In your Bitrise app settings, find the 'pr comments' toggle and configuration options. Select which information to post and choose comment mode.
- Auto-retries: add retry configuration to your test Steps (see examples above)
- Overhauled test reporting: now the default experience; just check the Testing tab on any build
- Flaky test detection: automatic when using auto-retries, use the
$BITRISE_FLAKY_TEST_CASESenvironment variable for automation - Quarantine: available for Enterprise plans, manage from Test Quarantine dashboard
Getting back to green faster than ever
Let's recap what just happened. A test failed, but now, it’s easier than ever to diagnose, debug and recover:
- You got actionable feedback directly in your PR
- One click revealed everything you needed to debug in the new test report view
- Flaky tests were automatically identified, flagged and retried until they passed
- Your team stayed unblocked while you fixed the real issue
Ready to make your test failures less painful? Check out the testing documentation or just push some code and watch it work.
Already using Bitrise? These features are available right now. Auto-retries, PR comments and the new test report are available for all plans. Test quarantine is exclusive to the Enterprise plan.
