6 questions your mobile CI/CD metrics can already answer

For many teams, Bitrise Insights means build duration and failure rate. Those are good starting points to evaluate your CI performance, but they often fall short when it comes to understanding questions about speed and quality.

This post covers six questions Insights can answer from the builds you already run: where the time goes between commit and merge, which tests are flaky, whether your build cache is working, whether builds are waiting for a free machine, what to fix first, and how to get alerted when something changes. For each question, we show which Insights page has the answer, what a good or bad number looks like, and what to do about it.

1. Git Insights and build metrics: where does the time go between commit and merge?

Cycle time is how long a change takes to go from first commit to merged. That's the wait your team notices. Build duration is only one part of it, but it's usually the number teams try to improve, because it's the easiest to measure.

If a change takes two days to merge, a ten-minute build isn't what's slowing you down. If it takes thirty minutes, the build is a third of the wait. You can't tell which situation you're in until you look at both numbers.

Work down the table and stop at the first level that explains most of the wait.

Every level reports percentiles instead of averages. p50 is the middle run: half your builds are faster, half are slower. p90 is the slow end: only 1 in 10 builds takes longer. An average hides those slow builds, and they're the ones engineers complain about.

Start with cycle time, then work out how much of it is CI. Subtract your median build duration from your median cycle time. What's left is time spent outside CI: writing code, waiting for review, and waiting to merge.

If builds are a small share, faster builds won't make shipping feel faster. Look at the development and review split instead. The usual causes are too few reviewers, pull requests that are too big, or changes held back and merged in batches.

Before you decide, open the pull request list. A few pull requests stuck for weeks need chasing individually. If every pull request is slow, the problem is your review process.

When builds are a big share, compare p50 and p90. Close together means builds are slow every time, so speeding up the slowest step will help. A p90 much higher than p50 means most builds are fine but some are much slower. Speeding everything up won't fix that. You need to find what makes the slow builds slow. Flaky tests, cache misses and builds waiting in a queue are the usual causes.

Before you assign anyone to a fix, check total duration. It shows how much CI time the problem uses across all runs, so you know how much a fix could save. Then filter by branch, workflow, step or test until you can name the cause. "This suite's p90 has grown 40% since April, and it's these four cases" gives you somewhere to start. "Builds are slower" doesn't.

2. Flaky test detection: which of our tests can't be trusted?

If p90 is much higher than p50, something is inconsistent. The most common cause is flaky tests: tests that pass and fail on the same code.

Failure rate can't tell a real defect from a flaky test, and the two need different responses. A real defect needs a fix. A flaky test failure needs a re-run, or quarantining the test. If you can't tell them apart, engineers start re-running every failed build, and a failed build stops being something anyone acts on.

Insights flags a test as flaky when it passes and fails on the same commit. It also checks repeat runs of a test inside one build, so it can catch flaky tests in builds that have no commit information. So a test has to run at least twice before Insights can flag it. And Insights finds flaky tests in runs that have already happened. It can't predict which tests will become flaky.

Some flakiness is normal as your app grows, so don't try to fix every flaky test. Most flaky failures come from a small number of tests. Work from the top of the ranked list, and the first few fixes will remove most of the noise.

For each test on the list, Insights shows the passing and failing runs with their commit and build. You can start debugging from those runs instead of trying to reproduce the failure. If a fix will take time, quarantine the test. It keeps running and reporting results, but a failure no longer blocks merges. You can quarantine it on one branch, workflow or pipeline instead of everywhere.

To estimate the cost, add up the flaky runs for your top three tests and multiply by your median build duration. That's roughly the CI time lost to re-runs. The engineer time spent investigating is higher, but harder to measure.

"We struggled with flaky tests, as they're really hard to pin down, especially determining the gravity of the problem. Insights was instrumental in finding those issues so we could resolve them."

Brandon Hawker, Software Engineering Lead, Fetch

At BuzzFeed, working out why a test kept failing used to be guesswork. Principal Software Engineer Josh Walker can now find the test and see its exact failure rate.

3. Build Cache analytics: is our build cache paying for itself?

Next, check whether your builds are redoing work they've already done. That's what a build cache is meant to prevent.

A remote build cache takes real effort to set up, and it's hard to show what you got for it. It can also stop working without any error. After a dependency upgrade, for example, builds can stop finding matching entries in the cache. Nothing fails. Builds just get slower.

If you use Bazel, Gradle or Xcode, you can also see metrics for each build tool command: how often it ran, how long it took (p50 and p90) and how often it failed.

Start with hit rate at p10. Your median can look healthy while some builds get almost nothing from the cache, and those are the slow builds people complain about.

Compare hit rate with last quarter after you upgrade a dependency, change your build tools or edit a workflow. Those are the changes most likely to break caching. When you need to show what the cache gives you, use saved build time. It's the most direct measure you have.

Error rate on commands matters too. A command that fails often is doing its work twice. That extra time shows up as a slower build, so it's easy to miss that the command is the cause.

4. Utilization: what's constraining our infrastructure?

If tests and caching don't explain the slowdown, check whether builds are waiting to start.

Your usage total shows what you spent. It doesn't show whether builds were stuck waiting for a free machine. You can be well within budget and still lose hours a week to builds queuing behind your concurrency limit (the number of builds you can run at the same time).

On Enterprise platform plans, the page also shows how much of your committed spend you've used, compared with how far you are through the billing cycle. If you're spending faster than planned, you'll see it partway through the cycle instead of at the end.

Where to find it: Insights → Utilization. It shows your current billing cycle by default. You can change the date range in the dropdown at the top right.

Go through the numbers in this order. Start with build count. If you're running more builds, that may explain higher costs on its own. If not, compare cached invocations with last quarter. A drop means your cache is doing less. Finally, look at concurrency and queued builds together. If you're near your limit but nothing is queuing, your capacity is about right. If builds queue regularly, your limit is costing engineers time, even if you're within budget.

5. Bottlenecks: what should we fix first?

Digging into each metric works best when you already suspect something and have time. Bottlenecks is for when you don't. It gives you a ranked list of what to fix.

Bottlenecks ranks problems by how much time they cost you: failing workflows, slow builds, resource usage, failing tests, slow tests and flaky tests.

Start here, especially if you haven't looked in a while. Bottlenecks tells you which workflow fails most, or which test costs the most time. It doesn't tell you why, so use the metrics above to find the cause.

Look at the list before you plan platform work. Otherwise it's easy to work on what you already suspected, instead of what's costing the most.

6. Alerts and dashboards: how do we stop needing to remember to look?

Everything above depends on someone remembering to look, and that usually loses out to delivery work. The slow changes these metrics catch, like a build getting a few seconds slower each week, are exactly the ones nobody opens a dashboard to check.

You can set alerts on almost every metric in Insights, including some that don't have their own chart. For each alert, you choose what to watch, a threshold, whether to alert when the number goes above it, below it, or either, and a time window of an hour, day, week or month. Alerts go to Slack, Microsoft Teams or email.

Where to find it: the bell icon above any chart, so you can create an alert from the metric you're looking at. You can copy an existing alert to another project or metric. Workspace owners manage workspace-level alerts. Project members can create project-level ones.

Start with a few alerts. If alerts fire too often, people stop reading them. Set thresholds where crossing the line means someone needs to act:

Record the numbers before and after any platform work. You can pick any date range, so you can compare the weeks before a change with the weeks after. A few months of before-and-after numbers is the clearest proof of what your platform work achieved, and you can't go back and collect it later.

Pin the metrics you check every week to a dashboard, so you don't have to set up the same filters each time. You can also export any breakdown as a CSV file.

At Fetch, each team sends a weekly report on build success and failure rates for the workflows it owns. Insights makes that report quick enough to put together every week.

Insights API and MCP: get your metrics without opening Insights

The Insights API is a read-only REST API for build and test metrics. It returns totals, trends over time, and flaky test cases for each project, so you can send them to tools like Datadog, Grafana, Databricks or Backstage, or run a scheduled flaky-test report. It covers builds and tests only: Git, cache and utilization metrics aren’t in the API yet. You sign in with a Bitrise personal access token or a workspace API token, and you only get data you could already see in Insights. The API is currently in open beta.

The Bitrise MCP server now includes Insights tools, so your AI agent can answer the same questions. Connect Claude, Cursor, GitHub Copilot or any MCP client and ask in plain language:

• “List the ten flakiest tests in the ios-app project this month.”
• “Compare p90 build time on main for the last two weeks with the two weeks before.”
• “Which workflows on release branches failed most often since Monday?”

The API and MCP tools come with paid Insights plans.

Your Insights checklist: six numbers to check first

1. Your median cycle time, and your median build duration next to it. What's left over is the part CI can't fix.
2. Your three flakiest tests, and how many CI minutes they cost you last month.
3. Your cache hit rate at p10, and saved build time this month.
4. How many builds queued last month, and whether you were at your concurrency limit when they did.
5. The top entry on your Bottlenecks list, and whether you'd have guessed it.
6. The one metric you'd most want an alert on, including ones with no chart, like spend against budget.

If a view is empty, you're usually missing a setup step:

Try it on your workspace

Insights is free for up to 500 builds and 100,000 test results a month, on any Bitrise CI or Build Cache plan. Start free or see what Insights covers. On an Enterprise plan? Your account team can go through this with you on your own workspace.