As a developer, when you think about CI/CD, you probably focus on build times, test results, and deployment pipelines. The infrastructure powering those builds? It's invisible (unless something goes wrong!).
At Bitrise, we've spent 10 years refining infrastructure decisions that most developers never see. In this post, we are pulling back the curtain on the infrastructure choices we've made and why they matter for reliability, consistency, and performance.
Reliability
Geographic redundancy
We operate infrastructure across three geographically distinct data centers: US-Chicago (ORD), US-Ashburn (IAD), and NL-Amsterdam (AMS). Why? It’s all about resilience. If any of one of our data centers experiences a natural disaster, power grid failure, or network outage, we’re prepared. With geographic redundancy, your builds will seamlessly keep running.
Redundant Tier 1 internet providers
Even within each data center, we maintain redundant internet providers. This means that a misconfigured BGP route or a fiber cut won't take down your builds. When ISP-level issues occur (and they do 😅), traffic automatically routes through backup providers. These decisions cost more and add to our operational complexity, but they're non-negotiable for infrastructure that teams depend on every day.
Continuous testing and observability
Unexpected problems are inevitable, which is why we employ multiple strategies to catch and fix them early. One of these is a suite of realistic mobile app projects that get built throughout the day, with defined Service Level Objectives (SLOs) and alerting when SLOs are breached. This allows us to detect problems before customers report them. Another strategy involves scanning build logs and alerting on problematic log output. After 10 years of operating a macOS cloud, we have accumulated a deep library of esoteric macOS issues that alert our team when observed. These are just a peek into the supports that surround the infrastructure layer that keeps your builds running.
Consistency
Consistency is often overlooked in favor of raw speed, but inconsistent performance can introduce flakiness and ultimately creates distrust and frustration with your CI system. We take a number of measures to maximize consistency.
Oversized storage
We spec our M4 Pro Mac Minis with 4TB SSDs. This is far more than most builds need. So why do we do this? All SSDs are rated to a certain number of write cycles. Apple hardware, while premium, is still consumer grade. In the CI context, Mac Minis are pushed much harder than the typical laptop. They are running file I/O intensive builds around the clock. An undersized SSD will prematurely reach its write cycle and result in performance regressions. So, we oversize the SSD on our machines so we can guarantee sustained, consistent performance throughout the life of the machine.
Custom cooling systems
Apple silicon is fast, but it generates heat. Under frequent load spikes (such as CI jobs), thermal throttling temporarily reduces clock speeds to manage temperatures until the fan control removes heat. Mac Minis were designed for desktop use where load spikes don’t happen as often, and such small performance loss is not noticed. But those losses can add up quickly across millions of builds. That’s why we optimize air flow to prevent thermal throttling in Mac Minis, and eliminate the measurable performance loss.
We also designed a custom shelf and cooling system specifically for our Mac Mini fleet to keep temperatures stable under heavy workloads. Our orchestration software monitors incoming workload spikes and can pre-emptively ramp up exhaust fans before builds even start. When the CI machine is kept cool, its full resources are available for the build job, and speed stays consistent.
Silent image distribution
In a multi-tenant cloud environment, one workload hogging resources from another workload is the noisy neighbor problem. We take steps to make sure Bitrise is never a noisy neighbor, but this is a challenge since we are constantly updating the VM images (hundreds of GB in size) to provide you with the latest tools and security patches. The solution? Our image distribution system uses the BitTorrent protocol to distribute images, and it coordinates with the job orchestration system so that CI machines only ever download and install new images when there are no jobs running on that machine. This ensures consistent build performance, unaffected by image updates.
Performance
We invest in the top spec’d hardware because it brings one of the highest returns on investment to developer productivity and your team’s ability to ship code fast.
Maxed-out M4 Pro specs
We don't use base-model Mac Minis. Our most recent Mac hardware are M4 Pro machines configured with:
- 14-core CPUs (vs. 12-core base). This gives you two additional performance cores.
- 64GB RAM (vs. 24GB base). This provides headroom for memory-intensive tasks (e.g. Bazel builds)
Data from XcodeBenchmark shows a roughly 15% difference in build times between base and fully-spec’d M4 Pro machines. Multiply that across all of your team's daily builds, and the impact is substantial.
10 Gigabit ethernet
Apple provides 1 GbE and 10 GbE network interface options for their Mac Minis. All of our machines have 10GbE. For builds that pull large dependency trees (think npm, CocoaPods, or Swift Package Manager), network I/O becomes a bottleneck. A 10x faster network interface can shave minutes off build times.
Co-located build caching
For teams using build caching, Bitrise hosts cache storage on the same high-speed switch fabric as build machines. This minimizes network hops, decreases latency, and maximizes throughput. Build cache servers are built with high memory configuration to maximize the “hot reads” from in-memory storage. A custom LRU eviction policy ensures the right entries stay in hot storage. The entire system is architected from the hardware up for the fastest possible builds.
Why CI Infrastructure Matters
We treat CI as a critical system, because without it, your dev team’s productivity grinds to a halt. When your CI runners go down, or slow jobs start queuing, your entire team is blocked. When builds fail due to flaky infrastructure, you waste time investigating phantom issues instead of shipping features.
At Bitrise, our goal is to make sure your team never has to deal with infrastructure headaches. We know your team's productivity depends on fast, consistent, reliable builds. That only happens when the underlying infrastructure is designed for it from the ground up. That’s what building without compromise looks like.
So, the next time you kick off a build on Bitrise, know that there's a lot happening beneath the surface to get your code merged as quickly and reliably as possible. When CI infrastructure is invisible, we know we've done our job right.
