Key takeaways:
- We benchmarked open-source Bazel and Gradle app build times on CircleCI with and without Bitrise Build Cache.
- With Bitrise Build Cache enabled for the Bazel and Gradle app builds, we discovered an 83% and 72% decrease in build times, respectively.
- We also observed the elimination of “Infrastructure Fail” errors.
Introduction
The two most important characteristics of continuous integration (CI) are that it first be reliable and second be fast. In this benchmarking exercise, we compared the build durations of open-source Bazel and Gradle apps on CircleCI, building with and without Bitrise Build Cache.
Bitrise Build Cache is a remote build caching platform that addresses the challenge of escalating CI and operational costs associated with long builds. Bitrise Build Cache reuses previously saved build and test outputs to minimize the work done in subsequent builds, shortening build durations and developer feedback loops.
Applications under observation
We selected two open-source applications to benchmark on CircleCI. For Bazel, we selected Livegrep, and for Gradle, we selected the Android Signal App. We chose these two open-source applications for the following reasons:
- Bitrise Build Cache supports apps built with Bazel and Gradle build systems.
- These are reasonably complex apps with long build durations.
- These apps demonstrate modular code base designs, ideal for any remote build caching system.
- You can freely reproduce our tests on CircleCI or the CI platform of your choice.
We publicly forked each application and made the necessary configurations to build each application on CircleCI with and without Bitrise Build Cache. Implementing Bitrise Build Cache is straightforward, and the configuration changes we made can be found in the forked versions of the Signal App and Livegrep.
CircleCI environment and configuration
In an effort to keep the benchmark tests as simple, consistent, and reproducible as possible, we opted for a commonly used Linux machine executor on a Linux Large machine, offering 4 CPU and 15 GB RAM.
Next, we defined a CircleCI workflow with 2 parallel jobs with only one difference: run-tests-with-bitrise-build-cache has Bitrise Build Cache enabled, while run-tests-without-build-cache is running the same commands, but without – you guessed it – Bitrise Build Cache.
While CircleCI does offer dependency caching, in the spirit of keeping this comparison as simple as possible, we did not implement this or any other optimizations for either job.
Benchmark results
After running an initial build of both applications to populate their respective build caches, we ran three rebuilds of their original commits for a side-by-side comparison of the two jobs described above. This approach resulted in the maximum possible cache hit rate and serves as an objective baseline for future testing of incremental code changes.
In the case of the Bazel Livegrep application, the average build time without Bitrise Build Cache was 11 min 46 sec. The parallel job with Bitrise Build Cache enabled resulted in a 2 min 0 sec average build time. This represents an 83% decrease in build time.
The Gradle Signal App finished its baseline build (without Bitrise Build Cache) with an average of 29 min 0 sec. With Bitrise Build Cache enabled the parallel job finished in an average of 8 min 5 sec, representing a 72% decrease in build time.
Additional observations
During our benchmark tests, we made a notable observation. While running the Android Signal App builds without Bitrise Build Cache, we encountered multiple failed builds (example). The reason? Infrastructure Fail. This was especially problematic as these builds would run for 30+ minutes before failing. Ideally, when a build fails, it fails quickly, but not so in the case of Infrastructure Fail.
We did not experience any similar failures with Bitrise Build Cache enabled.
This compelled us to investigate the potential cause of the build failures. It didn’t take long before we discovered that, without Bitrise Build Cache, we were exhausting the resources offered by our CircleCI Linux Large machine. Check out the RAM utilization during the build of the Android Signal App:
Now, compare this to the RAM utilization of the same build, but with Bitrise Build Cache enabled:
This clearly illustrates one of the many benefits of Bitrise Build Cache – the more efficient utilization of available CI resources.
There aren’t many things more frustrating than waiting 30+ minutes for a build to fail, especially if it could successfully (and reliably) complete in 8 minutes.
Wrap up
We observed in this rather simple and reproducible test that connecting supported build systems to Bitrise Build Cache improves the reliability and speed of CI jobs. This brings the practical benefits of improving developer experience, eliminating a productivity bottleneck, and reducing CI machine usage.
Regardless of what CI provider you’re using, if you’re building an app with Bazel or Gradle, it’s worth running some builds using Bitrise Build Cache. And if you already use a build cache, which many Bazel projects do, consider running a benchmark with Bitrise Build Cache to confirm you aren’t leaving additional speed improvements on the table.
To learn more about Bitrise Build Cache, explore our documentation and consider starting a free Bitrise Build Cache trial to test drive it for yourself.
Have thoughts, questions, or ideas about our benchmark comparison? Leave your thoughts here.
