Success is tough to measure when it comes to engineering teams. Elite teams know that you’ll only get part of the picture if you measure outcomes in exclusion to output. Of course, you measure your team and product for velocity and quality, but you also need to understand your team’s capabilities. For DevOps, the DORA metrics offer teams insights from a DevOps performance perspective.
Similarly, Mobile DevOps teams need to compare their outcomes to their output. Bitrise’s Mobile DevOps Assessment Survey (MODAS) seeks to do just that. And like DORA, MODAS touches on the Change Failure Rate (CFR) concept, albeit in a mobile-focused way: looking at what percentage of releases are hotfixes.
Change failure rate, meanwhile, indicates the efficiency of your team’s overall development process because it measures the number of code changes (or pull requests) that cause failures in production.
How to measure CFR and why it matters
CFR gives insight into the velocity of your product, which translates to how much value you deliver to your users. Measuring CFR examines your code for changes to your production environment, like server status and changes to code or configuration data, and how your deployments are managed.
In the simplest terms, CFR is the number of failed deployments divided by the total number of deployments. However, your change failure rate is different from your failure rate. CFR does not consider errors at the users’ end; that is, failures that are not due to developer error. Similarly, those incidents that occur after deployment without needing a change are not factored into the change failure rate.
Change failure rate only considers what happens after deployment and does not consider anything that happens before deployment. The change can be anything from added features to quick code fixes.
Some common examples of after-deployment code changes would be:
- Hotfixes deployments: a significant bug in your software is fixed with a small piece of code released as quickly as possible.
- Fix-forward/rollback: let’s say your team deploys a code change, but it’s not working as intended. The fix-forward is a quick code correction for the new error. However, sometimes it can be high risk to make a quick fix, and you need to roll back or revert to the last known good code revision.
- Patch: when a software update is inserted (or patched) into the code of an executable program.
Over time, and with insights from MODAS, your change failure rate should decrease as your team develops best practices, automates parts of its CI/CD pipeline, and increases its performance. The MODAS assessment looks at every part of your app development process to ensure performance and productivity.
Four ways to improve CFR
It is possible to improve your change failure rate immediately. However, understanding why failures happen leads to better code deployments. For us here at Bitrise, Mobile DevOps is just as much about the culture as it is about the technology. A strong Mobile DevOps culture is paramount to improving change failure rates. Once you have that in place, these steps become second nature.
1. Create a culture of QA and testing
From automated unit and UI interaction testing all the way to manual QA (quality assurance) testing, creating a Mobile DevOps culture that focuses on testing is the first step to improving change failure rates. Through different tests, mobile app developers and teams can gain valuable insights into each new release’s quality level and potential risk.
2. Improve your MTTR (mean time to recovery)
If you think about it, CFR is a stability measure, and so is your MTTR. Decreasing the time it takes your team to recover from failures caused by changes will also improve your change failure rate. The faster you recover, the less time your system is down.
3. Make smaller deployments more frequently
Controlling your deployment frequency and size allows for failures to be easily tracked and quickly fixed. Imagine having to dig through commits that go back days, or weeks, trying to find an elusive bug. If your deployments happen at shorter intervals, it’s easier to find the bug in a smaller deployment.
4. Know when to automate
Code reviews are a must for catching issues in code before they reach users and for maintaining a strong codebase, but the process can be quite a pain. So, knowing when to automate, say, for code reviews, will help you improve code quality and save valuable time for your team. But that’s not all — you can automate testing, building a new release package, and actually deploying the new version. The trick is knowing when to automate, and insights from the MODAS into your process can help you figure this out.
Next steps: optimize your end-to-end Mobile DevOps performance
ICYMI:
- Simply put, CFR is the number of failed deployments divided by the total number of deployments.
- Change failure rate considers changes to code after deployment. For example, hotfixes, rollbacks/roll forwards, and patches.
- You can immediately implement four things to improve CFR: test, improve your MTTR, make smaller deployments, and, more often, know when to automate to ease your team’s workflow.
- If you think about it, CFR (like MTTR) is a stability measure. How stable is your Mobile DevOps team?
- A Mobile DevOps culture constantly tests, monitors for improvement, and knows when to automate.
How your team manages incidents that lead to failure is just as important as how many deployments your team can make. Tracking incidents like broken code that leads to failures can provide more well-rounded insights and can show what your team needs to improve.
Gaining these insights into what’s happening to your builds gives your team the information they need to be elite and high performing. The MODAS assessment takes less than 15 minutes to complete, and once completed, your team gains insights into how to improve their end-to-end Mobile DevOps performance
If you’d like insight into how your mobile development, mobile testing, mobile monitoring, mobile delivery, and mobile collaborations stack up — jump into MODAS now. Or see how Bitrise can help improve your change failure rate and optimize your mobile builds.
.jpg)
.jpg)
