What was the force upgrade process at Uber?

Everyone who joined our Building Mobile Apps At Scale webinar will know how fascinating it was to hear, first of all, Gergely Orosz’s journey at Uber, and how Uber's mobile engineering team needed to respond to the challenges of scaling rapidly. And we were thrilled to follow that with an absorbing Q&A where Gergely was joined by Pooja Bhaumik and Franz Busch, who responded to a wide range of questions. 

This is the first part of a series of blogs summarizing that Q&A— both the questions answered on the night and those that there wasn’t time for. This first part focuses on questions surrounding app development and processes. Coming separately later, we’ll examine refactoring vs rebuilding (which was a big topic of interest), and engineering teams and culture.   

The answers for those questions answered on the night have been edited down and summarized to keep them relatively concise. If you prefer, you can watch back the full recording.

Today we’ll be answering:

• Could you explain more about force update experience at Uber?
• Is force upgrading recommended in favor of keeping multiple API versions in parallel?
• What are the best practices for using feature flags?
• Is Uber (or any other big) app able to disable a feature remotely? How do you go about that? That would avoid the problem of having old versions of an app still having that deprecated feature
• Is Uber considering using a cross-platform framework, e.g. Flutter?
• If you are building a new feature on an existing native mobile app, what criteria do you use to determine the best option: native code vs Flutter vs React Native?
• We have an ongoing discussion around presentation layer design patterns in our organization (50+ devs per platform). At this scale, would your recommend a single design pattern for presentation layers (i.e. MVP, MVI, MVVM, Redux, etc…), or generate a “platform level” plugin architecture in which product teams can choose the most appropriate pattern for their feature rather than be forced to subscribe to a single pattern?
• Do you have learnings about how to test out new architectural patterns or spiking new technology to find out if they work for your app? And how do you evaluate it before you decide to adapt this architecture for the full app (in rewrite or step-by-step integration)?
• At what stage did you decide to move away from xcodebuild to Bazel?
• Can you cover the decisions/journey around the mapping engine you originally used and how that evolved?

Could you explain more about force update experience at Uber?

Gergely: “There are two parts to this. One of them is that you might have some sort of crazy bug in the binary that you want to fix. And you want to make sure that people are no longer using that old version (this is the more rare type, by the way). And if you have this and you don't have force upgrade, you're in trouble. 

So what you typically do is you release a version after a few weeks, you somehow tell people to upgrade. And for example, WhatsApp and Messenger do a really good job. If you open them, after a while they tell you they're just not going to start until you do an update, they have this rolling window, pretty much. Uber doesn't have this. 

The more common thing is that you want to retire backend APIs, because you don't want to support a bunch of stuff or you want to shut them down. If you forgot to ship a piece of code that can turn off like older versions, you're in deep trouble, because you can technically never turn that off. And the problem we had at Uber — we had force updates in place, we never tested it too much, but we never used it. Because every time when we were about to use it, the business looked at it and said like, ‘oh, there's still one and a half percent of our users using the older Uber app’ (which was before the rewrite) and this means, like, $100 million per year in revenue. And that's that's too much, we're not going to do that. And we were locked in this problem that we had too many valuable users. 

My suggestion to anyone is always have a window of certain months. Look at American Express and at all the banking apps — it's not a great experience, but it makes sure that you're going to be supported, because you have this testing problem of testing old apps. And then at Uber, even really old versions work surprisingly well. But it's a huge pain for the engineering team. We have to balance customers who don't update versus engineers.”

Franz: “We had a similar experience. We actually didn't ship with force update in the beginning and it really bit us at one point because we had some legal requirements we needed to enforce and we could only ship that in a new app version. So what we had to do was, in a bunch of APIs, we had to implement an app version check and return an error for these versions saying that they need to upgrade, which was really a sub-optimal experience, because it was just an error alert instead of popping up the App Store and updating it. 

I think the best practice nowadays is this force upgrade window with a soft window, probably like the Twitch app is doing — you open it, it asks you to upgrade, but you can still use it for some time. I think that's a perfect balance between user experience and developer convenience.”

Is force upgrading recommended in favor of keeping multiple API versions in parallel?

Gergely: “Force upgrading helps with being able to retire old API versions. See more in the Force Upgrading section in Building Mobile Apps at Scale.”

What are the best practices for using feature flags?

Pooja Bhaumik has written about feature flags for Increment magazine and we also have our own introductory guide to feature flags. 

Pooja: “Feature flags are basically like superpowers for mobile teams. When we have these long approval cycles, feature flags help us toggle features on and off, but they do clog up our code bases. 

A best practice is once a feature is 100% live, you can kill that feature flag because usually a feature flag is probably there for experimentation or for A/B testing — if that’s the reason, do remove the flag associated with it. Some developers do not like that advice, because they have been in situations where a very old feature flag has helped us reopen a feature when a business requirement was there. It is only for some flags. Otherwise, definitely remove it.” 

Gergely: “I would say just remove it regardless!”

Pooja: “It can be very hard to convince some developers! But yeah, testing is definitely very hard. Because if there are two features with this tool, feature flags are there and if you turn on one, it should not affect the other feature in some way. So having predefined test cases helps us. 

And you can also have versioning strategies. Because if you have turned off a feature flag due to a problem, you cannot turn that back on once you have just uploaded a new build, because the older users who have not updated the app are still going to get the old bad code. Definitely having a maintenance system is going to help you rather than doing it manually.

Is Uber (or any other big) app able to disable a feature remotely? How do you go about that? That would avoid the problem of having old versions of an app still having that deprecated feature

Gergely: "Yes, most of Uber's functionality is controlled by feature flags, so a lot of it can be turned on or off. There are strict protocols on these though, given the impact. It does not solve the problem of old versions of the app: old versions are a pain because you have to keep testing them and ensure they work."

Is Uber considering using a cross-platform framework, e.g. Flutter?

Gergely: “We did look into whether we should use React Native. The problem is there’s two types of companies. One doesn’t have the budget to hire all these different engineers — so to build a feature, you need an iOS engineer and an Android engineer and you could save a bunch by having one engineer do that. 

And then there are companies who have other problems and Uber was in this second phase. They cared less about costs but what was really frustrating — as it is at every single mobile company — is time to ship. You have a bug to fix. You do the bug fix and ship it out on the build train, people have to update, etc. Would it not be nicer just to push code live, a bit like on the web? This is why Uber, for example, looked at React Native. In the end, Uber decided not to do this but we built something in-house, which also didn’t work that well. 

So the short answer is that Uber isn’t considering Flutter or React Native or similar for two reasons. First of all, Uber does not want to be dependent on a third party. And Uber does have the in-house expertise to build these best-in-class experiences. The other question is would I consider using Flutter or React Native or something else if I did a startup? Absolutely. I would not do what Uber is doing. I think you need to know where you are. Don’t follow what Uber or whoever does.” 

If you are building a new feature on an existing native mobile app, what criteria do you use to determine the best option: native code vs Flutter vs React Native?

Gergely: "Look at trade-offs. React Native and Flutter add one more dependency and abstraction. Prototype approaches, and validate how they work on the things you care about (e.g. performance, code reusability, ease of learning etc.). There is no ‘best’ solution, only trade-offs. Verbalize what is important to you and choose the tool that is the best fit or has the fewest trade-offs."

We have an ongoing discussion around presentation layer design patterns in our organization (50+ devs per platform). At this scale, would your recommend a single design pattern for presentation layers (i.e. MVP, MVI, MVVM, Redux, etc…), or generate a “platform level” plugin architecture in which product teams can choose the most appropriate pattern for their feature rather than be forced to subscribe to a single pattern?

Franz: “We were basically at a similar point with our iOS application where we had a lot of different architectures used in the beginning and we saw a lot of problems with that. And what we did is we wrote down a list of requirements that we needed from our architecture — like deep linking, navigation, testability, all of that stuff. And we evaluated what was out there. 

And we found that the RIBs architecture that Uber was building was perfect for that, because it allowed us to scale very greatly because the interfaces were the same. It was plug and play. And we could actually exchange things easily. Testability was great and reactive code was included. 

So I think at that level, it's really, really important to have a single way of doing the overall architecture when it comes to the specific presentation layer. I think this is where your architecture needs to be as flexible to just swap it out. So if Apple comes around with Swift UI now or just Jetpack Compose on Android, you should be able to just rewrite the UI without changing the business logic and really the RIBs architecture for large teams. It scales. I can only recommend it for anybody.”

Gergely: “The background here was that at Uber we had MVP. And we kind of said, ‘everyone follow the MVP on Android and MVC on iOS.’ And we just kept running into circles and the opportunity that we had — we knew we would grow to more than 100 engineers, we have the platform to spend about three months prototyping all the approaches. So they did MVP, MBI, MVVM, Viper. Redux wasn't a thing back then so we didn't do that. And they liked Viper the most, but it was missing a few things — code generation, specifically. So that's how they came up with RIBs. And there are talks on how they did it. 

Now, I'm not saying you need to do this, by the way. There are downsides of RIBs. It's heavyweight. It's a pain to change — deliberately. It's really hard to mess up, but it's hard to extend, etc. So I'm not saying copy it. 

I like what you're doing, that you're prototyping. But the presentation layer is not the important one. What's important is the navigation. Look at your most complex screen. How many teams own that today? And how can they avoid stepping on each other's toes. That's where you will need immutable states, for example. I will suggest: do your research, talk with other people at similar companies, because RIBs — that was five years ago and that might not be the best thing today. But I think you're on the right track. Prototype, do the approach, and then just go with something. Just don't lock everyone up by saying ‘you need to use this’.”

Do you have learnings about how to test out new architectural patterns or spiking new technology to find out if they work for your app? And how do you evaluate it before you decide to adapt this architecture for the full app (in rewrite or step-by-step integration)?

Gergely: “Protype, prototype, prototype! Coinbase has a good writeup on how they did this with RN: Announcing Coinbase’s successful transition to React Native.”

At what stage did you decide to move away from xcodebuild to Bazel?

Gergely: “We first moved to Buck in 2016, later to Bazel. More on moving to Buck: Faster Together: Uber Engineering’s iOS Monorepo.” 

Can you cover the decisions/journey around the mapping engine you originally used and how that evolved?

Gergely: “I did not work on mapping. If you google "uber engineering blog mapping" there are lots of engineering posts from Uber though.