A deep dive into asdf and version managers

Version managers, in general, are hacks. But they are useful hacks, as they solve a real problem: we all work on multiple projects at the same time, and each project needs a different version of tooling like Go, Python, or Ruby. It’s not enough to install a single version system-wide and hope for the best. Even if we can get away with using a single version across all projects, it would only be temporarily and accidentally working. Not to mention the hard-to-debug issues caused by the version mismatch between the local dev env, the CI env, and the production env.

Version managers solve some of the above problems by creating project-specific environments where an explicitly defined tool version is active, not your system-wide installation. They work by hijacking your shell, so when you call npm install or python3 main.py, the command is delegated to the right version of Node.js or Python.

Throughout this guide, I’ll be using asdf as the version manager to demonstrate concepts, but I want to mention the dozens of other version managers out there: rbenv for Ruby, pyenv for Python, nvm for Node.js, fvm for Flutter, or mise for a lot of the former languages in one tool. They are all implemented in similar ways and share some ideas, although differ in small ways. I’ve chosen asdf for this article for two reasons:

1. It’s a generic version manager with a plugin system. You can manage all your tools by a single tool and need to keep in mind only one tool syntax.
2. It’s the version manager of choice for Bitrise stacks. You can expect to have it preinstalled on all Bitrise stacks, as well as tools like Go, Python, Ruby managed by asdf (more on this below)

Under the hood

To understand what a version manager is doing, let's see what python3 actually is when asdf is set up on a system:

1$ type -a python3
2python3 is /Users/oliverfalvai/.asdf/shims/python3
3python3 is /opt/homebrew/bin/python3
4python3 is /usr/bin/python3

Even though there is a Homebrew-provided and a default macOS Python in my $PATH, asdf's python3 is the highest priority. Let's look at what this file is:

1$ cat /Users/oliverfalvai/.asdf/shims/python3
2#!/usr/bin/env bash
3# asdf-plugin: python 3
4# asdf-plugin: python 3.10.9
5# asdf-plugin: python 3.11
6# asdf-plugin: python 3.11.7
7exec /Users/oliverfalvai/.asdf/bin/asdf exec "python3" "$@" # asdf_allow: ' asdf '

We'll get back to this shim file shortly, but for now, just keep in mind that it’s a bash script that delegates the call to something else.

Version managers record the selected tool versions in config files. asdf has a concept of a “global” and a “local” selection. The globally selected version is recorded in ~/.tool-versions, and when the current working directory has no other definition, the global one is takes effect. But if the current directory also has a .tool-versions file (or a .ruby-version, .node-version), this local definition overrides the global one.

We can always check the effective versions by running asdf current:

1golang          1.20            /Users/oliverfalvai/.tool-versions
2nodejs          18              /Users/oliverfalvai/test-workspace/.node-version
3ruby            3.2             /Users/oliverfalvai/.tool-versions

In this context, the effective Node.js version comes from a local .node-version file, but Ruby and Golang versions are not defined locally, so asdf falls back to the global definition.

Version managers in CI/CD environments

In most CI/CD environments, your workflows are executed in a virtual machine that is destroyed after the build completes (sometimes called an ephemeral VM). Whatever change your workflows and scripts make to the system is lost when the VM is destroyed. Your next build will start from the same state as any other build.

So why are version managers used in such environments? Why can't you just upgrade the system-wide Python, or add a new version of Ruby to $PATH? Here are a few reasons:

1. Your friendly CI/CD vendor installs multiple versions of Ruby, Python, and Node.js in the build environment because there is no single right version for everyone, and installing your required version in every build would be a waste of compute and time (some tools have prebuilt binaries, but others are compiled from source). Installing multiple versions of a tool side-by-side requires some kind of a version manager.
2. You want to enforce tool versions across local dev envs and the CI/CD env. By using a version manager and a .node-version file, the given Node version is enforced for consistency and prevents hard-to-debug issues caused by running a different version of Node in the different environments.
3. You might have a monorepo where different sub-projects use different tool versions. With a .tool-versions or .node-version file placed in each sub-directory, switching to the correct tool version becomes automatic and explicitly managed.

Bitrise stacks

All Bitrise stacks have asdf preinstalled, but as of September 2024, there are some small differences because of the gradual migration from other version managers.

These languages and tools are asdf-managed on all stacks:

• Ruby: asdf-managed on both macOS and Linux. rbenv (the old system) is also present on macOS stacks for smooth migration, but this will eventually be removed.
• Node.js: asdf-managed on both macOS and Linux.
• Go: asdf-managed on macOS stacks and Ubuntu 22. The Ubuntu 20 stack has a single Go version installed, but that stack is now deprecated.

The following tools have partial support or not managed with asdf on all stacks yet:

• Tuist: Starting with the Xcode 16 stable stack, Tuist is installed via asdf, and you can easily switch Tuist versions. Tuist officially recommends Mise as the version manager, but it’s all powered by the asdf-tuist plugin, which Bitrise stacks use as well. On older macOS stacks, a single Tuist version is installed manually.
• Java: On Linux, versions are managed via Debian's alternatives system. On macOS, it's still managed with jenv. We also have the set-java-version step that works across macOS and Linux. Note: there exists a Java + asdf integration, but we ran into some performance issues, so we haven’t consolidated on asdf (or another tool) yet.
• Python: It's complicated. Let’s start with Linux, because the situation is cleaner there. Ubuntu 22 has full asdf Python support and preinstalls the latest (3.12.x) Python via asdf. The Ubuntu 20 stack is now marked as deprecated, so we don’t recommend using it, but it also has asdf set up, even though the default Python version is the system-wide one. On macOS, we are in the middle of a gradual migration from previous workarounds and version managers. Because we don’t want to break existing stable stacks, you might see tooling differences between different macOS stacks. The existing stable stacks (Xcode 14.x, 15.x) have pyenv set up, but the default Python version comes from Homebrew ([email protected]). On the edge stacks (as well as the upcoming stable Xcode 16 stacks), we consolidated everything under asdf: pyenv is gone, asdf is set up, and the latest Python (3.12) is installed under asdf. There is also pipx installed and set up, which allows reliably installing Python packages that are just executables (no more manual venv creation).
• Flutter: on Ubuntu 22, we started managing Flutter via its ASDF plugin, anticipating multiple version requests in the future. There is a single version installed at the moment though. All other stacks install a single Flutter version manually.

Shims

This article wouldn’t be complete without at least mentioning shims, because it’s a common source of confusion and error messages.

asdf and many other version managers use a mechanism called shimming. The idea is the following:

• Version managers need to somehow override the definition of ruby and other binaries in the current shell environment in order to invoke the right version of ruby. The right version of ruby depends on the context, such as the current working directory and the presence of a .ruby-version file.
• Instead of modifying $PATH at every command invocation, special shim files are added to $PATH once, at shell initialization time. This is why which ruby returns ~/.asdf/shims/ruby and not the path of an actual Ruby executable.
• When you call ruby, this shim file selects the right version at runtime and delegates the call to the right installation.

If we look at the actual implementation of a shim in ASDF, we can see that it's just a bash script and some extra metadata to keep track of installed versions.

1$ cat ~/.asdf/shims/ruby
2#!/usr/bin/env bash
3# asdf-plugin: ruby 3.1.2
4# asdf-plugin: ruby 3.1.4
5# asdf-plugin: ruby 3.2.2
6exec /Users/oliverfalvai/.asdf/bin/asdf exec "ruby" "$@" # asdf_allow: ' asdf '

A problem with shims: installing global packages into $PATH

When we install a new tool version (such as asdf install golang 1.21.0), asdf is aware that this new version is installed and can be activated. When we make a given version active, asdf records this in a .tool-versions file, as well as updating the metadata of the various shim files.

But what happens when we install a Go/NPM/Python/Ruby package as a CLI tool that we want to execute? For example:

• cordova, an NPM package
• yamllint, a Python package
• fastlane, a Ruby gem
• golangci-lint, a Go package

asdf is not aware of this install event because we run it through a tool’s package manager (e.g. npm install -g cordova) and not asdf. It can't update its config files to record which tool version was used to install the executable. In this example, asdf is unaware which Node.js version was used to install the cordova NPM package. These packages only make sense for a given tool version (or runtime, to be more precise), but the expectation for these "global" packages is to have them available in $PATH at all times.

asdf solves this by creating a shim file for each executable of a tool, not just the main one like ruby, python or go. In the case of Node.js, each globally installed NPM package gets a shim file. We can check this by listing the shims directory at ~/.asdf/shims:

1$ ls -la ~/.asdf/shims/
2total 268
3drwxr-xr-x 1 root root 4096 Apr 22 09:51 .
4drwxr-xr-x 1 root root 4096 Apr 22 09:43 ..
5-rwxr-xr-x 1 root root  166 Apr 22 09:44 2to3
6-rwxr-xr-x 1 root root  171 Apr 22 09:44 2to3-3.12
7-rwxr-xr-x 1 root root  200 Apr 22 09:51 appcenter
8-rwxr-xr-x 1 root root  163 Apr 22 09:51 bin-proxy
9-rwxr-xr-x 1 root root  260 Apr 22 09:51 bundle
10-rwxr-xr-x 1 root root  261 Apr 22 09:51 bundler
11-rwxr-xr-x 1 root root  163 Apr 22 09:51 commander
12-rwxr-xr-x 1 root root  161 Apr 22 09:51 console
13-rwxr-xr-x 1 root root  198 Apr 22 09:51 cordova
14-rwxr-xr-x 1 root root  363 Apr 22 09:51 corepack
15-rwxr-xr-x 1 root root  206 Apr 22 09:44 dart
16-rwxr-xr-x 1 root root  160 Apr 22 09:51 dotenv
17-rwxr-xr-x 1 root root  257 Apr 22 09:51 erb
18-rwxr-xr-x 1 root root  162 Apr 22 09:51 fastlane
19-rwxr-xr-x 1 root root  199 Apr 22 09:51 firebase
20-rwxr-xr-x 1 root root  209 Apr 22 09:44 flutter
21-rwxr-xr-x 1 root root  257 Apr 22 09:51 gem
22-rwxr-xr-x 1 root root  220 Apr 22 09:44 go
23-rwxr-xr-x 1 root root  223 Apr 22 09:44 gofmt
24[...]

When we execute one of these via $PATH, the shim delegates the call to the tool version active at the time. The metadata in the file links the given executable to the tool versions that truly have this executable installed. Here is an example shim file for cordova (an NPM package), with the metadata recording that this executable is provided by two Node.js versions:

1$ cat /Users/oliverfalvai/.asdf/shims/cordova
2#!/usr/bin/env bash
3# asdf-plugin: nodejs 20.12.2
4# asdf-plugin: nodejs 21.4.0
5exec /Users/oliverfalvai/.asdf/bin/asdf exec "cordova" "$@" # asdf_allow: ' asdf '

There is only one problem remaining: asdf is not notified when we run npm install -g cordova or gem install fastlane, so it can't set up new shims for the newly installed packages! This manifests in errors like this:

1$ go install github.com/golangci/golangci-lint/cmd/[email protected]
2$ golangci-lint .
3bash: golangci-lint: command not found

This is where the asdf reshim subcommand comes into the picture. When we run asdf reshim golang, it sets up the missing shims and removes dangling ones. In theory, we should run asdf reshim after globally installing any tool-specific package that we expect to be available in $PATH.

In practice, more and more asdf plugins run asdf reshim automatically after installing a global package:

• asdf-ruby injects a Rubygems plugin into gem install calls in order to reshim gems after a change
• asdf-nodejs has a wrapper for the npm command to call reshim at the right time
• asdf-python has a similar wrapper for the pip command
• asdf-golang has no such mechanism yet, the issue is tracked here. We need to run asdf reshim golang after a go install for now.

Is this even a good idea?

You might say that using a version manager is a lot of added complexity and friction. This is partly true, but above a certain project complexity and team size, it’s better to have some upfront friction (adopting a version manager locally and in CI, then declaring dependencies explicitly) to avoid unexpected problems in the future, such as:

• hard-to-setup local dev envs for new joiners, as well as existing local setups breaking suddenly
• a mismatch of local, CI, and prod environments (the famous “but it works on my machine!”)
• unpredictable tool (and other dependency) upgrades

There is also an industry-wide pattern to take this one step further and skip the version manager altogether:

• Gradle has the Gradle Wrapper: instead of calling a system-wide gradle CLI, you call a ./gradlew file from the project directory. It downloads and caches the correct Gradle version needed by the project, then forwards every command to that Gradle instance
• The Javascript ecosystem has corepack: when it’s activated, you can add a packageManager field to the project's package.json and define an alternative package manager like yarn or pnpm there, including the exact version of the tool. After this, yarn or pnpm calls in the context of the project directory will use the correct package manager version, similar to Gradle Wrapper.
• Python's virtualenv, when activated, adds .venv/bin to $PATH, so if your project has a dependency that provides an executable (e.g. pytest), then it's instantly available in your shell.