Toolbx, a developers new best friend!

Toolbx (formerly known as Toolbox) is our tool to allow you to use containers as a tool to make development simpler and better. It allows you to put each project you work on into a separate container and have a different set of tools and libraries installed for it. It even allows you to work on a different operating system, like RHEL, on top of Fedora Linux. To learn a little about both the history and the future for this project we will interview Toolbx’s author Debarshi Ray.

Christian Schaller: Hi Debarshi. To kick this interview off, can you tell us a little about yourself and how you became involved with open source development?

Debarshi: Hi Christian. I am Rishi. I am part of the Red Hat Desktop Team where we work on the Fedora and RHEL Workstations, and these days also on Silverblue. I first heard of free software or open source in 1999 through a newspaper article on Linux. I knew a little bit of C, and had been told that one could write everything from operating systems to video games with it. Hence the idea that I could actually look at the sources of a real piece of software was very appealing. However, I didn’t have access to the Internet and all I had was the shared family computer running Windows, so I had to wait.

A few years later, when I got my own computer, I decided that I had to put Linux on it. Somebody passed me a Fedora Core 1 DVD, and I was hooked. At some point I heard of this thing called Google Summer of Code, and in 2007 I applied and was selected for an internship at Fedora. Then in 2009, Rahul Sundaram, from Fedora, told me that this application called Gnote that we were shipping in Fedora needed a new upstream maintainer. I was already lurking around the GNOME community by then, and had been doing C++ as part of my day job. So I picked it up. Two years later, I applied for a position in some team at Red Hat. The recruiter spotted my Fedora and GNOME background, redirected me to the Desktop Team where an opening had miraculously shown up, and here I am.

Christian Schaller: So tell us a little about the origin of Toolbx, what were the ideas and goals at the outset of the project?

Debarshi: In the beginning of 2018, there was a surge in interest around what was then called Fedora Atomic Workstation. A decision was taken to rebrand it as Silverblue for better visibility, but it had one big problem. While Silverblue was already a fantastic option for non-technical users, it was quite annoying to set up a development environment on it for writing software. Typically, developers have to install a bunch of development tools, frameworks and editors on top of the core operating system, and on Silverblue every equivalent of ‘dnf install’ requires a reboot. Moreover, every such operation tampered with the core OS image and could potentially negate the two biggest advantages of Silverblue – robustness and testability.

We needed something better.

After several rounds of head scratching and ideating, Matthias Clasen showed me this small script that he had found under the CoreOS organization on GitHub. It was called ‘toolbox’ and could be used on an OSTree-based OS to present the user with a command line shell that behaved just like a traditional package-based OS. That felt like exactly what we wanted for Silverblue.

Sadly, while the script was a really cool proof-of-concept, it had several fundamental issues. First, it required root privileges to be used. That was a deal breaker. We couldn’t mandate that a user must have root privileges to be able to write programs on Fedora. Plus, we had a desire to use such a shell by default in our terminal emulator. Second, it used rkt, and while rkt was a really cool project, it was becoming increasingly clear that its days were numbered.

So, we kept looking for alternatives. It was around that time that the first murmurs around rootless Podman began to surface. That was the magic bullet we were looking for. Podman was a promising project with a solid community forming around it, and we no longer needed root access.

I started validating our ideas through this script written in POSIX shell, and word spread. In many ways, the growth of Toolbx closely tracked that of rootless Podman. Soon I found myself emailing my script to not just the Podman developers to shake out bugs, but to various other people in the community. It was clear that we needed to put up a Git repository somewhere.

However, Red Hat had just acquired CoreOS. This made the state of the CoreOS organization on GitHub unclear to us, and I wanted to make it clear that this was happening under the Fedora banner. So, I decided to call it ‘fedora-toolbox’ to avoid stepping on anybody’s toes since ‘toolbox’ is such a common word, and stuck it under my personal namespace on GitHub.

A few months passed, and we were seeing some serious interest in the project from outside the Fedora community. Meanwhile, Fedora CoreOS was now a thing, and they clearly had similar needs as Silverblue. So some people like Colin Walters, Ray Strode and Dusty Mabe convinced me to drop the `fedora-` prefix, and we became Toolbox.

Another few months later, several of the usual suspects suggested that we move under the Containers organization on GitHub, and here we are.

Christian Schaller: Current versions of Toolbx are written in Go, while the initial versions were written in shell, what was the reason for this change?

Debarshi: For the first two years of its existence, Toolbx was indeed written in POSIX shell. However, the shell isn’t a pleasant environment to program in, even less so if you want to stay POSIX compliant and not commit to something like Bash. It isn’t very expressive – even simple concepts require a lot of mental gymnastics to implement; and the lack of well-scoped variables made it a nightmare to keep the code clean as it grew. Finally, since every keyword is theoretically implemented by a command, the script was beginning to feel sluggish as it matured.

Go was a natural choice since huge swathes of the Open Container Initiative landscape are written in it – Podman is in Go, Kubernetes is in Go, and so is Docker. Given that Podman is such a critical dependency of Toolbx, it made sense to stay aligned.

Christian Schaller: So how does a development pet container work? What are the advantages of using Toolbx as opposed to just developing directly in your host OS?

Debarshi: First of all, if you are running Fedora Silverblue, then the advantage of Toolbx is quite obvious. It gives you a command line shell that behaves just like a traditional package-based OS, where you are not forced to reboot after every ‘dnf install’.

However, Toolbx is in no way limited to immutable operating systems. Lots of people use it on package-based OSes like Fedora Workstation because of several secondary benefits. First, it gives you a level of separation between the host OS and your development environment. Even if you mess up /etc or /usr inside your development environment, it won’t break your entire host OS. Second, you can set up several different environments on the same host OS. These can be programming language-based or project-based. They can even have entirely different OSes. eg., Fedora Rawhide or Fedora 33, or Red Hat Enterprise Linux, on a Fedora 35 host.

Fundamentally, a development pet container is different from the usual (cattle) container in that they are long-lived. The user is expected to use them interactively through the command line interface over an extended period of time. These containers are very rarely, if ever, thrown away and re-spun.

Christian Schaller: So tell us a little about how developers can use Toolbx to improve their developer workflow?

Debarshi: Developers can use different Toolbx containers to test their code under different build and run-time environments. For example, it’s very easy to create a container that’s different from the host OS and comes with a different set of compilers, interpreters, linkers and shared libraries. It’s much faster and lightweight to set up a container than a virtual machine, so it’s a fantastic option, whenever possible.

Second, it’s easy to have pre-canned container images that are tailored for a specific project. Or, one can turn a container into an image and share it with others to demonstrate a new feature or help reproduce a problem.

Christian Schaller: What about UI tools, like IDEs? Can you use them with Toolbx?

Debarshi: Yes, you can, but the developer experience isn’t very polished at the moment. These days we are shipping IDEs like GNOME Builder and Visual Studio Code as Flatpaks, while the vast majority of developer tools aren’t available inside the Flatpak run-time. It’s a lot easier to access those tools from inside a Toolbx. So, what we want is a way for IDEs running as Flatpaks to access developer tools from a Toolbx.

Some time ago, Christian Hergert added support to GNOME Builder for developing against a Podman container. Since every Toolbx container is also a Podman container, this means that you already get a decent user-experience using GNOME Builder with Toolbx.

Currently, there’s some ongoing work to make that possible with Visual Studio Code. Owen Taylor has created the toolbx-VSCode project to be a bridge between a Toolbx container and VS Code, and several community members have been trying the same using VS Code extensions. This is still work-in-progress, but we hope to mature it soon.

Christian Schaller: Is there any kind of development that Toolbx is not suitable for? For instance would toolbx be useful for Linus Torvalds doing kernel development?

Debarshi: Toolbx might not be able to, at least not fully, cover any kind of software development that involves working on very low-levels of the host OS. eg., components involved in the boot sequence like the bootloader, kernel, systemd, display managers, etc..

In such cases, one could work on the code and build it inside a Toolbx, but then install the build artifacts from a shell running on the host. For those who know what they are doing, OSTree-based OSes can be unlocked to make modifications to /usr, which can be useful in such cases. It still gets the job done without cluttering up the host OS with lots of development tools, frameworks and editors.

That said, I am sure there are gaps in the developer experience that need to be filled. We always love to hear from people working on different kinds of code bases, and see what we can do to cover their use-cases and help them use Silverblue as their primary OS.

Christian Schaller: So I know you can now set up a RHEL environment in Toolbx even when running on Fedora. What does a user need to do to create a RHEL container. Can you tell us a little about the things this enables a user to do.

Debarshi: Yes, earlier this year, for Fedora 34, we added experimental support for creating RHEL environments with Toolbx, using the gratis Universal Base Image provided by Red Hat. You just need to add the ‘–distro rhel –release 8.4’ flags with the ‘create’ command.

Note that this support is still experimental. We are currently working with folks inside Red Hat to improve and expand this support, and hopefully the results will soon be available to the public.

I think this is a pretty important step forward in how free software communities and contributors see and interact with RHEL. Historically, RHEL was difficult to access and work with for upstream communities, so most of them ended up using community rebuilds like CentOS. However, these days it’s possible to get gratis RHEL for a variety of use-cases, and we have been trying to make it easier to consume on Fedora Silverblue and Workstation. For example, in Fedora 27 we added support to GNOME Boxes to easily create gratis, self-supported RHEL VMs. This is another step in that direction.

Christian Schaller: What has the focus been for Toolbx development leading up to Fedora Workstation 35?

Debarshi: One of the big things that we did during the Fedora 35 cycle was to increase our test coverage. Ondřej Míchal and Oliver Gutierrez have done some fantastic work on this. They have made sure that the Toolbx test suite also gets run whenever new podman and toolbox RPMs are built for Fedora. Since Toolbx relies so heavily on Podman for its container manipulation, it’s crucial to test that changes to Podman don’t adversely affect Toolbx. In the past this testing was done manually whenever a new podman RPM would be built, and sometimes we would miss testing a build which would lead to user-visible regressions. Having it automated is a big step-up.

We also spent some time polishing the experience of system administrators using Toolbx for debugging their systems. This is of special interest to Fedora CoreOS. As part of this, Toolbx now offers a toolbox.conf configuration file under the ‘containers’ namespace.

Christian Schaller: What do you see as the next steps on the roadmap for Toolbx?

Debarshi: We certainly need to improve the integration of IDEs with Toolbx.

I’d like to make users feel more comfortable spending more time inside a Toolbx. Currently, people still feel the need to use a shell running on the host OS for various tasks. Some of these are to use tools like rpm-ostree, flatpak or podman to update, install or manipulate things on the host. Having wrappers for them inside the Toolbx that forward the call to the host under the covers would be useful.

Christian Schaller: If people want to get involved with Toolbx development, what is the best way to get in touch with you and the community? 

Debarshi: I, and the other Toolbx contributors, hang out in the #silverblue IRC channel on Libera.Chat, or the equivalent Silverblue Matrix room on Fedora’s Matrix server. We also lurk on other Fedora communication channels — IRC, Matrix and mailing lists. Finally, you can always file an issue against the Toolbx project at https://github.com/containers/toolbox