FreeBSD containers on macOS

Podman, like Docker Desktop, aims to support running Linux containers on popular client operating systems (macOS, Windows) so that people can work on these systems and deploy into the cloud. It does this by spawning a Linux VM and connecting to it from the outside.

But what if your servers run FreeBSD? Thanks to the fantastic work of Samuel Karp with runj and the containerd suite and Doug Rabson with ocijail and the Podman suite, there are lots of options for running containers on the server. It would be nice if you could have the same affordances as Linux users when working on other systems.

For the last few weeks, I've been intermittently working on the bits necessary to make this possible. There are a few things that need to work for Podman to be able to connect to a guest:

• It provides configuration via the QEMU firmware configuration interface.
• The configuration information (including host SSH keys) is in the Ignition format.
• Host filesystems are exposed via 9p-over-virtio

Juniper wrote an implementation of 9p-over-virtio for FreeBSD. Steve Wills updated it somewhat and Doug Rabson cleaned it up to work on the latest FreeBSD.

FreeBSD lacks a driver for QEMU's firmware configuration interface. This is a simple key-value store that can be exposed as a read-only filesystem in the guest. Linux has a big kernel module to do this but NetBSD had a nice model, with a tiny kernel driver and a FUSE filesystem. I reimplemented the NetBSD interface and ported their FUSE filesystem to FreeBSD.

With those building blocks in place, it should be build a FreeBSD VM image that podman could use. This is quite easy. FreeBSD has an incredibly powerful tool called Poudriere (French for powder keg), which can build package sets, including with an overlay of custom ones, and can build VM images (and a large number of other things) including them, along with other default configuration.

I put together a Poudriere configuration that builds a VM that should be the right shape for Podman to use. You can use this on a FreeBSD machine or VM to generate the VM image. This will be the latest -CURRENT, with the 9p-over-VirtIO patches applied, the kernel module for the QEMU firmware configuration interface, and all of the podman tools applied.

The only hiccup was that ocijail uses Bazel to build. Bazel depends on, as far as I can tell, every possible programming language ever written. It doesn't build on FreeBSD/AArch64 because it depends on OpenJDK, which is broken on 64-bit Arm (write once, debug everywhere still holds, apparently). The Poudriere scripts add a CMake build system that sidesteps this problem.

I originally wrote some custom code to parse Ignition but then realised that a simple shell script around jq would do everything I needed.

So now you can provide the VM image that this builds to the podman machine init command and it will almost do the right thing. Unfortunately, podman does not know that this is a FreeBSD VM and so tries to mount the host filesystems by using Linux-specific commands.

I have started working on Podman to improve this, first by teaching it to keep track of the OS that it thinks is running in a guest and then by making it use FreeBSD mount commands if the guest is FreeBSD. These changes are not yet upstream (and probably do things at the wrong layer, so I expect will need changes before they can be).

With this set of things, I can now start a FreeBSD VM with podman:

$ podman machine init --cpus $(sysctl -n hw.ncpu) --image-path podmanvm.img --rootful --machine-os freebsd  test
...
$ podman machine start test
Starting machine "test"
Waiting for VM ...
Mounting volume... /Users:/Users
Mounting volume... /private:/private
Mounting volume... /var/folders:/var/folders
...        

More importantly, I can now run FreeBSD containers on my Mac:

$ uname -mrs
Darwin 22.5.0 arm64
$ podman pull docker.io/dougrabson/freebsd13.2-minimal
Trying to pull docker.io/dougrabson/freebsd13.2-minimal:latest...
Getting image source signatures
Copying blob sha256:b4127cc3a80bc585848922d741286c161aed800d5c34c930838b56114f819ecc
Copying blob sha256:a0f25d73ad616d07d848fef64d2444da4dced4b5feab90ab30d4b9081171795a
Copying blob sha256:0ee5d54f49bfe3c9ca0edba199d042013cb339e9a52081911ba6773d68201312
Copying blob sha256:39bbeab0b5691d83f375130697e116959ae585955dd3fd6996b85773b4023685
Copying config sha256:c5ee94515c6c35099efc59178915c482dff4d0c12b927389c5f964c0f7f96612
Writing manifest to image destination
c5ee94515c6c35099efc59178915c482dff4d0c12b927389c5f964c0f7f96612
$ echo hello from the host > /private/tmp/test/hello
$ podman container run --mount type=bind,src=/private/tmp,target=/private/tmp --rm freebsd13.2-minimal /bin/cat /private/tmp/test/hello
hello from the host
$ podman container run --rm   freebsd13.2-minimal uname -mrs
FreeBSD 15.0-CURRENT arm64
$ podman container run --rm   freebsd13.2-minimal freebsd-version
13.2-RELEASE-p2        

I'm looking forward to upstreaming these changes to both FreeBSD and Podman.

There's still more work to do. This is sufficient for a proof of concept but has a lot of limitations (not least that it works only with the QEMU back end in Podman, not any of the others). Contributions and help welcome!