Porting OBuilder to FreeBSD

OBuilder is a tool for performing arbitrary, reproduceable builds of OCaml-related software within a sandboxed environment. It is used by the CI team at Tarides to provide OCaml-based Continuous Integration (CI) for projects like opam-repo-ci, ocaml-ci, and multicoretest-ci. Originally written for Linux, OBuilder had Windows and macOS support added later. Previous blog posts have covered porting to macOS and OCaml CI in general.

Here we cover the work to add the remaining Tier-1 supported architecture for OCaml that's missing from OBuilder and FreeBSD. With this work, the CI systems maintained by the team at Tarides will be able to support FreeBSD.

The Challenge

Being initially Linux-centric, OBuilder is architected around three major requirements:

• Initial build environments are Docker images.
• Sandboxing is performed using the Open Container Initiative (OCI) tool runc.
• A filesystem with snapshot capabilities is needed and acts as a cache of identical build steps.

Neither of the first two items are available under FreeBSD (a Docker client is available, but there is no native Docker server); therefore, alternative solutions must be found. As for the filesystem requirement, FreeBSD has been supporting Sun's ZFS filesystem out of the box for many releases now. ZFS support already existed for Linux and macOS, and it is being used in the macOS port (more details here).

Fortunately, the existing architecture for OBuilder encapsulates these needs as Fetcher, Sandbox, and Store modules, respectively, so the only work required would be to write FreeBSD-specific Fetcher and Sandbox modules.

The Fetcher

Initially we tried to fetch Docker images without using the docker command. We have an existing script, download-frozen-image, of this attempt in the moby GitHub project (the open source parts of Docker). Although if we use that script to fetch the Docker image's various layers and apply them in order, the result won't be beneficial from a FreeBSD perspective because the available Docker images are filled with Linux binaries. These can run under FreeBSD with the help of the compatibility module, but the OCaml toolchain would believe it was running under Linux, so it would build Linux binaries. This is not the solution we are looking for. Until Docker is available under FreeBSD, there won't be a repository of FreeBSD images suitable for OBuilder. Such images will, at least in the beginning, be built locally in the Tarides CI network.

Given this, it makes sense to expect .tar.gz archives to be available, then we can simply download and extract them to implement the Fetcher module. Moreover, FreeBSD provides its own fetch command, which is able to download files over http and https. It can also use file:// URIs, which turned out to be very helpful during development. There is currently no attempt to support aliases or canonical names, so all the (from ...) stanzas in OBuilder command files will need to be adjusted for use with FreeBSD. This limitation can be overcome by prepopulating the OBuilder cache with the most-used images under their expected names on the OBuilder worker systems.

The final solution for the Fetcher on FreeBSD uses ZFS to store the base images as ZFS datasets on each machine. These get mounted into the jail with the approriate installs of OCaml, opam, and a Git clone of the opam-repository. We ended up with a layout of:

The base image name is used in the (from ...) stanza of the OBuilder spec files to select an OCaml version. Interestingly, we used a similar layout in the macOS OBuilder port. More details are available in the freebsd-infra ansible scripts.

The Sandbox

FreeBSD has come with its own sandboxing mechanism, named jail, since the late 1990s. In addition to only having access to a subset of the file system, jails can also be denied network access, which fits the OBuilder usage pattern where network access is only allowed to fetch build dependencies.

In order to start a jail, the jail command is invoked with either a plain text file providing its configuration or with the configuration parameters (in the "name=value" form) on its command line.

In order to keep things simple in OBuilder, and since the jail configuration will only need a few parameters, they are all passed on the command line. This might be a problem if the length of the run command, as specified in the OBuilder command file, reaches the FreeBSD command-line size limit. Since this limit is a few hundred kilobytes, it does not seem to be a serious concern.

The jail invocation will provide:

• A unique jail name
• The absolute path of the jail filesystem
• The command (or shell script) to run in the jail
• The user on behalf of which the command will be run. This requires the user to exist within the jail filesystem (/etc/passwd and /etc/group entries).

More options may be used to allow for network access or specify commands to run on the host or within the jail at various states of the jail lifecycle.

Also, for processes running under the jail to behave correctly, a stripped-down devfs pseudo-filesystem needs to be mounted on the /dev directory within the jail. While this can be done automatically by jail(8) using the proper mount.devfs option, care must be taken to correctly unmount this directory after the command run within the jail has exited. In order to be sure there will be no leftover devfs mounts, which would prevent removal of the jail filesystem at cleanup time, OBuilder unconditionally runs an umount command after the jail command exits.

Lastly, since most (if not all) OBuilder commands will expect a proper opam environment configuration, it is necessary to run the commands within a login shell. Such a shell can only be run as root. Therefore the command that will run within the jail is:

  /usr/bin/su -l obuilder_user_name -c "cd obuilder_directory && obuilder_command"

The jail-based sandbox environment provided by FreeBSD OBuilder closely mirrors the original Linux runc-based implementation because it targets an operating system-level virtualisation. In both cases, we expect to see similar performance and stability from the FreeBSD port. With the Fetcher and the Sandbox modules written, a complete OBuilder run can be attempted.

Integrating with OCluster

OCluster is a larger system that processes build requests on a cluster of servers, each running an OCluster worker that uses OBuilder as a library. In order to make the FreeBSD systems compatible with OCluster's needs, a few more adjustments are necessary. A Docker client for running health checks, a ZFS pool, and a FreeBSD base image all need to be addressed.

Fortunately the Docker client is available as a FreeBSD package: pkg install -y docker would do the trick; however, we do something even simpler and create a shell script which does nothing. OCluster worker uses this script as a healthcheck. In future, we plan to change this to target a more appropriate FreeBSD health check.

As we hinted at earler, FreeBSD needs to be setup with a ZFS pool on a separate disk or a separate partition. The basic idea is that OBuilder will store base images plus build-state snapshots on this pool, and it is better to keep it separate from the main system. Additionally, the usage patterns of this pool involve a huge amount of reads, writes, snapshot creations, and deletions, so it makes operational sense to isolate it and allocate it on different storage than the operating system storage.

The FreeBSD base images are created by building opam and OCaml from source, then initialising an opam repository from a Git clone of the opam-repository GitHub repo. More details are available in the freebsd-infra ansible scripts. With some FreeBSD knowledge, this should allow anyone to setup an OCluster worker on FreeBSD. In future, these base images could be built on a single machine and copied between machines using zfs send on the source machine and zfs recv on the build machine.

Currently we have a single server running FreeBSD 13.2 providing OCaml 4.14 and 5.1 builds on x86_64. This modest Dual Xeon machine (16 cores) is easily handling the load from opam-repo-ci, ocaml-ci, and opam-health-check, with a peak observed throughput of 40 jobs per hour. This initial deployment has confirmed the performance and stability expectations we had.

Future Work

Some future work that we would like to do:

• Optimising I/O using an in-memory OverlayFS. A similar setup on Linux has given us some impressive performance improvements.
• Supporting FreeBSD on ARM (if there is sufficient interest in this architecture)
• Investigate OCluster performance on FreeBSD using DTrace

Conclusion

The modular design of OBuilder has allowed for it to be easily adapted to run under FreeBSD. A few FreeBSD systems are currently being set up as OBuilder workers within the OCluster orchestrator used by Tarides for automated OCaml package testing. Support has been added to opam.ci.ocaml.org for checking opam packages, and ocaml.ci.dev has FreeBSD builds available for OCaml projects hosted on GitHub and GitLab. In addition, the FreeBSD-specific instance of opam health check is providing base-level metrics of the repository health on the now-supported platform. Numerous packages need fixing, and we encourage the community to have a look and lend the maintainers a hand.

Extending OBuilder's capabilities to include FreeBSD as a Tier 1 platform is a crucial step to ensure the robustness and accessibility of OCaml-related software development. By implementing FreeBSD-specific modules for the Fetcher and Sandbox components, developers will be empowered to perform reproducible builds within a sandboxed environment on FreeBSD. The inclusion of FreeBSD in the OBuilder ecosystem will contribute to the overall growth and adoption of OCaml, facilitating the development of reliable and efficient software on this platform.

Please get in touch with us if you are interested in FreeBSD support, and tell us what architecture/version of FreeBSD you'd like to be supported. Or better yet, get involved with the OCurrent project! We are also active on Discuss.