Announcing Unikraft Support for MirageOS Unikernels

We are happy to announce the release of Unikraft backend support for MirageOS unikernels! Our team, consisting of Fabrice Buoro, Virgile Robles, Nicolas Osborne, and me (Samuel Hym), have been working on this feature for the past year. We’re excited to bring it to the community and hope to see many people try it out.

This post will give you an overview of the release, including some background on Unikraft and performance graphs. You will already be familiar with most of this if you read our post on Discuss.

What is Unikraft and Why Did We Choose It?

Unikraft is a unikernel development kit: a large collection of components that can be combined in any configuration that the user wants in the unikernel tradition of modularity. Unikraft's scope is larger than that of Solo5 (a backend that MirageOS also supports). It aims to make it easy to turn any Unix server into an efficient unikernel.

In fact, the initial motivation behind exploring Unikraft as a MirageOS backend was to experiment and see what performance levels we could reach. We were particularly excited about using their Virtio-based network interface, as virtio is currently only implemented for one specific x86_64-only backend in Solo5. Some of the immediate performance differences we observed are detailed further down, but performance is not all we hope to gain from the Unikraft backend in the long run.

Unikraft is on the road to being multicore compatible (i.e., having one unikernel using multiple cores). While this is not ready yet and significant effort is needed to get there, it means that the MirageOS backend will eventually benefit from these efforts and support the full set of OCaml 5 features.

Furthermore, the Unikraft community (which is quite active) is experimenting with a variety of other targets, such as bare-metal for some platforms or new hypervisors (e.g. seL4). Any new target Unikraft supports can then be supported ‘for free’ by MirageOS too. For example, the Unikraft backend has already resulted in Firecracker being a new supported virtual machine monitor (VMM) for MirageOS.

Lastly, since Unikraft is POSIX-compatible (for a large subset of syscalls), it has the potential to enable MirageOS unikernels to embed OCaml libraries that have not been ported yet. This compatibility could be especially useful for large libraries which are hard to port (owl comes to mind).

How Does Unikraft Support Work?

Adding the new MirageOS backend required that we create or modify a series of components:

• An OCaml cross compiler that can build the new backend by building its corresponding runtime and providing a way to build unikernel images (instead of normal executables).
• New libraries for Unikraft system support, including its network and block devices.
• Support for the new backends in the mirage tool.

Using Unikraft with a QEMU or a Firecracker backend is as simple as choosing the unikraft-qemu or the unikraft-firecracker target when configuring a unikernel.

The OCaml/Unikraft Cross Compiler

To build the OCaml cross compiler with Unikraft, we used the Unikraft core, Unikraft lib-musl, and musl. The musl library is the C library recommended by Unikraft for building programs using the POSIX interface. The combination made it easy to build the OCaml 5 runtime, particularly because it provided an implementation of the pthread API, which is now used in many places in the runtime. It could also potentially make it easier to port some libraries that depend on Unix to work on Unikraft backends.

The OCaml cross compiler builds upon the work that has been upstreamed to ease the creation of cross compilers, using almost the same series of patches as for ocaml-solo5. So the only versions of the compiler that are currently supported for OCaml/Unikraft are OCaml 5.3 and 5.4. All the patches have been upstreamed to OCaml so there should no longer be any patches required by OCaml 5.5.

Note that we didn’t go with the full standard Unikraft POSIX stack, which includes lwIP to provide network support. We had a prototype at some point relying on lwIP to validate our progress on other building blocks, but it raised many incompatibility issues with the standard MirageOS network stack, so we dropped support for lwIP for now. Instead, we developed the libraries required to plug the MirageOS stacks into the low-level interfaces provided by the Unikraft core.

The New MirageOS Libraries for Unikraft Support

Unikraft support comes with packages using the standard names, mirage-block-unikraft and mirage-net-unikraft, to support the block and network devices. These libraries are implemented directly on top of the low-level Unikraft APIs, and therefore use virtio on both QEMU and Firecracker VMMs.

To evaluate the quality of the implementations for these devices, we ran a couple of small benchmarks using OCaml 5.3 and Unikraft 0.18.0. You can find the benchmarks (including the unikernels along with some scripts to set them up and run them) in the benchmarks directory in @Firobe’s fork of mirage-skeleton, benchmarks branch.

Network Device

To measure the performance of the network stack, we tweaked the simple network skeleton unikernel to compute statistics and used a variable number of clients all sending 512MB of null bytes. We have run this benchmark on both a couple of x86_64 laptops and on an LX2160 aarch64 board, all running a GNU/Linux OS.

We have observed a lot of variability in the performance of the solo5-spt unikernel (sometimes better, sometimes worse than unikraft-qemu) depending on the actual computer used, so these measurements should be taken with a grain of salt.

On two different x86_64 laptops:

On the LX2160 aarch64 board:

Block Device

To measure the performance of the block devices, we wrote a simple unikernel copying data from one disk to another. We can see that the performance of unikraft-qemu is lower than solo5-hvt for small buffer sizes, but fortunately, the situation improves with larger buffer sizes. We only ran this benchmark on an x86_64 laptop, as there is currently an issue with two block devices on aarch64 on Unikraft.

It is worth mentioning that I/Os can be parallelised, which also yields a significant performance boost. Indeed, mirage-block-unikraft can leverage the parallelised virtio backend of QEMU and Firecracker, which solves the problem of limiting I/Os to what the hardware supports in terms of both parallelism and sector size.

Current Limitations

1. In our tests, only Linux appeared to be well supported for compiling Unikraft at the moment. As a result, we have restricted our packages to that OS for now.
2. Unikraft supports various backends. At the moment, we’ve only added support and tested its two major ones: QEMU and Firecracker.

Try it Out

To try the new Unikraft backend for MirageOS, you need to use an OCaml 5.3 or 5.4 switch, so that you can install mirage and the OCaml/Unikraft cross compiler. The short version could be:

$ opam switch create unikraft-test 5.4.0 # or 5.3.0, and if needed
$ opam install mirage ocaml-unikraft-backend-qemu ocaml-unikraft-x86_64

See below for some explanations about the numerous OCaml/Unikraft packages. From then on, you can follow the standard procedure (see how to install MirageOS and how to build a hello-world unikernel) to build your unikernel with the Unikraft backend of your choice, which should boil down to something like:

$ mirage configure -t unikraft-qemu
$ make

Details About the Various Packages for the OCaml/Unikraft Cross Compiler

The OCaml cross compiler to Unikraft is split up into 14 packages (see the PR to opam-repository for more details) so that users can:

• Choose which of the backends (QEMU or Firecracker) and which of the architectures (x86_64 and arm64) they want to install, where all combinations can be installed at the same time.
• Choose which architecture is generated when they use the unikraft OCamlfind toolchain by installing one of the two ocaml-unikraft-default-<arch> packages.
• Install the ocaml-unikraft-option-debug to enable the (really verbose!) debugging messages.

Furthermore, virtual packages can be installed to make sure that one of the architecture-specific packages is indeed installed:

• ocaml-unikraft can be installed to make sure that there is indeed a unikraft OCamlfind toolchain installed.
• ocaml-unikraft-backend-qemu and ocaml-unikraft-backend-firecracker can be installed to make sure that the unikraft OCamlfind toolchain supports the corresponding backend.

Those virtual packages will be used by the mirage tool when the target is unikraft-qemu or unikraft-firecracker.

All those packages use one of two version numbers. The backend packages use the Unikraft version number (0.18.0 and 0.20.0 have been tested and packaged) while the latest OCaml cross-compiler packages use version 1.1.0.

Conclusion

We are still experimenting with this new backend. We expect to run it in production in the coming months, but it may need improvements nevertheless. Notably absent from this release is an early attempt to leverage Unikraft’s POSIX compatibility to implement Mirage interfaces instead of hooking directly to Unikraft’s internal components. This early version used Unikraft’s lwIP-based network stack instead of Mirage’s (fooling Mirage into thinking it was running on Unix), and it may be interesting to revisit this kind of deployment, in particular for easy inclusion of Unix-only OCaml libraries in unikernels.

We are eager for reviews, comments, and discussion on the implementation, design, and approach of this new Mirage backend, and hope it will be useful to others.

You can connect with us on Bluesky, Mastodon, Threads, and LinkedIn or sign up for our mailing list to stay updated on our latest projects. We look forward to hearing from you!