Irmin v2

We are pleased to announce Irmin 2.0.0, a major release of the Git-like distributed branching and storage substrate that underpins MirageOS. We began the release process for all the components that make up Irmin back in May 2019, and there have been close to 1000 commits since Irmin 1.4.0 released back in June 2018. To celebrate this milestone, we have a new logo and opened a dedicated website: irmin.org.

Our focus this year has been on ensuring the production success of our early adopters -- such as the Tezos blockchain and the Datakit 9P stack -- as well as spawning new research projects into the practical application of distributed and mergeable data stores. We are also very pleased to welcome several new maintainers into the Mirage project for their contributions to Irmin, namely Ioana Cristescu, Craig Ferguson, Andreas Garnaes, Clément Pascutto and Zach Shipko.

New Major Features

New CLI

While Irmin is normally used as a library, it is obviously useful to be able to interact with a data store from a shell. The irmin-unix opam package now provides an irmin binary that is configured via a Yaml file and can perform queries and mutations against a Git store.

$ echo "root: ." > irmin.yml
$ irmin init
$ irmin set foo/bar "testing 123"
$ irmin get foo/bar

Try irmin --help to see all the commands and options available.

Tezos and irmin-pack

Another big user of Irmin is the Tezos blockchain, and we have been optimising the persistent space usage of Irmin as their network grows. Because Tezos doesn’t require full Git format support, we created a hybrid backend that grabs the best bits of Git (e.g. the packfile mechanism) and engineered a domain-specific backend tailored for Tezos usage. Crucially, because of the way Irmin is split into clean libraries and OCaml modules, we only had to modify a small part of the codebase and could also reuse elements of our OCaml-git codebase as well.

The irmin-pack backend is available for use in the CLI and provides a significant improvement in disk usage. There is a corresponding Tezos merge request using the Irmin 2.0 code that has been integrated downstream and will become available via their release process in due course.

As part of this development process, we also released an efficient multi-level index implementation (imaginatively dubbed index in opam). Our implementation takes an arbitrary IO implementation and user-supplied content types and supplies a standard key-value interface for persistent storage. Index provides instance sharing by default, so each OCaml runtime shares a common singleton instance.

Irmin-GraphQL and “browser Irmin”

Another new area of huge interest to us is GraphQL in order to provide frontends with a rich query language for Irmin-hosted applications. Irmin 2.0 includes a built-in GraphQL server so you can manipulate your Git repo via GraphQL.

If you are interested in (for example) compiling elements of Irmin to JavaScript or wasm, for usage in frontends, then the Irmin 2.0 release makes it significantly easier to support this architecture. We’ve already seen some exploratory efforts report issues when doing this, and we’ve had it working ourselves in Irmin 1.0 Cuekeeper so we are excited by the potential power of applications built using this model. If you have ideas/questions, please get in touch on the issue tracker with your usecase.

Wodan

Irmin’s storage layer is also well abstracted, so backends other than a Unix filesystem or Git are supported. Irmin can run in highly diverse and OS-free environments, and so we began engineering the Wodan filesystem as a domain-specific filesystem designed for MirageOS, Irmin and modern flash drives. See the OCaml Workshop 2017 abstract on it for more design rationale.

As part of the Irmin 2.0 release, Wodan is also being prepared for a release, and you can find Irmin 2.0 support in the source. If you’d like a standalone block-device based persistence environment for Irmin, please try this out. This is the preferred backend for using Irmin storage in a unikernel.

### Versioned CalDAV

An application pulling all these pieces together is being developed by our friends at Robur: an Irmin-based CalDAV calendaring server that even hosts its DNS server using a versioned Irmin store. We'll blog more about this as the components get released and stabilised, but the unikernel enthusiasts among you may want to browse the Robur unikernels future branch to see how they are deploying them today.

A huge thank you to all our commercial customers, end users and open-source developers who have contributed their time, expertise and financial support to help us achieve our goal of delivering a modern storage stack in the spirit of Git. Our next steps for Irmin are to continue to increase the performance and optimise the storage, and to build more end-to-end applications using the application core on top of MirageOS.