Tarides

OCaml-CI Renovated

by Navin Keswani and Tim McGilchrist on Jul 12th, 2023

OCaml-CI started with the goal of making a better continuous build system for OCaml projects. When we began in 2019, the goals were clear: it should provide a zero-configuration experience for OCaml projects using opam and Dune, and it should use an incremental architecture to avoid expensive recomputation of builds. We're delighted to announce that we achieved these goals, and OCaml-CI is currently tracking over five hundred repositories and processing over a hundred thousand jobs daily. This is inspiring news to those already using OCaml-CI or developers looking for a CI solution for their OCaml project.

Throughout 2022, the Tarides CI team worked on rennovating OCaml-CI, focusing on improving the usability of the website, adding build history for branches, supporting new platforms, and launching experimental build support. We will cover all of those things in this blog post and hope you find them useful.

There is also a Discuss thread on CI Best Practices.

What is OCaml-CI?

Continuous Integration, or CI, performs a series of automated steps (or jobs), e.g., building and testing code. With it, developers can confidently and regularly integrate code into the central repository, relying on the automated CI system to detect and even fix problems early. This reduces production issues and leads to more robust and secure software. OCaml-CI is a Continuous Integration tool tailored for OCaml projects.

VALUE PROPOSITION OF OCAML-CI

  • OCaml specific / no configuration
  • Check on various platforms
  • Linting like version bounds (upper and lower) and project metadata
  • Incremental caching of builds

OCaml-CI adds value by targeting just OCaml projects that are written with the standard OCaml opam tooling for package management and Dune for building code. Because OCaml-CI targets a specific language, it does not require any configuration. Instead, it derives the necessary information from metadata in the opam and dune project files.

You never have to teach OCaml-CI about how to build OCaml!

Additionally, it can do clever things like linting your opam and dune files to check for common mistakes. It also checks the upper and lower version bounds of packages to see where they break. The biggest feature over most popular CI systems is that OCaml-CI can derive which hardware and operating system platforms a package supports and do builds across all of those platforms! That means if you want to check on Linux ARM64 or MacOS x86_64 or Linux s390x you can. All that comes with the added benefit of a caching strategy based on the incremental architecture, which won't repeat builds if not necessary.

Lets look at the new features that were added!

Redesigned UI Using Dream

When OCaml-CI came into existence in 2019, the focus was on providing a CI system based on an incremental architecture, so the user interface (UI) for OCaml-CI was kept simple. In 2022, the team worked with a designer to develop a consistent and contemporary theme to make the site look and feel like a modern website. We decided on a tech stack of Dream for the web framework, Tailwind CSS for styling, Tyxml for HTML generation, and Omigrate for database migrations. In the Design and Implementation section below, we cover the technical reasons for each choice.

Build History

Over the years, people consistently requested a provided build history. So we added a new history page to show the build history of a branch. This feature allows users to conveniently access and view historical builds in the context of similar builds of the branch. It shows every commit built by OCaml-CI, a summary of each build (including build status, the time at which the build started, and the running time of the build), and links to the commit's build page.

Live Updates

Pages now automatically update with new information as the build progresses! OCaml-CI adds build steps as they are created, so build statuses and runtimes all update as the build occurs.

In 2023, it’s unusual to have to refresh a page to update the information, so this is just us catching up!

Enriching Build Information

Timestamps and durations relevant to a build and each of its steps are now available. This feedback enables development teams to monitor the resources and time taken for their builds. It gives them what they need to identify bottlenecks and opportunities for faster build times.

Summary of a Repository's Health

When looking at an organisation's page, you will now see a summary of the default branch for each of your repos. Inspired by the UI of BuildKite, we hope to provide teams with a view of their builds that indicates the overall health of their repository. The chart of the last 15 builds makes anomalous builds easy to identify and investigate.

Mobile Version

OCaml-CI can now be conveniently used from a variety of devices. We have rewritten our pages to be responsive to mobile devices, choosing to pare information to the essentials for small screens. All the functionality of the main site is still available on the mobile version, so you can view logs or navigate to the GitHub PR for a build.

Mobile Screenshot

Experimental Builds

Experimental builds was a conceptual feature that was added to OCaml-CI in order to support build types that might not be stable or to introduce ones without breaking CI for all projects. Think of them as a kind of feature flag. Experimental builds are clearly labelled with (experimental) in the UI and will not report as failures if every other build passes. They let us boldly introduce new features like supporting new platforms or running new linting checks like package lower bounds.

macOS Experimental Builds

Using Experimental builds, we added support for macOS (both x86_64 and ARM64) to OCaml-CI. These builds will run on the latest two versions of OCaml on both architectures. Currently these are marked as experimental as we work towards making macOS builds more efficient. The path to supporting macOS has been a long one, starting back in late 2021, and has gone through two different implementations before reaching a stable state in early 2023. We have plans to publish a post going deeper into the technical details soon.

Design and Implementation

The story of why technical decisions are made on a project are often as interesting as the project itself. Here we will go through the thoughts about the technologies used and why.

As previously mentioned, we kicked off this work with a designer to help develop a consistent and contemporary theme for the site. They created a set of designs in Figma and also made example HTML pages using Tailwind CSS as a basis for the style sheets. Tailwind works by scanning HTML files, JavaScript components, and any other templates for class names to generate the corresponding styles and then write them to a static CSS file. There is an opam package tailwindcss that wraps this all up for us.

We decided to use Dream to replace OCaml-CI’s previous web layer based on Cohttp library for the following reasons:

  • Cohttp is a low-level library, so we had to hand-roll solutions to standard patterns that are generally provided by web frameworks. For example, we had to solve the CSRF problem consistently throughout our usage of forms and also construct our own solution to show flash messages. We understood Cohttp but were interested in taking the opportunity to investigate other web frameworks in the OCaml landscape.
  • Inspired by frameworks like Sinatra (of Ruby fame) and Flask (from Python), our colleagues Rudi Grinberg and Thibaut Mattio (and others) had constructed a web framework called Opium. They suggested that we check out Dream.
  • We were impressed by its elegance and polish. Dream has brilliant documentation, a ton of examples, and convenient functions and support for several standard patterns in web development. It also uses common OCaml types, so adding it to the project would be relatively straightforward. We immediately saw an opportunity to accomplish several things at once:
    • Support a promising project by adopting it and contributing to it
    • Create a non-trivial example of using Dream for the community
    • Reduce complexity, modernise the UI, and make it easier to add new features
    • Have a lot of fun!

As we began to work with Dream, we made the following choices:

  • We chose to work with TyXML over Eml so that we would have the guardrails of typed templates to help write correct HTML. This proved to be challenging in the beginning, but examples from the Opium project really helped our team figure out how to wield TyXML correctly.
  • Our team did not have any CSS expertise and, frankly, was a little bit at sea with how to implement some of our designer's suggested designs. Tailwind CSS really helped us out here. In particular, it made it possible for us to achieve responsiveness for different screens and light and dark modes.
  • For our database layer, we chose to work with Omigrate, so we could introduce migrations and develop our information model with confidence.

We are working on improving the signup process and on introducing js_of_ocaml to replace the plain JavaScript that we previously introduced.

We’d Love Your Feedback

If you have an OCaml project hosted on GitHub or GitLab and would like to test drive OCaml-CI, please follow our getting-started guide. There are many popular projects already using OCaml-CI to improve their development, and we want to see your project too.

Please open an issue on https://github.com/ocurrent/ocaml-ci if you run into any problems or to suggest improvements and point out missing features. The Tarides team wants to support more OCaml platforms like Windows and FreeBSD so we can cover the full OCaml Tier 1 supported platforms and to continue improving the UI experience.

If you are curious about web development in OCaml, we recommend checking out Dream. Please use our code for reference and ask us questions or make suggestions for improvements. We are using the following technologies:

  • current_incr - Self-adjusting computations
  • OCurrent - a CI/CD pipeline OCaml eDSL
  • Lwt - OCaml promises and concurrent I/O
  • Tyxml - Typed HTML and SVG
  • Capnp_rpc - OCaml Cap'n Proto RPC library

If we can learn and improve from your experience, we all win! Thank you!

Acknowledgements

The Tarides engineers that delivered this work are Étienne Marais, Ben Andrew, and Navin Keswani. We got much support and feedback from several of our Tarides colleagues and others in the OCaml community, and we are very grateful for all we learned from them. Special mention to Thibaut Mattio and Tim McGilchrist.