OCaml 5.3: Features and Fixes!

We have a brand new OCaml release on our hands! 5.3 comes packed with features, fixes, and optimisations, including the return of some ‘familiar faces’. Support for the MSVC port is returning, as is statistical memory profiling now compatible with multicore projects.

This post highlights new and restored features, notable changes and user experience improvements, plus some bug fixes. There is no way that I can cover everything in this update, so I recommend that you check out the Changes document on GitHub for the full list of contributions!

MSVC

The 5.3 release restores support for the MSVC port of OCaml on Windows, marking the last remaining platform from 4.x to regain support in 5.x. This is part of a wider effort to achieve feature parity between OCaml 4.14 and OCaml 5, of which compaction is a previous example, making the transition between versions as smooth as possible. The bulk of the effort is summarised in PRs #12954 and #12909 opened by David Allsopp, Antonin Décimo, and Samuel Hym (review by Miod Vallat and Nicolás Ojeda Bär).

Since the OCaml 5 runtime uses C11 atomics, supported platforms need to be compatible with them as well. Visual Studio 2022 introduced experimental support for C11 atomics which made the MSVC port of OCaml 5 possible, but the team needed to test out the feature first. This exploratory effort led to several bug reports addressed by Microsoft, and once these were completed (alongside a lot of other work, including fixing the winpthreads library of the mingw-w64 project to that it builds with MSVC), the MSVC port was ready for public release.

As part of the project bringing MSVC back the team explored clang-cl, an alternative command line interface to Clang designed to be compatible with the MSVC compiler cl.exe. This was helpful because clang-cl has a different set of warnings and tips to MSVC, and using it effectively gave them a ‘second opinion’ on their code. The main PR for this side of the project is #13093.

Statmemprof

OCaml 4.14 had support for statistical memory profiling, a feature of the language that can sample memory allocations allowing tools like Memtrace to help users identify how their programs are using memory. The multicore update introduced significant complexity to the process which made it necessary to drop support for 5.0; but work soon commenced to restore support under our feature parity banner! In 5.3, statmemprof makes its return, now equipped with multicore capabilities.

So how does it work? Statmemprof can check the allocation of memory at some given frequency (lambda) per word or unit of data. By sampling a fraction of allocations at random, we are able to monitor programs in a language like OCaml which allocates high rates of memory. It would be far too expensive performance-wise to monitor every allocation.

The new design has a lot in common with the OCaml 4 implementation of statmemprof, but with several tricky optimisations and changes to account for the significant complication of multiple domains and threads. Delve into the details in the PRs #12923 and #11911 by Nick Barnes (external reviews by Stephen Dolan, Jacques-Henri Jourdan, and Guillaume Munch-Maccagnoni).

Deep Effect Handlers

OCaml 5.0 came with experimental support for algebraic effects, which allow users to describe computations and what effects they are expected to create. A handler essentially manages a computation by monitoring its execution and keeping track of resulting effects. This ‘management’ can be done in two ways, deeply or shallowly. A shallow effect handler monitors a computation until it either terminates or generates one effect, only handling that effect. A deep effect handler always manages a computation until it terminates and handles all of the effects performed by it.

PR #12309 (Leo White, Tom Kelly, Anil Madhavapeddy, KC Sivaramakrishnan, Xavier Leroy and Florian Angeletti, review by the same, Hugo Heuzard, and Ulysse Gérard) introduces effect syntax for deep effect handlers, rules that define the structure for writing them, compatible with the type checker and with support for pattern matching. This change aims to simplify the code needed to use deep effect handlers, improving user experience. Note that you can still use shallow effect handlers, and there is a good tutorial for using both in the correspondingly updated manual page.

Debugging Improvements

Another long-term project coming to fruition in this update are the several improvements to debugging on macOS. The platform is popular with a wide variety of OCaml users, including compiler developers, and they need good debugging workflows for their programs.

LLDB is the only supported native debugger on macOS, for both the ARM64 and x86_64 architectures. The improvements enable several new features:

• #13163 enable frame pointers on macOS x86_64 (Tim McGilchrist, review by Sébastien Hinderer and Fabrice Buoro): This PR introduces support for a common technique used by profiling tools including Linux perf, eBPF, FreeBSD, and LLDB, called stack-walking. Various performance tools use stack walking to reconstruct call graphs for programs, and frame pointers are what enable them to do so.
• #13241, #13261, #13271, add CFI_SIGNAL_FRAME to arm64 and RISC-V runtimes for the purpose of displaying backtraces correctly in GDB (Tim McGilchrist, review by Miod Vallat, Gabriel Scherer and KC Sivaramakrishnan): This change helps sync up the runtime for the arm64 architecture for macOS (and the RISC-V runtime) with the amd64 and s390x runtimes. The two additional PRs add improvements to the first.
• #13136 Compatible LLDB and GDB Python extensions (Nick Barnes): This PR replaces some old GDB macros (used to debug OCaml programs) with faster and more capable extensions, and makes those extensions available in LLDB. This is especially useful to macOS users who can’t use GDB.

OS-Based Synchronisation for Stop-the-World Sections

PR #12579 (B. Szilvasy, review by Miod Vallat, Nick Barnes, Olivier Nicole, Gabriel Scherer and Damien Doligez) improves user experience by replacing generic busy-wait synchronisation with OS-based synchronisation primitives, namely barriers and futexes. The change has significant performance benefits, especially on Windows machines, where spinning was causing long wait times. You can learn more about it in our blog post on the project.

User Experience Improvements

• #12868 Refresh HTML manual/API docs style (Yawar Amin, review by Simon Grondin, Gabriel Scherer, and Florian Angeletti): An update to the OCaml Manual which simplifies the colours, removes the gradients, and fixes the search button. It’s a nice improvement to a part of the OCaml ecosystem that is visible to users of all different backgrounds and contexts.
• #13201, #13244 (Sébastien Hinderer, review by Miod Vallat, Gabriel Scherer and Olivier Nicole), and #12904 (Olivier Nicole, suggested by Sébastien Hinderer and David Allsopp, external review by Gabriel Scherer) various improvements to TSan: These three PRs represent the continuous work being put in to bring improvements to TheadSanitizer or TSan. They include speedups and the ability for users to choose which PRs they want to run the TSan testsuite on.
• #13014 (Miod Vallat, review by Nicolás Ojeda Bär) add per function sections support to the missing compiler backends: This PR is an example of how much focus there is on ensuring that each native backend is equally supported, having features available across all Tier-1 platforms. Here, the compile-time option function–sections was re-enabled on all previously unsupported (POWER, riscv64, and s390x) native backends.
• #11996 emancipate dynlink from compilerlibs (Sébastien Hinderer and Stephen Dolan, review by Damien Doligez and Hugo Heuzard): The dynlink library used to depend on compilerlibs, having to embed a copy of compilerlibs meaning that it would be compiled twice, costing the user in time and performance. After the change, the build time and size of both dynlink.cma and dynlink.cmxawere reduced.

Miscellaneous Bug Fixes

These two bug fixes grew out of internship projects at Tarides, it's great to see how these projects can benefit the language as a whole.

• #13419 (B. Szilvasy and Nick Barnes, review by Miod Vallat, Nick Barnes, Tim McGilchrist, and Gabriel Scherer): This PR addressed resource leaks that caused memory bugs in the runtime events system.
• #13535 (Antonin Décimo, Nick Barnes, report by Nikolaus Huber and Jan Midtgaard, review by Florian Angeletti, Anil Madhavapeddy, Gabriel Scherer, and Miod Vallat): Expanded the documentation for Hashtbl.create to explain that negative values are allowed in the hash table but will be disregarded.

These bug fixes stem from discoveries made during the release cycle of the 5.3 update. Catching and fixing broken bits of code is an important but often lengthy part of the release process.

• #13138 (Gabriel Scherer, review by Nick Roberts): This PR is an old one, first opened eight years ago in 2016! Optimised pattern matching with mutable and lazy patterns was observed to result in occasions where seemingly impossible cases were taken, causing unsoundness issues. After lengthy efforts to narrow down the cause, the problem has been fixed for 5.3!
• #13519 (Sébastien Hinderer, report by William Hu, review by David Allsopp): This PR restored backward compatibility lost when renaming some items in Makefile.config.
• #13591 (Antonin Décimo, review by Nick Barnes, report by Kate Deplaix): This PR fixed a problem whereby compiling C++ code using the OCaml C API resulted in a name-mangled caml_state on Cygwin. The fix ensured that installed headers were compatible with C++ and protected the ones that were not with CAML_INTERNALS.
• #13471 (Florian Angeletti, review by Gabriel Scherer): Added a flag to define the list of keywords recognisable by the lexer, making adding future keywords to OCaml easier.
• #13520 (David Allsopp, review by Sébastien Hinderer and Miod Vallat): Fixed the compilation of native-code versions of systhreads.

What’s Next?

Work on OCaml continues! The next few months will bring more features and bug fixes to the language, with focus on big changes like the relocatable compiler, unloadable runtime, and laying the ground work for project-wide renaming and other powerful navigation and refactoring features. The OCaml changelog is the place to go to keep up with what’s new, as well as the OCaml Discuss forum.

You can connect with us on Bluesky, X, Mastodon, Threads, and LinkedIn or sign up for our mailing list to stay updated on our latest projects. We look forward to hearing from you!