Feature Parity Series: Restoring the MSVC Port

After the release of OCaml 5, restoring any features that were left out of the initial release has been a high priority for our teams and collaborators. We call this effort our 'feature parity' project, and compaction is one example of a feature being brought back to OCaml 5 under its banner.

In this post, we look at another returning property, MSVC support, and the steps along the path to implementation. If you want to skip straight to the code, check out the #12954 pull request (and the dozen more linked from it!) in the OCaml repo. Let's dive in!

MSVC Support for OCaml on Windows

First, let's explain what 'MSVC support' means. In general, OCaml supports compilation to Windows through three separate toolchains: Cygwin, mingw-w64, and MSVC. The mingw-w64 toolchain was available for OCaml 5 from the moment the update launched. Cygwin was restored in OCaml 5.1, but MSVC support has lagged behind until now!

The delay stemmed from MSVC's initial incompatibility with C11 atomics, which the OCaml 5 runtime requires. David Allsopp had been exploring possible ways to overcome this incompatibility, testing how C++ atomics worked with older compilers. Eventually, however, Microsoft introduced support, albeit experimentally.

To restore the port, the team needed to ensure that the C11 atomics support was reliable, port winpthreads onto MSVC, and create a continuous integration (CI) workflow.

Since it is Microsoft's own C/C++ compiler, MSVC is popular and well-known to many developers and Windows users. Bringing compatibility with the compiler to OCaml 5 is an important step towards enabling more users to adopt the latest version of OCaml and explore its new features!

C11 Atomics and Bug Reports

C11 is a version of the C standard. Since the OCaml runtime is written almost entirely in C, the general standard lets us specify which properties of the C compiler can be used to build OCaml. If we didn't rely on a standard, we would have to list supported C compiler versions individually (GCC from one such version, Clang from another such version, etc). Instead, developers know that the OCaml runtime environment supports any C compiler that is C11-compliant.

For OCaml 5 and onwards, the C compiler must be C11 compliant and support C11 atomics. All we need to know about atomics for this post is that the C11 atomic spec enables the compiler to help the programmer access data that can be shared between multiple cores. Without it, the developer would need to use other synchronisation mechanisms such as mutexes, that require more code, both to write and to run. So C11 atomics go beyond what most of us associate with atomicity and are key to writing sound and efficient code in a multicore setting.

Once the Visual Studio 2022 release introduced experimental support for C11 atomics, it provided a much clearer path for the team to work on restoring MSVC support. This team included David Allsopp, Antonin Décimo, and Samuel Hym from Tarides, but of course, the success of the project relied on the collaboration, input, and reviews of many open-source OCaml community members. With C11 atomics support in Visual Studio 2022 being experimental, the team needed to ensure that all the sequential tests passed and identify places where parallel tests failed due to bugs in MSVC. The team created several bug reports against the MSVC compiler as a result of this project.

The bug reports include:

• Missing Atomic Stores When Dereferencing Pointer to Atomics: MSVC version 19.38.33128 lacked support for pointers to atomic values and was not emitting atomic stores when writing to a dereferenced pointer for an atomic variable.
• Compound Assignment Operators are not Atomics on Pointers to Atomic Values: Again, MSVC 19.38.33128 lacked support for pointers to atomic values, and this bug report highlighted that MSVC did not generate atomic code for compound assignment operators with a pointer to an atomic value.
• Pointers to Atomic Values Should be Reloaded: In version 19.38.33128 MSVC failed to generate atomic code, meaning that pointers were not reloaded when they needed to be, causing threads to spin indefinitely.

Thanks to great support from Microsoft, these bugs were resolved, and C11 atomics support was satisfactory to enable MSVC support for multicore OCaml. This was the biggest roadblock to the project's success, and with it cleared, the team turned their attention to new challenges.

Winpthreads and MSVC

The next hurdle on the road to success was another Windows-specific compatibility issue. OCaml 4.* had limited support for threading in the form of the optional systhread library, but the runtime itself made no use of it. That completely changed with OCaml 5! The abstraction used to enable threading support was Unix's posix threads, known as 'pthreads'. At the time, the runtime was prepared in the hope that a Windows version could be implemented in the future.

However, the original multicore PR could use the winpthreads part of the mingw-w64 library to provide a pthread implementation for the Windows MinGW port. The intention then was that it would be a temporary workaround allowing all the existing pthreads code to be reused, partly due to the belief that it would only work for mingw-w64 and not for MSVC.

Upon further investigation, David discovered a library demonstrating that winpthreads could be compiled with MSVC without introducing too many dependencies. Samuel and Antonin worked on formalising the process of extracting the winpthreads sources from the mingw-w64 project to use them for the MSVC port. Antonin also contributed directly to the mingw-w64 project to patch its winpthreads component.

Thanks to this work, the initially temporary winpthreads workaround has been implemented as a submodule for MSVC. This lets the new MSVC port use pthread.h via the winpthreads submodule (instead of using winpthreads implicitly as provided by the mingw-w64 GCC compiler).

The future of pthreads in OCaml is still up for discussion, with one school of thought being that reimplementing OCaml's use of pthreads in a more abstract way would allow its primitives to function without the full weight of the POSIX spec, resulting in better performance. Work has started to remove winpthreads and use modern Windows APIs for the MSVC and MinGW-w64 ports.

CI

Finally, the team added a continuous integration workflow enabling GitHub Actions for MSVC testing.

As most OCaml compiler developers use a different port than MSVC, and since there are many differences between MSVC and the other ports, being able to test MSVC in CI helps the developer be confident that their modifications do not break the code. In particular, part of the CI workflow includes a check to make sure the assembly used in the MSVC port (meaning that it's written in MASM syntax) is kept consistent with the assembly used in the MinGW port (written in GNU Assembler syntax). This check has already allowed OCaml core developers to catch a few PRs that only updated the assembler in GNU syntax, catching a problem early and preventing it from affecting the program.

A Note on the Unloadable Runtime

Before we leave you, let's briefly review another feature parity project, the return of the unloadable runtime. This project was paired with the return of the MSVC port as two features that were important to the community. The 'unloadable runtime' is a feature that cleans up OCaml resources, including the stack, heap sections, code fragments, buffers, tables, and more, when OCaml is used as a shared library. For example, if a host program uses OCaml as a library, when control returns to the host program, the unloadable runtime ensures proper resource cleanup.

The return of this feature was requested by the community, and our team worked hard to make the restoration a reality. The PR associated with this effort is #12964, which you can check out to learn more about the process behind the changes. The PR has been merged and is expected to be released with the 5.4 update.

Stay in Touch!

Keep an eye out for future updates on restored features on our blog. For a broader overview of the 5.3 update, you can check out our release blog post covering the changes.

You can connect with us on Bluesky, Mastodon, Threads, and LinkedIn or sign up for our mailing list to stay updated on our latest projects. We look forward to hearing from you!