Announcing ciao-lwt: A Library for Migrating Lwt to Eio

The I/O library Eio, which uses effects and direct-style concurrency, was released in 2024. Since then, users have seized the opportunity to test it in their own projects, and several OCaml devs have ported applications to Eio.

Now, with a new library called ciao-lwt, users can automate part of the migration process from Lwt to Eio. One of our engineers, Jules Aguillon, has been developing the library and using it for the Ocsigen project. This post shares his work and will introduce you to ciao-lwt, how to try it, and what limitations you should expect.

The project is made possible thanks to a grant from the NLnet Foundation, which funds research and development projects furthering internet technologies and the open internet, and the NGI Zero Core fund of the European commission.

Why Would I Switch to Eio?

Ultimately, the concurrency library you choose comes down to a matter of taste, but Eio has some nice characteristics that you may find worth the switch. Since it is direct-style, Eio does not require you use a monad for concurrency, which gets rid of the so-called ‘function colouring problem’. The resulting code is faster, less complex, and has some nice security capabilities.

You can read our blog post on Eio 1.0 for more context.

Ciao-Lwt

The library contains a collection of tools for translating an Lwt library into Eio code. Lwt marks concurrent code and non-concurrent (sync and async) code using bind operators or bindings, and functions must be explicitly marked as sync or async for the program to run. These are the ‘bind’ and ‘map’ operators, including Lwt.bind, Lwt.map, let*, let+, as well as the infix operators >>= and >>|.

The first step in turning Lwt into Eio code is to get rid of the bind operators. They weave through every part of Lwt code and are time-intensive to remove one by one. Ciao-lwt can automate the process and remove the bindings for you. However, the library has some limitations, which will be explained in more detail below.

At the moment, the library contains the following tools:

• lwt-ppx-to-let-syntax: Removes instances of lwt_ppx and replaces them with Lwt library function calls,
• lwt_lint: Finds implicit forks in Lwt code,
• lwt-log-to-logs: Rewrites files containing Lwt_log and migrates them to use Logs,
• lwt-to-direct-style: Finds and rewrites Lwt and other Lwt modules, turning them into Eio code instead.

Lastly, something to bear in mind is that ciao-lwt uses Merlin's index to locate every use of Lwt.

How Do I Try It?

To get started with ciao-lwt, the first thing to do is visit the repo and install the tools in the opam switch you’re using to build your projects using the command:

opam install ciao_lwt lwt_lint lwt_ppx_to_let_syntax

To make reviewing the change easier, make sure your code is formatted. The tool will entirely reformat the file it touches, which may make actual changes harder to see.

The first step is to remove any use of lwt_ppx (for example the let%lwt syntax):

lwt-ppx-to-let-syntax .
dune fmt # Remove formatting changes created by the tool

This operation is purely syntactical, the tool simply walks the given directory tree and parses every .ml files it finds, updating the files that contain usages of lwt_ppx.

Before running the next tool, try eliminating common causes of implicit forks:

lwt-lint .

This operation is also purely syntactical. The tool warns about every occurrence of let _ = .. and ignore that doesn’t have a type annotation. This helps you find cases where an Lwt promise is disregarded. To silence each warning, add a type annotation, for example: let _ : my_t = .. and ignore (.. :my_t).

If you use Lwt_log, you can migrate to Logs easily with:

dune build @ocaml-index # Build the index (required)
ciao-lwt to-logs --migrate .
dune fmt # Remove formatting changes created by the tool

This tool works similarly to ciao-lwt to-eio described below. It is provided as a separate command because your program will likely work as before but it lets you review this step independently and it simplifies the next step.

Finally, migrate to Eio:

dune build @ocaml-index # Build the index (required)
ciao-lwt to-eio --migrate .
dune fmt # Remove formatting changes created by the tool

This operation migrates the common uses of Lwt, but the transition is not yet complete.

Limitations & Considerations

Ciao-lwt is still considered experimental and a work-in-progress, which you should bear in mind when you try it. Your feedback and input is very welcome and will help the team improve the tools.

It sounds obvious, but as a promise-based concurrency library, Lwt creates a lot of promises. Everything that is concurrent in Lwt is a promise; it specifies actions that will happen at a later time. Some promises are so-called ‘implicit forks’, which do not use the bindings we mentioned earlier.

Let's look at an implicit fork:

let _ =
  let a = operation_1 () in
  let* b = operation_2 () in
  let* a = a in
  Lwt.return (a + b)

Here, let a = operation_1 () in 'forks', meaning it creates a concurrent thread. Since there are no binding operators or Lwt function calls, ciao-lwt can't detect this fork syntactically or with Merlin's index.

As a result, while Lwt would run operation_1 and operation_2 concurrently, after ciao-lwt converts it to Eio it would instead run sequentially:

let _ =
  let a = operation_1 () in
  let b = operation_2 () in
  let a = a in
  a + b

Users need to be aware of how ciao-lwt handles 'implicit forks' so they can fix bugs introduced in the migration.

The most helpful tool to verify your new code is the OCaml compiler. Your resulting code will likely not typecheck, and OCaml's typechecker can guide you towards the manual changes you will need to make. It's not foolproof, and you will still need to be on the lookout for concurrency bugs, but ciao-lwt's tools in combination with the OCaml compiler will give you a nice head start on your journey from Lwt to Eio.

Until Next Time

Tarides remains committed to creating new tools that make new and old workflows easier. We hope ciao-lwt proves useful to you, and appreciate any feedback you have to share.

You can connect with us on Bluesky, Mastodon, Threads, and LinkedIn or sign up for our mailing list to stay updated on our latest projects. We look forward to hearing from you!