Irmin in the Browserby Odinaka Joy on Aug 2nd, 2022
Over the past six months, I have been working on using Irmin in the browser, including
irmin-server and the GraphQL interface. This has been fun and a great learning journey for me. Before this internship,
irmin-server was primarily a Unix-based application. My project was to port
irmin-server to work in the browser and design interfaces for people to interact with the store (Irmin stores).
I was paired to work with Patrick Ferris as my mentor and with the entire Irmin team, who all contributed immensely to this project.
Irmin is simply a data store (database). It is based on the same design principle as Git with features to merge and branch data stores. Irmin has several stores (
irmin-git) and store interfaces (
irmin-server is a high-performance server for Irmin. For efficient communication, it implements a specialised wire-protocol to send and receive data over a bytestream. It wraps an Irmin store, providing a way to connect to the server and access the store via its API using a client. But the client makes an assumption that the user is on a Unix machine, which makes
irmin-server primarily a Unix-based application.
In this modern age, it's become a necessity to make applications "offline first." Offline-First applications function without being affected by the intermittent lack of a network connection. It usually implies the ability to sync data between multiple devices. Irmin as a data store supports multiple backends, making it very portable. Plus, Irmin's mergeable replicated data-types make it much easier to build applications that can transform the state offline and resynchronise the state later, just like Git. With this concept, resynchronising Irmin stores (from server to client) is much simpler on
irmin-server, which implements a specialised wire protocol for efficient communication. Making
irmin/irmin-client work in the browser simply means that it would be possible to create offline-first web applications.
More information on offline-first applications can be found here
An initial summary of the problem was published on this issue, but here is a quick breakdown of the problems we identified.
irmin-serverwas tightly coupled around
irmin-serverwas initially designed to be a Unix-based application that established communication with a client via
conduit-lwt-unix. This became a problem because
conduit-lwt-unixcannot establish a communication from a browser. This meant that there was a need to abstract the I/O module so that every client will provide its I/O.
- Reuse some internal modules: We needed to reuse the
irmin-serverinternal logic related to the protocol but provide a portable I/O interface that can work in the browser.
- Provide a browser communication channel: We needed a non-blocking way to establish a channel to create communication between
irmin-serverand the browser, and also pass data across this channel.
Thanks to Zach Shipko, who abstracted the I/O library and split out
irmin-client-cli to have their own I/O module that depends on
conduit-lwt-unix (here), a client can connect to a running
irmin-server using its own I/O module. While he was working on the restructuring, I spent my time working on a sample project that combines
irmin-graphql (more on this project).
With the coupling out of the way, the next step was to create
irmin-client-jsoo, a browser client with its own I/O module.
irmin-server initial architecture had to be restructured to accommodate other platforms. To achieve this,
irmin-client was no longer coupled with a specific I/O implementation. Rather, a Unix-based one was provided over conduit flows, which are
Lwt_io input and output channels. This channel was established over a TCP connection or a Unix domain socket.
irmin-server can communicate with two (2) clients:
irmin-client-cli from a command line and
irmin-client-unix from a Unix-based machine. This project was about creating a third client:
irmin-client-jsoo, to be called from browser applications.
After considering other options to create a communication channel for
irmin-client-jsoo, like HTTP, RPC, etc., Patrick suggested WebSocket, so we decided to go with WebSocket, a bidirectional communication protocol between client and server.
irmin-server uses flows to communicate between the server and the client and flows are bytestreams. WebSocket provides a bidirectional communication channel in the browser, but it is not stream-oriented rather it is message-oriented.
TCP (Transmission Control Protocol) is a type of protocol or standard to transfer information over the Internet while WebSocket is a message-oriented application protocol, which uses TCP as the transportation layer.
The idea behind the WebSocket protocol consists of reusing the established TCP connection between a client and server. Even though WebSocket is built on TCP, the data it passes is always either sent as a whole "message" or not at all. These implementations are non-blocking.
Since we are avoiding a full redesign of the
irmin-server protocol, we had to make the message-oriented process seem like bytestreams of data.
irmin-server from the browser is very easy. You can achieve that by following these steps:
irmin-server, using this command:
opam pin add git+https://github.com/mirage/irmin-server/commit#013a28fd1507f8ba69494515533119804903aa99
- Set up the server.
open Lwt.Syntax module Store = Irmin_mem.KV.Make (Irmin.Contents.String) module Server = Irmin_server.Make (Store) let main = let uri = Uri.of_string "ws://localhost:9090/ws" in let config = Irmin_git.config "penit" in let* store = Store.Repo.v config in let* main = Store.main store in let* server = Server.v ~uri config in let () = Format.printf "Listening on %a@." Uri.pp uri in Server.serve server let () = Lwt_main.run main
- Create the client and ping the server.
module Store = Irmin_mem.KV.Make (Irmin.Contents.String) module Client = Irmin_client_jsoo.Make (Store) let config = Irmin_client_jsoo.config (Uri.of_string "ws://localhost:9090/ws") let client = Client.Repo.v config in Client.ping client
More examples can be found on here
Simple Mini GitHub:
I worked on this project to experiment with combining
dream. This turned out simpler than I thought. You only need to expose
irmin-graphql schema. In this application, you simply enter a GitHub repository, and the repository details such as name, date, author, commit message, and README file will be displayed. You can also open
/graphiql and make queries.
The full code can be accessed here.
Pen-it-down is a note app that uses
irmin-server to show an offline-first functionality. Users can type in their notes without being bothered about internet connectivity. You can create, edit, delete, and sync your notes to the server.
The full code can be accessed here.
Working on this project was challenging! I am so glad I had the opportunity to work on it, even though there were days I felt lost. Some days I was confused because it seemed I was doing the wrong thing. Other days I was happy because things worked as expected! It’s basically been about research and experimenting for me. I learned a lot from Patrick and Zach. I was exposed to networking concepts like the network layers, client-server handshake, data encryption, and decryption, and I got to try out WebSocket for the first time. I look forward to building more projects with OCaml.