Irmin in the Browser

Introduction

Over the past six months, I have been working on using Irmin in the browser, including irmin-server and the GraphQL interface. This has been fun and a great learning journey for me. Before this internship, irmin-server was primarily a Unix-based application. My project was to port irmin-server to work in the browser and design interfaces for people to interact with the store (Irmin stores).

I was paired to work with Patrick Ferris as my mentor and with the entire Irmin team, who all contributed immensely to this project.

Irmin and irmin-server

Irmin is simply a data store (database). It is based on the same design principle as Git with features to merge and branch data stores. Irmin has several stores (irmin-mem, irmin-indexeddb, irmin-fs, irmin-chunk, irmin-git) and store interfaces (irmin-http, irmin-graphql).

irmin-server is a high-performance server for Irmin. For efficient communication, it implements a specialised wire-protocol to send and receive data over a bytestream. It wraps an Irmin store, providing a way to connect to the server and access the store via its API using a client. But the client makes an assumption that the user is on a Unix machine, which makes irmin-server primarily a Unix-based application.

irmin/irmin-client in the Browser

In this modern age, it's become a necessity to make applications "offline first." Offline-First applications function without being affected by the intermittent lack of a network connection. It usually implies the ability to sync data between multiple devices. Irmin as a data store supports multiple backends, making it very portable. Plus, Irmin's mergeable replicated data-types make it much easier to build applications that can transform the state offline and resynchronise the state later, just like Git. With this concept, resynchronising Irmin stores (from server to client) is much simpler on irmin-server, which implements a specialised wire protocol for efficient communication. Making irmin/irmin-client work in the browser simply means that it would be possible to create offline-first web applications.

More information on offline-first applications can be found here

The Problems

An initial summary of the problem was published on this issue, but here is a quick breakdown of the problems we identified.

1. irmin-server was tightly coupled around conduit-lwt-unix: irmin-server was initially designed to be a Unix-based application that established communication with a client via conduit-lwt-unix. This became a problem because conduit-lwt-unix cannot establish a communication from a browser. This meant that there was a need to abstract the I/O module so that every client will provide its I/O.
2. Reuse some internal modules: We needed to reuse the irmin-server internal logic related to the protocol but provide a portable I/O interface that can work in the browser.
3. Provide a browser communication channel: We needed a non-blocking way to establish a channel to create communication between irmin-server and the browser, and also pass data across this channel.

The Solutions

irmin-server was tightly coupled around conduit-lwt-unix

Thanks to Zach Shipko, who abstracted the I/O library and split out irmin-client-unix and irmin-client-cli to have their own I/O module that depends on conduit-lwt-unix (here), a client can connect to a running irmin-server using its own I/O module. While he was working on the restructuring, I spent my time working on a sample project that combines dream with irmin-graphql (more on this project).

With the coupling out of the way, the next step was to create irmin-client-jsoo, a browser client with its own I/O module.

irmin-server was primarily a Unix-based application

The irmin-server initial architecture had to be restructured to accommodate other platforms. To achieve this, irmin-client was no longer coupled with a specific I/O implementation. Rather, a Unix-based one was provided over conduit flows, which are Lwt_io input and output channels. This channel was established over a TCP connection or a Unix domain socket.

Right now, irmin-server can communicate with two (2) clients: irmin-client-cli from a command line and irmin-client-unix from a Unix-based machine. This project was about creating a third client: irmin-client-jsoo, to be called from browser applications.

Enable communication from the browser

After considering other options to create a communication channel for irmin-client-jsoo, like HTTP, RPC, etc., Patrick suggested WebSocket, so we decided to go with WebSocket, a bidirectional communication protocol between client and server.

The Challenges

irmin-server uses flows to communicate between the server and the client and flows are bytestreams. WebSocket provides a bidirectional communication channel in the browser, but it is not stream-oriented rather it is message-oriented.

TCP (Transmission Control Protocol) is a type of protocol or standard to transfer information over the Internet while WebSocket is a message-oriented application protocol, which uses TCP as the transportation layer.

The idea behind the WebSocket protocol consists of reusing the established TCP connection between a client and server. Even though WebSocket is built on TCP, the data it passes is always either sent as a whole "message" or not at all. These implementations are non-blocking.

Since we are avoiding a full redesign of the irmin-server protocol, we had to make the message-oriented process seem like bytestreams of data.

More on irmin-client-jsoo

Communicating with irmin-server from the browser is very easy. You can achieve that by following these steps:

1. Pin irmin-server, using this command: opam pin add git+https://github.com/mirage/irmin-server/commit#013a28fd1507f8ba69494515533119804903aa99
2. Set up the server.

open Lwt.Syntax
module Store = Irmin_mem.KV.Make (Irmin.Contents.String)
module Server = Irmin_server.Make (Store)

let main =
  let uri = Uri.of_string "ws://localhost:9090/ws" in
  let config = Irmin_git.config "penit" in
  let* store = Store.Repo.v config in
  let* main = Store.main store in
  let* server = Server.v ~uri config in
  let () = Format.printf "Listening on %a@." Uri.pp uri in
  Server.serve server

let () = Lwt_main.run main

Check out this implementation

3. Create the client and ping the server.

module Store = Irmin_mem.KV.Make (Irmin.Contents.String)
module Client = Irmin_client_jsoo.Make (Store)

let config = Irmin_client_jsoo.config (Uri.of_string "ws://localhost:9090/ws")
let client = Client.Repo.v config in
Client.ping client

More examples can be found on here

My Projects

Simple Mini GitHub: I worked on this project to experiment with combining irmin-graphql with dream. This turned out simpler than I thought. You only need to expose irmin-graphql schema. In this application, you simply enter a GitHub repository, and the repository details such as name, date, author, commit message, and README file will be displayed. You can also open /graphiql and make queries.

The full code can be accessed here.

Pen-It-Down: Pen-it-down is a note app that uses irmin-indexeddb and irmin-server to show an offline-first functionality. Users can type in their notes without being bothered about internet connectivity. You can create, edit, delete, and sync your notes to the server.

The full code can be accessed here.

Conclusion

Working on this project was challenging! I am so glad I had the opportunity to work on it, even though there were days I felt lost. Some days I was confused because it seemed I was doing the wrong thing. Other days I was happy because things worked as expected! It’s basically been about research and experimenting for me. I learned a lot from Patrick and Zach. I was exposed to networking concepts like the network layers, client-server handshake, data encryption, and decryption, and I got to try out WebSocket for the first time. I look forward to building more projects with OCaml.