Tutorial: Building a Browser Extension With Irmin

Irmin is a collection of OCaml libraries that makes it easy to build applications with Git-like data stores. We recently released irmin-client and irmin-server as official Irmin packages. These packages open up a new way to use Irmin by implementing a custom protocol that lets you write a client application that can interact with a remote data store as if it is local using the Irmin Store API.

In addition to creating a simple example, we also thought it would be fun to build a browser extension that demonstrates a real-life application of these packages and the portability of Irmin in the browser. We created irmin-bookmarks, a browser extension for saving bookmarks in a Git repository. This post gives an overview of the project!

Creating a Browser Extension

At the core of a browser extension is its manifest.json. This is the primary metadata for the extension that tells the browser about the extension: its name, its icons, what permissions it needs, what extension features it uses, etc.

Here is the manifest.json for irmin-bookmarks:

{
  "manifest_version": 2,
  "name": "Irmin Bookmarks",
  "version": "1.0",
  "description": "Save bookmarks to a local git repository. Powered by Irmin.",
  "icons": {
    "48": "icons/icon.png",
    "96": "icons/icon@2x.png"
  },
  "options_ui": {
    "page": "options/index.html"
  },
  "permissions": [
    "storage",
    "tabs"
  ],
  "browser_action": {
    "default_icon": "icons/icon@2x.png",
    "default_title": "Add bookmark!",
    "default_popup": "popup/index.html"
  }
}

We will focus on the browser_action key in this article, but you can read more about the keys in this file on MDN. Note: this key has been renamed to action in version 3 of the manifest specification; we are using version 2 for the widest browser compatibility.

The browser_action key defines the look and behaviour of the button that represents our extension. We want the UI to display when our button is clicked, so we set default_popup to an HTML page that will display our UI for adding a bookmark.

The UI for adding a bookmark looks like this:

UI for the menu of saving a bookmark in the browser, shows a pop-up card with the option to click 'save' the current webpage to bookmarks. It also lets users name and add notes to the bookmark.

The HTML page for the UI for adding a bookmark, popup/index.html, has a simple body definition:

<body>
  <div id="ui"></div>
  <script src="popup.js"></script>
</body>

The UI is created and managed through popup.js, which is compiled from the following OCaml code to JavaScript using js_of_ocaml:

(* extension/popup/popup.ml *)
open Shared

let main () =
  let open Lwt.Syntax in
  let* tab = Browser.tabs |> Tabs.active in
  let model =
    let name = Tab.title tab in
    let url = Tab.url tab in
    let created_at = Date.now () in
    Model.v ~created_at ~name ~url ~notes:""
  in
  let* client = Client.connect () in
  Ui.bind client model

let () = Document.on_content_loaded @@ fun _ -> Lwt.async main

We only need a subset of the browser and extension APIs, so we wrap what we need using Brr in one shared file. Here is a snippet from this file that shows how to bind to the tabs browser extension API, as used above:

(* snippet from extension/shared/ext.ml *)
module Tab = struct
  type t = Jv.t

  let title t = Jv.get t "title" |> Jv.to_string
  let url t = Jv.get t "url" |> Jv.to_string
end

module Browser = struct
  let v : Jv.t = Jv.get Jv.global "browser"
  let tabs : Tabs.t = Jv.get v "tabs"
end

The core UI code for the popup is in extension/popup/ui.ml. Since our UI is not that complicated, the code implements rendering as a simple recursive function based on the state of the UI that calculates the appropriate DOM elements and replaces them as-needed in the <div id="ui"></div> element of our HTML page:

(* render snippet from extension/popup/ui.ml *)
let rec render t =
  let+ elems =
    match t with
    | Disconnected e -> Lwt.return [ msg "error" e ]
    | Connected { client; model; _ } ->
        let () =
          Lwt.async @@ fun () ->
          let* model =
            let+ saved_model = Client.load client model in
            match saved_model with None -> model | Some m -> m
          in
          render (Loaded { model; client; error = None })
        in
        Lwt.return [ msg "info" "Loading..." ]
    | Loaded { client; model; error } ->
        let ui =
          header ()
          :: form model (fun model ->
                 Lwt.async @@ fun () ->
                 let* r = Client.save client model in
                 match r with
                 | Ok _ -> Window.close () |> Lwt.return
                 | Error error ->
                     render (Loaded { error = Some error; model; client }))
        in
        (match error with None -> ui | Some err -> msg "error" err :: ui)
        |> Lwt.return
  in

  let ui = Document.lookup_by_id "ui" in
  let _ =
    elems
    |> List.map Brr.El.to_jv
    |> Array.of_list
    |> Jv.call (Brr.El.to_jv ui) "replaceChildren"
  in
  ()

You can see references in the extension code to Model and Client. We now turn to the core part of the extension: writing our integration with irmin-client and irmin-server!

Creating Our Client and Server

Like the rest of Irmin, irmin-client and irmin-server are libraries meant to be used in applications:

irmin-server lets you write a server application that exposes an Irmin store's API using a custom protocol via an HTTP or WebSocket connection.
irmin-client lets you build a client application that can connect to a remote Irmin store served by irmin-server.

The client and server are wrappers around Irmin stores, so the first step is to decide how to set up our store. When creating an Irmin store, you need to make some choices:

Which Irmin backend do I want to use?
What is the content type for my store?

For irmin-bookmarks, we chose to use irmin-git as our backend since we wanted our bookmarks stored in a Git repository for easy backing up and sharing to a remote Git host. Some other backends that Irmin provides are:

irmin-fs for simple filesystem-based storage
irmin-pack for optimised storage of large amounts of data with features like garbage collection
irmin-mem for in-memory-only storage

To create our server module, we only need the following three lines of code:

module Store = Irmin_git_unix.FS.KV (Model)
module Codec = Irmin_server.Conn.Codec.Bin
module Server = Irmin_server_unix.Make_ext (Codec) (Store)

The first line creates our Irmin store: a key-value (KV) store that persists to disk (FS) in a Git-compatible repository (Irmin_git_unix). We will discuss our custom content type, Model, later. The second line defines the wire encoding for communication between our client and server. We choose a binary encoding (Codec.Bin), but JSON is also available. The final line uses our codec and store to create a server that can bind to a local port for our client to connect. For the complete server binary, see server/main.ml.

A few more lines of code are required to setup our client, but not many!

module Store = struct
  module Git_impl = Irmin_git.Mem
  module Sync = Git.Mem.Sync (Git_impl)
  module Maker = Irmin_git.KV (Git_impl) (Sync)
  include Maker.Make (Model)
end

module Codec = Irmin_server.Conn.Codec.Bin
module Client = Irmin_client_jsoo.Make_codec (Codec) (Store)

Our client compiles to JavaScript via js_of_ocaml as a part of our browser extension, so our store setup looks a little different from the server. Instead of using a filesystem-backed Git repository, we use the in-memory Git implementation (Irmin_git.Mem). It is the same setup as our server: a key-value store backed by an in-memory Git repository. In the last line, we create our client for the browser using our code and store. Note that we use Irmin_client_jsoo since we are compiling for the browser (jsoo is shorthand for js_of_ocaml).

That's all there is to setting up the core server and client! Now let's take a look at our Model.

Bookmark Model

Irmin stores support custom serialisable and mergeable content types. These types define not only how to encode and decode the type for persistence but also how to perform a 3-way merge when conflicts arise. For our model, we use a simple merge algorithm of picking the "latest" updated one, based on a clock timestamp.

let merge_m ~old x y =
  ignore old;
  (* Simple merge: pick "latest" updated model *)
  Irmin.Merge.ok @@ if x.updated_at > y.updated_at then x else y

let merge = Irmin.Merge.(option (v t merge_m))

Irmin has built-in content types for strings and JSON. Since our bookmarks contain a few fields of information, and we would like a human-readable format in our repository, we use JSON for our serialisation format. The JSON content type exposed by Irmin is low-level, so we define a module to wrap this lower level type by mapping from our custom type to Irmin's JSON type:

let t = Irmin.Type.map Json.t of_json to_json

You can look at model.ml to see our complete model. Here is an example of what a bookmark looks like in our repository:

{"updated_at":1696621685526,"created_at":1696621683493,"name":"Tarides","notes":"Building Functional Systems","url":"/"}

Using the Irmin API

Our client only uses a small part of the Irmin store API to list, load, save, and delete our bookmarks.

let list t =
  let* tree = tree t in
  (* [Store.Tree.fold] *)
  Tree.fold ~order:`Undefined
    ~contents:(fun _path m acc -> m :: acc |> Lwt.return)
    tree []

let load t model =
  let key = Model.key_path model in
  (* [Store.find] *)
  find t key

let save t model =
  let f tree =
    let key = Model.key_path model in
    (* [Store.Tree.add] *)
    Tree.add tree key model
  in
  update t f ~info:(Info_jsoo.v ~author "Update %s" model.url ())

let delete t model =
  let f tree =
    let key = Model.key_path model in
    (* [Store.Tree.remove] *)
    Tree.remove tree key
  in
  update t f ~info:(Info_jsoo.v ~author "Delete %s" model.url ())

Each bookmark is stored in the Git repository using a unique path. Loading, saving, and deleting is as easy as passing this path to the corresponding store functions.

An example list of saved bookmarks, here the bookmarks are: Irmin, OCaml.org, and Tarides.com

For listing our bookmarks, we can accumulate all of our models in the store's tree. If our repository contained a large number of bookmarks, we would need to implement some kind of pagination API, but simple accumulation is enough for our demo application.

Concurrent Atomic Updates

An interesting function to look at more closely is update. This function is used when saving and deleting and performs atomic updates to our store, even if multiple tabs are concurrently writing.

let update t f ~info =
  catch @@ fun () ->
  let repo = repo t in
  (* Get latest tree for main branch *)
  let* main = of_branch repo Branch.main in
  let* head = Head.get main in
  (* Apply [f] to the tree on main to get our new tree *)
  let* tree = Commit.tree head |> f in
  (* Commit this tree *)
  let* commit = Commit.v repo ~info ~parents:[ Commit.key head ] tree in
  (* Merge commit to main *)
  let* main = of_branch repo Branch.main in
  merge_with_commit main commit ~info:(Info_jsoo.v ~author "Merge to main")

The key to how this works is merging!

Simply writing a commit directly to the main branch, like when using Irmin's set_tree, can fail when done concurrently because updating the main branch reference is done using a compare-and-swap operation. To avoid this issue, we first create a new commit with our changes and then attempt to merge it to the main branch using merge_with_commit. When performed concurrently, the process of merging our updated commit into the latest commit on main and updating the reference is retried if it fails. The process will terminate either with a successful merge and update or a merge conflict that cannot be handled automatically.

When merging, there are two conflict scenarios:

The same bookmark is added or updated on main and added or updated in the update commit. This will be resolved by our merge function since it picks the "newest" model.
The same bookmark is deleted either on main or the update commit and added or updated in the other. This will result in a merge conflict since the custom merge function of our model is not called when one side is deleted.

The current code in the extension simply propagates the error in the second case which means that a user needs to try again. This case could be handled specially to build an improved user experience, but was sufficient for our demo application. The important aspect is that our extension handles concurrent updates correctly and can automatically resolve conflicts in many cases.

Wrapping up

And that's it! Take a look at the project's repository to see that it only takes about 500 lines of OCaml code to have a fully working browser extension that saves, loads, lists, and deletes bookmarks in a local git repository.

Check out the project's README for how to build and use the extension. If you give any Irmin packages a try and run into issues, feel free to open issues or PRs on the Irmin repository. Happy hacking!

Tutorial: Building a Browser Extension With Irmin

Creating a Browser Extension

Creating Our Client and Server

Bookmark Model

Using the Irmin API

Concurrent Atomic Updates

Wrapping up

Open-Source Development

Explore Commercial Opportunities

Stay Updated on OCaml and MirageOS!

Subscription Succesful

Subscription Succesful