Creating the SyntaxDocumentation Command - Part 1: Merlin

OCaml development has never been more enchanting, thanks to Merlin – the wizard of the editor realm. The magic of Merlin is something that makes programming in OCaml a very nice experience. By Merlin, I don't mean the old gray-haired, staff-bearing magic guy. I'm talking about the editor service that provides modern IDE features for OCaml.

Merlin currently has an arsenal of tools that enable code navigation, completion, and a myriad of others. To use Merlin, we give it commands, kinda like reading spells from a magic book. In Merlin's magical world, each spell (command) works in a particular way (the logic), requires a specific set of items to work (the inputs), and produces a specific set of results (the output).

This article is the first of a three-part series. Here, we'll be looking at how to implement a new command in Merlin, taking the SyntaxDocumentation functionality as a case study. The second will explore how we integrate this command with ocaml-lsp-server, and in the final article, we'll learn how to include this command as a configurable option on the VSCode OCaml Extension.

The SyntaxDocumentation Command

The Problem

Before going into the implementation details, let's take some time to understand what this command is all about and why it's even needed in the first place.

A common challenge faced by OCaml developers is the need for quick and accurate documentation about their code's syntax. While OCaml is a powerful language, its syntax can sometimes be complex, especially for newcomers or developers working on unfamiliar codebases.

Without proper documentation, understanding the syntax can be like navigating a maze blindfolded. You may spend valuable time sifting through hundreds of pages of documentation to find what you are looking for. Googling "syntax symbols" doesn't really help much unless someone faced the same problem and specifically used the exact syntax. This inefficiency not only slows down development, but it also increases the likelihood of errors and bugs creeping into the codebase. Programming should be about the solution you're implementing, not just about the language's syntax, so having a quicker way to understand syntax will go a long way to make programming in OCaml a much nicer experience.

The Solution

Most programmers write code using a text editor, such as Vim, Emacs, VSCode, etc. An editor typically has a basic interface for writing and navigating code by using a cursor. Whenever the user's cursor is over some code, the editor tells us what that code is and provides further information about the syntax. The SyntaxDocumentation command basically grabs the code under the cursor and uses Merlin's analysis engine to extract relevant information about its syntax. This information is then presented back to the user.

The Implementation

To implement the SyntaxDocumentation command, we'll use a simple three-step approach:

1. The Trigger: What the user should do to trigger this command.
2. The Action: What actions should be executed when the user has triggered this command.
3. The Consequence: How the results should be presented when the action(s) are completed.

The trigger here will be a simple hover, such as placing your cursor above some code. We won't go into detail how this works. (See Part 2).

In this article, our focus will be on step 2 and 3. This covers which actions to run after the trigger and what the result should be.

1. New Commands

For our SyntaxDocumentation command to be possible, we have to let Merlin know of the new command by defining it. This involves telling Merlin the name of the command, what the command needs as input, and what Merlin should do if this command is called.

To create a new command, we need to add its definition to some files:

• new_commands.ml: In this file, we define our new command and indicate the inputs it requires. By giving it a helpful description, the user can learn about it from the help menu.

command "syntax-document"
    ~doc: "Returns documentation for OCaml syntax for the entity under the cursor"
    ~spec: [
        arg "-position" "<position> Position to complete"
        (marg_position (fun pos _pos -> pos));
    ]
    ~default: `None
    begin fun buffer pos ->
        match pos with
        | `None -> failwith "-position <pos> is mandatory"
        | #Msource.position as pos ->
            run buffer (Query_protocol.Syntax_document pos)
    end
;

Basically, this tells Merlin to create a new command called syntax-document that requires a position. Msource is a helpful module containing useful utilities to deal with positions.

• query_protocol.ml: In this file, we specifically define our command's input and output types. Here we tell Merlin that our command needs an input of type position from the Msource module, and our output can either be No_documentation or that documentation with the type syntax_doc_result has been found.

| Syntax_document
    : Msource.position
    ->  [ `Found of syntax_doc_result
        | `No_documentation
        ] t

syntax_doc_result is defined as a record that contains a name, a description, and a link to the documentation:

type syntax_doc_result =
{
    name : string;
    description : string;
    documentation : string
}

So basically our command will receive a cursor position as input and either return some information (name, description, and a documentation link) or say no documentation has been found.

• query_json.ml: In this file, we write the code for how Merlin should format the response it sends to other editor plugins, such as Vim and Emacs.

| Syntax_document pos ->
    mk "syntax-document" [ ("position", mk_position pos) ]

Here, mk_position is a function that takes our cursor position and serialises it for debugging purposes.

| Syntax_document _, response -> (
    match response with
    | `Found info ->
            `Assoc
            [
                ("name", `String info.name);
                ("description", `String info.description);
                ("url", `String info.documentation);
            ]
    | `No_documentation -> `String "No documentation found")

This code serialises the output into JSON format. It is also used by the different editor plugins.

Great! Now that we have seen how Merlin handles our inputs and outputs, it's time to understand how we convert this input into the output.

2. Command Logic

The implementation of our logic is found in the file query_commands.ml.

Before diving deep, let's understand a few concepts that we'll use in our implementation.

• Source Code: refers to the OCaml code written in a text editor

• Parsetree: a more detailed internal representation of the source code

• Typedtree: an enhanced version of the Parsetree where type information is attached to each node

Basically, the source code is the starting point. The parser then takes this source code and builds a Parsetree representing its syntactic structure. The type checker analyses this Parsetree and assigns types to its elements, generating a Typedtree.

The Merlin engine already provides a lot of utilities that we can use to achieve all of this, such as:

• Mpipeline: This is the core pipeline of Merlin's analysis engine. It handles the various stages of processing OCaml code, such as lexing and parsing.
• Mtyper: This module provides us with utilities for interacting with the Typedtree.
• Mbrowse: This module provides us with utilities for navigating and manipulating the nodes of the Typedtree.

Our implementation code is:

| Syntax_document pos ->
    let typer = Mpipeline.typer_result pipeline in
    let pos = Mpipeline.get_lexing_pos pipeline pos in
    let node = Mtyper.node_at typer pos in
    let res = Syntax_doc.get_syntax_doc pos node in
    (match res with
    | Some res -> `Found res
    | None -> `No_documentation)

We are using the Mpipeline module to get the Typedtree and the lexing position (our cursor's position in the source code). Once this position is found, we use the Mtyper module to grab the specific node found at this cursor position in the Typedtree. Mtyper uses the Mbrowse module to navigate through the Typedtree until it arrives at the node that has the same position as our lexing position (cursor position).

Example: Let's consider a simple variant type:

type color = Red | Green

The node tree will be:

[ type_kind; type_declaration; structure_item; structure ]

Once we have received this node tree, we pass it to our custom module Syntax_doc.ml and call the function get_syntax_doc within it. This pattern-matches the node tree and extracts the relevant information or returns no information. An excerpt from our custom module is presented below:

...

let get_syntax_doc cursor_loc node =
    match node with
    | (_, Type_kind Ttype_variant _) :: (_, Type_declaration _) :: _
        ->
        Some {
                name = "Type Variant";
                description = "Represent's data that may take on multiple different forms.";
                documentation = "https://v2.ocaml.../typev.html";
            }
...
    | _ -> None

The output returned conforms to type syntax_info = Query_protocol.syntax_doc_result, which is defined in query_protocol.ml.

Results and Tests

After Merlin runs the logic, it has to return some results for us. To test that our code works well, we use the Cram testing framework to check it's functionality. Example: Say we write the following source code in a file called main.ml:

type rectangle = { length: int; width: int}

We can use Cram to call Merlin and ask it to get us some information:

$MERLIN single syntax-document -position 1:12 -filename ./main.ml < ./main.ml

Here we are telling Merlin to give us information for the cursor position 1:12, which means the first line on the 12th column (begin from the first character of line 1 and count 12 characters). This will place our cursor over the word rectangle.

When we run this test, the result returned will be:

{
   "name": "Record types",
   "description": "Allows you to define variants with a fixed...",
   "url": "https://v2.ocaml....riants.html"
}

Conclusion

We have finally come to the end of the first part of this article series. In this part, we explored the problem we are trying to solve, hypothesised about a possible solution, and then implemented the solution. This implementation is like the engine/backend of our solution.

It was a very fun and exciting experience working on this command. I got to learn a lot about OCaml, especially how Merlin works internally. In Part 2 of this series, we'll look at how this new functionality has been integrated to work with various editors such as Vim, Emacs, and VSCode.

Here are some lessons I learned on this journey:

• An idiomatic approach is better. Coming from a non-functional background, I am used to writing code a certain non-functional way and wasn't yet baptised in the functional programming waters. I would frequently write these long if-else statements instead of just using pattern matching, the OCaml way. Once I understood it, it became like magic dust for me. I used it very much.
• Read the documentation. When working with a new project, the best thing you can do is spend some time reading through the documentation and looking at how the code is written. Most times, something you may be struggling to write may have already been implemented, so you just have to use it. Stop reinventing the wheel.
• Ask questions. I can't count how many times I got stuck and had to scream for help, like a kid in a candy shop who lost his toy. I have amazing mentors who are always willing to point me in the right direction. This support literally feels like wielding Thanos' gaunlet!