Editor Support for Typed Lua

Lua is a powerful, efficient, lightweight and dynamically typed scripting language supporting a variety of different programming paradigms. Typed Lua extends Lua with an optional type system, allowing the use of both static and dynamic typing in the same code base, while preserving existing Lua run-time semantics. It achieves that by introducing optional type annotations and applying local type inference. This richer type system not only enforces stronger program behavior guarantees, but also facilitates the development of more powerful developer tools.

Programmers nowadays expect that their editors/IDEs suggest accurate completion candidates, perform asynchronous - as you type - error checking and support type queries such as “go to definition” or “find all references”.

The main goal of the project was to add such features for Typed Lua in a wide range of popular editors. Furthermore, we investigated a design for a modern, asynchronous editor plugin architecture for the vis editor.

Leveraging the Language Server Protocol

Traditionally, every editor had to implemented a custom integration for any language it wanted to support, thus resulting in $m*n$ different implementations of often varying quality. More recently, Microsoft proposed the Language Server Protocol (LSP) to standardize a JSON-RPC API for editor agnostic access to language specific analyzing servers. The idea being, that every editor with LSP client support is immediately able to leverage all existing language servers. Thereby reducing the number of configurations to $m+n$.

Towards that goal we implemented a language server for Typed Lua. More concretely, our contributions are split across three separate repositories each featuring corresponding documentation:

• A Typed Lua language server implementing the JSON-RPC protocol and using the Typed Lua type checker to enable features such as:

  • Find all references
  • Renaming of variables
  • Goto definition
  • Completion candidates
  • Type information on hover
  • Linting as you type

• Changes to the Typed Lua type checker to better handle invalid parsing states and more conveniently deal with the Abstract Syntax Tree (AST) structure.

• A Visual Studio Code plugin for Typed Lua. This is mostly based on the example plugin provided by Microsoft, but exposes Lua specific configuration settings and sets up the communication with the language server.

These features are demonstrated in the following screen cast showing the Visual Studio Code plugin talking to the language server.

There are also some things which will need further work:

• Improved handling of partial parse trees during editing. I have a partially working version where the LPeg grammar was explicitly extended with a number of recovery points. But at this time it does not yet seem to work reliably.

• Better handling of table fields. Some operations only work with variables where it would also be convenient to apply them to arbitrary table fields.

• Support for LuaDoc comments. This would be required to display more helpful completion information.

• Support for code formatting.

A Critique of the Language Server Protocol

While the basic idea of the LSP, standardizing a protocol for editor and development tool communication, is sound, the concrete implementation has a few ugly warts.

Most importantly all addressing within a document is expressed using UTF-16 code units. For an outsider it looks like this was not a deliberate decision, but merely a consequence of the underlying native JavaScript string type.

This is an unfortunate choice, especially for a Unix environment where UTF-8 is the predominant encoding, requiring unnecessary conversions between the documents native encoding and the LSP requirements. Curiously JSON-RPC payload itself is specified to be UTF-8 encoded.

In our opinion an encoding agnostic addressing scheme using absolute byte offsets would be preferable. Unfortunately, it seems unlikely that such a change will happen any time soon.

For certain use cases, such as lifting sam’s structural regular expressions to understand language constructs, it would also be desirable to expose more information of the abstract syntax tree (AST).

Vis Editor Work

GSoC also enabled me to work on vis my open source editor which extends vi’s modal editing with built-in support for multiple selections and combines it with sam‘s structural regular expression based command language.

All commits pushed to the master branch (there are a few yet to be merged design prototypes) can be seen from this search query.

I will now give a short description of some notable changes:

• Cursors and selections (cursors are now singleton selections) have been unified based on a sound theoretical foundation.

  Selections can be stored in marks and all common set operations (union, intersection, minus, complement) can be applied to them. Similarly, registers have been extended to hold a list of strings i.e. one element per selection.

• Overhauled build system, supporting incremental compilation with minimal dependencies. The configure script is a portable hand-written shell script adapted from musl libc. The Makefile strives to be POSIX compatible, it has been verified to work with both GNU as well as BSD make.

• An Alpine Linux Docker image has been used to build a self-contained vis executable which is statically linked against the musl C library, the Lua interpreter and the LPeg module.

  The run time Lua files are extracted from a tar archive which is embedded in the ELF image. During this work I discovered a possible dangerous bug in the libuntar implementation which can lead to data loss. A patch was submitted to and subsequently merged by the upstream libuntar project.

• Improved portability, vis now compiles on all 22 Debian architectures including kFreeBSD and GNU/Hurd.

• Improved code quality, issues reported by the Coverity Scan static analyzer were reviewed and where appropriate fixed.

• Better documentation generated by readthedocs.org using a combination of sphinx, Doxygen and the breathe plugin.

• Scintillua lexer improvements have been submitted to the upstream project which should benefit a number of Lua based editors such as Scintilla and Textadept.

• Extended and cleaned up test suite.

Besides these already merged changes, GSoC also gave me the opportunity to focus on some long-standing design issues.

Client Server Architecture

Since its inception vis’ window management has deliberately been very spartan, we believe window management should best be left to the window manager.

Then main question which was investigated is how the functionality between client and server should be split? How much logic should reside in the client? As an example in kakoune the client merely collects key strokes and forwards them to the server where all the editing logic happens. Conceptually it seems nicer to only provide a minimal server API which would allow different kinds of clients. At times it might be convenient to connect to the same editing session using a heavily mouse driven interface (think acme) while another client would simultaneously provide the power of modal editing. As such the server process would only provide means to:

• Perform editing operations i.e. insert, delete, snapshot, undo, redo.

• Broadcast notifications to all listening clients.

• Query current document state content, length, modified and translation between line numbers and byte offsets.

• Get and set marks as well as registers which are shared among all sessions.

While elegant, such an interface might be too minimalistic and cause rather lot of inter-process communication between client and server. Profiling will be necessary once a non-toy implementation is available.

RPC protocol serialization

The question arises what kind of message serialization/RPC protocol to use for client server communication. These days the most popular choice seems to be JSON. However, because vis aims to handle arbitrary (including binary) data, JSON is not really suitable. MessagePack seems like a promising candidate, not the least because it has a simple implementation for both C and Lua.

Modern Asynchronous Editor Plugin Architecture

Once we have a client/server architecture we need to deal with possibly concurrent editing operations. The user and the plugin (or perhaps multiple plugins) are racing, and the final editing result can depend on the order of execution. The Xi editor project published a nice series of articles touching on this issue. It proposes an approach based on Conflict-free Replicated Data Types (CRDTs). As such I spent a substantial part of the last period of the GSoC program surveying existing literature to become familiar with the topic.

Vis currently uses a piece chain as its core data structure. It was relatively simple to implement (basically a long double linked list) which due to mmap(2) usage provides fast file loading and then degenerates linearly with the number of non-consecutive editing operations. This limits the amount of possible structural sharing among different revisions which is crucial for CRDTs. In order to eventually support balanced tree based structures the existing block layer was refactored. An initial implementation of a monoid cached tree was started, but is not yet readily usable.

Miscellaneous

GSoC allowed me to participate in the wider FLOSS community. As an example I was able to give a talk about vis at the Chaos Computer Club (CCC) Switzerland congress.

Another example of a rather daunting, non-coding related task is the long standing issue regarding the name conflict of vis(1) which is provided by the BSD base systems. In retrospect it probably was a bad idea to try to allude to Rob Pike’s cat-v considered harmful talk. I spent some time to write a checklist of things to be renamed and a script to process the source code. This eventually lead to investigations whether to use the a source code formatter such as clang-format to enforce coding conventions based on a mixture of coding styles (e.g. OpenBSD, Linux, suckless, etc.).

Due to the GSoC support I was also able to fix a few issues in abduco and dvtm two of my other open source projects.

Looking good #lua

I am glad to have been able to participate in a GSoC edition. I learned more about Lua, LPeg the Language Server Protocol and related technologies as well as the challenges of remote work. Furthermore, GSoC allowed me to spend time on some of my own projects and get to know new people around the globe. Pondering editor related design issues (e.g. trying to leverage the LSP for structural editing) will keep me busy for the foreseeable future.

For further GSoC editions I would like to encourage the LabLua organization to foster the community feeling a bit more (e.g. by providing a dedicated IRC channel or participating in the regular one).