Natural Language Syntax Highlighting

(authored by human unless marked 🤖)

Natural-Syntax-LS is a language server that highlights different parts of speech (POS) in plain text.

Full highlighting	Partially disabled (via customization)

Installation

🤖 Install with a current stable Rust toolchain (Rust 1.91 or newer):

cargo install --git https://github.com/SichangHe/natural_syntax natural_syntax_ls

🤖 The older crates.io release still needs libtorch. This implementation runs ONNX in Rust and requires no libtorch download or library-path configuration.

🤖 On first launch, the server downloads approximately 100 MB of English MobileBERT model files from Hugging Face. Subsequent launches reuse the local cache. Allow the first launch to finish before requesting highlighting. The model cache follows HF_HOME, defaulting to ~/.cache/huggingface.

🤖 Without Rust, npm install -g ./natural_syntax_js installs the equivalent JavaScript language server natural-syntax-ls-js; see natural_syntax_js.

🤖 The JavaScript library also runs model inference in browsers through WebAssembly. See the browser example.

Editor setup

✅ NeoVim setup with LSPConfig

Please paste the below register_natural_syntax_ls function in your Nvim configuration, call it, and set up natural_syntax_ls like any other LSPConfig language server. Please see my config for an example.

The natural_syntax_ls_setup function.

local function natural_syntax_ls_setup(capabilities)
    require('lspconfig.configs').natural_syntax_ls = {
        default_config = {
            cmd = { 'natural-syntax-ls' },
            filetypes = { 'text' },
            single_file_support = true,
        },
        docs = {
            description = [[The Natural Syntax Language Server for highlighting parts of speech.]],
        },
    }
end

You can customize by setting init_options when calling the setup function:

require('lspconfig')['natural_syntax_ls'].setup {
    init_options = {
        token_map_update = { -- Customize your POS-token mapping here. E.g.:
            -- Disable coordinating conjunctions highlighting.
            CC = vim.NIL, -- `nil` does not work because it gets ignored.
            -- Highlight wh-determiners as enum members without any modifiers.
            WDT = { type = "enumMember" },
            -- Highlight determiners as read-only classes.
            DT = { type = "class", modifiers = { "readonly" } },
        },
    },
}

Customizations:

• I only set the filetypes field to text, but you can enable natural-syntax-ls for any other file types as well. Note that, though, the language server's semantic tokens supersede Tree-sitter highlighting by default.
• By specifying the token_map_update field in init_options, you can customize the mapping between parts of speech and semantic tokens.
  • The default mapping is in the pos2token_bits function in semantic_tokens.rs.
  • Part of speech tags are the variants of the PartOfSpeech enum in lib.rs.
  • Token types and modifiers are variants of TokenType and TokenModifier in semantic_tokens.rs, all in camelCase.

🤖 Visual Studio Code setup

🤖 Build and install the extension in editors/vscode:

cd editors/vscode
npm install
npm run package
code --install-extension natural-syntax-ls-0.0.1.vsix

🤖 It starts natural-syntax-ls for plain text files. The settings naturalSyntaxLs.serverPath and naturalSyntaxLs.tokenMapUpdate choose the server command and customize highlighting.

❓ Other editor setup

Community plugins are welcome: it should be straightforward to implement plugins since Natural-Syntax-LS implements the Language Server Protocol (LSP).

Selected specification

Prediction Scheduling

For a single document, only one prediction is scheduled at a time. When a prediction is ongoing, new updates are queued and the latest update replaces any previous updates queued.

Debugging

We use tracing-subscriber with the env-filter feature to emit logs¹. Please configure the log level by setting the RUST_LOG environment variable.

🤖 The ONNX implementation requires no DYLD_LIBRARY_PATH setting.

Future work

• Customizing the mapping between part of speech and semantic token.
• Support languages other than English. This simply requires a new model.
• Incremental updates and semantic token ranges.
• Do not overwrite Markdown/LaTeX syntax highlighting.

Footnotes

1. https://docs.rs/tracing-subscriber/latest/tracing_subscriber/#feature-flags ↩