🧬 RouterOS LSP for better syntax checking & command completion in editors like VSCode & NeoVim

Let’s start with “What is a LSP?”

The Language Server protocol is used between a tool (the client) and a language smartness provider (the server) to integrate features like auto complete, go to definition, find all references and alike into the tool

—from https://langserver.org

As discussed in posting below, I created a “working prototype” of an LSP Server for RouterOS script (and configuration). In theory, it should be VERY similar to the CLI and “edit” command on RouterOS itself — just in any modern editor. See post #2 for some background. My intent was not on creating my own LSP for RouterOS - but here we are… And much more could be done. But at high level is will accurately do completion for RouterOS and does same validation the CLI does for syntax error (i.e. like “edit”) in tools like Visual Studio Code and NeoVim.




I committed the code discussed in post #3 below to GitHub:

https://github.com/tikoci/lsp-routeros-ts

and if any wanted to contribute to it, feel free to discuss below or submit a “pull request” with any purpose fixes/changes to GitHub

It’s written in TypeScript and based on Microsoft’s LSP example codes. I’ve mainly tested it VSCode, but did see it working in NeoVim (nvim). The GitHub project tikoci/lsp-routeros-ts readme has WAY more details on installation and usage, as well as setuping for further development.

The newest (v0.3.4) release now appears in VSCode’s Extension ‘Marketplace’, to allow for easy install and update. To install – in Visual Studio Code — use Shift + ⌘ + X to bring up “Extensions” in VSCode, then search for “RouterOS LSP” and this plugin will appear. Next, hit the “Install” button, where it will prompt you it’s pre-release from an unverified developer. See post #6 below for more details

Alternatively, to use it in VSCode – using downloadable VSIX file — it’s pretty easy…

  1. Download lsp-routeros-ts.vsix file from https://github.com/tikoci/lsp-routeros-ts/releases – you may have to unzip it after download
  2. Install using Terminal: code --install-extension ~/Downloads/lsp-routeros-ts.vsix - adjust path as needed
  3. Launch VSCode using “code” at terminal (or however)
  4. Select “Extensions” from left most panel, then search and select “RouterOS LSP” - if installed correctly above, it should show up
  5. In main panel on right-side, with “RouterOS LSP” shown, hit the :gear: then pick “Settings”
  6. In the Settings, you’ll need to specify a router and credentials to use as “backend” to do validations. The provided account only need read,api,rest-api permissions – do not use a “full” admin account – the LSP just need be able to call the read-only “/console/inspect…”. The readme has more details on configuration.
  7. The settings should take effect right away (although restarting VSCode cannot hurt) and the LSP will process any *.rsc file that’s opened in VSCode. With usual ⌘+Space bring up any “completions” (like CLI would use tabs) in any *.rsc file. You can also select “RouterOS” in bottom right status bar to “force” a file to be processed by the lSP — only files ending in .rsc will automatically load the RouerOS LSP

NeoVim is more complex - but it works. Again the readme on GitHub has more details, especially on setup. With the “RouterOS internals” of how it works discussed more in post 2/3 below — with some quick GIF of the LSP “in action”.

I’m sure there are thing broken and MUCH more work could be done. And some known bugs… I kinda just want to get to some “strawman” with all the needed plumbing to “run” the LSP code (i.e. while there dozens of files, only TWO are the actual code of a few hundred lines).

But if you do try it out, and have a problem feel free to post below (or file issue in GitHub). I cannot promise anything — since I kinda want to more “test the concept” than maintain an LSP.

7 Likes

Background

This topic recently arose from a commentary between @Kentzo and me in the 7.20beta thread concerning an “LSP”. While many folks may not know what an LSP is… it is an API that most modern text editors use (like VSCode) to do, among other things, syntax checking, and coloring syntax. See https://lang-server.org for more info.

While there are RouterOS code editor plugins, including @Kentzo’s, these all use regular expressions. This works…but using regex is not going to 100% accurate. And an LSP can do WAY more things with script/config… depending on how LSP is implemented. For example, LSP can do stuff like “Go to definition” to find where variables are declared, and more (see https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#languageFeatures for possible operations)

Although I’ve thought about it… I don’t have time to actually develop a RouterOS LSP - the initial “plumbing” to create an LSP project framework is more work than I want to take on… But there are plenty of videos on YouTube on how to create an LSP from scratch & imagine LLMs be pretty good at JavaScript and LSP – since 10000000x more common than RouterOS syntax. Plus some RouterOS LSP likely be a useful step to anyone attempting to develop an MCP for AI agents.

So did want to expand on HOW ONE COULD develop a RouterOS LSP using the undocumented /console/inspect request= commands that are built to RouterOS (and thus available via CLI/SSH/REST/API), so that part is possible if one wanted to create a RouterOS LSP. The only downside of using /console/inspect is a future RouterOS LSP needs to use REST and connect to an online router to work, while most LSP work “offline” since LSP are typically embedded/included with a compiler/interpreter. Now the nice part of using /console/inspect as source of LSP code editor is that it be 100% accurate at highlighting errors specific to the version/packages on the “LSP connected” RouterOS device.

Specifically, /console/inspect takes a “request” parameter. This is similar to LSP model where code editor makes requests, and the LSP responds with the metadata for the LSP request. RouterOS “inspect” supports a few “requests” that help in an LSP. Specifically:

/console/inspect request=<tab>
child     completion     error     highlight     self     syntax

Of main interest to a future LSP are “request=highlight” and “request=completion”. While there are dozens of things an LSP can provide to a code editor about a language like RouterOS config/script, “highlight” and “completion” map directly into the LSP APIs.

So request=highlight input=<script/config/line> corresponds with the “Document Highlight” LSP API. RouterOS provides the following data which can “feed” the LSP:

:put [:serialize to=json [/console/inspect request=highlight input="/ip/address/add interface=ether1 address=1.1.1.1/24" as-value ]]
{"highlight":["dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","cmd","cmd","cmd","none","arg","arg","arg","arg","arg","arg","arg","arg","arg","syntax-meta","none","none","none","none","none","none","none","arg","arg","arg","arg","arg","arg","arg","syntax-meta","none","none","none","none","none","none","none","syntax-meta","none","none"],"type":"highlight"}

and via REST it already be JSON.

Each of the elements in the highlight array corresponds to each character in the input= so the above was for the command:

  /ip/address/add interface=ether1 address=1.1.1.1/24
# ddddddddddddCCC-aaaaaaaaaM-------aaaaaaaM-------M--

The values returned from request=highlight mostly match what /terminal/style so it provides EXACTLY the same colors as RouterOS UIs. So some possible values for highlight’s array are:

/terminal/style <tab>
ambiguous     comment     error     escaped     none     syntax-meta     syntax-noterm     syntax-old     syntax-val     varname     varname-global     varname-local     style

And there is also one/most additional ones. Specifically “obj-inactive” and “syntax-obsolete” that I know. Here are some examples of invalid commands provided:

:put [:serialize to=json [/console/inspect request=highlight input="/ip/address bad-param" as-value ]]
{"highlight":["dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","none","obj-inactive","obj-inactive","obj-inactive","obj-inactive","obj-inactive","obj-inactive","obj-inactive","obj-inactive","obj-inactive"],"type":"highlight"}
:put [:serialize to=json [/console/inspect request=highlight input="/ip/address add bad-param" as-value ]]         
{"highlight":["dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","none","cmd","cmd","cmd","none","error","none","none","none","none","none","none","none","none"],"type":"highlight"}
:put [:serialize to=json [/console/inspect request=highlight input="/ip/address add address " as-value ]]
{"highlight":["dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","none","cmd","cmd","cmd","none","arg","arg","arg","arg","arg","arg","arg","syntax-obsolete"],"type":"highlight"}
:put [:serialize to=json [/console/inspect request=highlight input="/ip/address add address=" as-value ]]  
{"highlight":["dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","none","cmd","cmd","cmd","none","arg","arg","arg","arg","arg","arg","arg","syntax-meta"],"type":"highlight"}
:put [:serialize to=json [/console/inspect request=highlight input="/ip/address add address2=" as-value ]]
{"highlight"":["dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","dir","none","cmd","cmd","cmd","none","error","none","none","none","none","none","none","none","none"],"type":"highlight"}

And same request=highlight can be used to VALIDATE RouterOS config since CRITICALLY an element with “error” or “obj-inactive” in highlight’s array means the script will NOT work and has, well, a syntax error (at the array index where “error”/“obj-inactive” was found in the array matches the char position of the command/script/config provided in inspect’s input= attributed used to get the highlights).

“request=highlight” supports multi-line input, so you can check an entire script with newlines/spaces/etc & look for the “error states” in results from “/console/inspect request=highlight input=<script/config/line>” were found. If none were found, then the syntax is at least valid (although may not work).

The other /console/inspect thing that’s useful is request=completion. Similarly, it takes an input= with the script/config to test. With the LSP API having a similar “Text Document Completion” capability, so if editor requests completion, RouterOS can make SAME suggestion as in CLI does - but in any LSP support editor. See https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#textDocument_completion. This one is more complex, and IDK exactly the meaning of all the field, although, “preference”/“show” likely used to determine if something merely valid/possible syntax, or if should be used in completion.

For example, request=completion returns stuff like:

/console/inspect request=completion input="/ip/address/add interface=ether1 address=1.1.1.1/24"
Columns: TYPE, COMPLETION, STYLE, OFFSET, PREFERENCE, SHOW, TEXT
TYPE        COMPLETION  STYLE        OFFSET  PREFERENCE  SHOW  TEXT                                                                                 
completion  <value>     none             49  -1          no    literal value that consists only of digits, letters and characters -.,:<>/|+_*&^%#@!~
completion              none             51  80          no    whitespace                                                                           
completion  ;           syntax-meta      51  40          no    end of command                                                                       
/console/inspect request=completion input="/ip/address/add interface=ether1 address=1.1.1.1/24c"
Columns: TYPE, COMPLETION, STYLE, OFFSET, PREFERENCE, SHOW, TEXT
TYPE        COMPLETION  STYLE        OFFSET  PREFERENCE  SHOW  TEXT                                                                                 
completion  <value>     none             49  -1          no    literal value that consists only of digits, letters and characters -.,:<>/|+_*&^%#@!~
completion              none             52  80          no    whitespace                                                                           
completion  ;           syntax-meta      52  40          no    end of command          

# and adding a <space> at end of input= does show the attributes that are possible                                                             
/console/inspect request=completion input="/ip/address/add interface=ether1 address=1.1.1.1/24 " 
Columns: TYPE, COMPLETION, STYLE, OFFSET, PREFERENCE, SHOW, TEXT
TYPE        COMPLETION  STYLE        OFFSET  PREFERENCE  SHOW  TEXT                                   
completion  !           none             52          80  no    whitespace                             
completion  broadcast   arg              52          95  yes   Broadcast address                      
completion  comment     arg              52          96  yes   Short description of the item          
completion  copy-from   arg              52          96  yes   Item number                            
completion  disabled    arg              52          96  yes   Defines whether item is ignored or used
completion  netmask     arg              52          95  yes   Network mask                           
completion  network     arg              52          96  yes   Network prefix                         
completion  ;           syntax-meta      52          40  no    end of command

While perhaps other /console/inspect things might be useful to a future LSP, the above would get someone pretty far in creating an LSP for RouterOS. And, my existing tikoci/restraml project on GitHub make HEAVY use of request=child (to determine the AST/“command tree”) and request=syntax (which, sometimes, has help or the valid values of an attribute) to generate RouterOS RAML/OpenAPI schema. For example, request=syntax using similar command as above tells you valid values for LAST attribute in input= (so address=):

/console/inspect request=syntax input="/ip/address/add interface=ether1 address=1.1.1.1/24"
Columns: TYPE, SYMBOL, SYMBOL-TYPE, NESTED, NONORM, TEXT
TYPE    SYMBOL     SYMBOL-TYPE  NESTED  NONORM  TEXT                     
syntax  Netmask    definition        0  no      IpNetmask | Num          
syntax  IpNetmask  definition        1  no      A.B.C.D                  
syntax  Num        definition        1  no      0..32    (integer number)

but this get more complex to “map” to the LSP API IMO. Plus “/console/inspect request=syntax” data is NOT actually complete as to “valid values” (shown in text= value). And there is a path= option to request=syntax, that isn’t so easy to map from LSP code… but it does have what F1 help would return:

/console/inspect request=syntax path=ip,address,add                                        
Columns: TYPE, SYMBOL, SYMBOL-TYPE, NESTED, NONORM, TEXT
TYPE    SYMBOL     SYMBOL-TYPE  NESTED  NONORM  TEXT                                   
syntax             collection        0  yes                                            
syntax  address    explanation       1  no      Local IP address                       
syntax  broadcast  explanation       1  no      Broadcast address                      
syntax  comment    explanation       1  no      Short description of the item          
syntax  copy-from  explanation       1  no      Item number                            
syntax  disabled   explanation       1  no      Defines whether item is ignored or used
syntax  interface  explanation       1  no      Interface name                         
syntax  netmask    explanation       1  no      Network mask                           
syntax  network    explanation       1  no      Network prefix

Anyway, thought I’d share what I know since the topic came up & all of this is undocumented and potentially underutilized.

1 Like

Initial Implementation Notes

I tried a TypeScript-based sample LSP server from Microsoft, and added REST calls to /console/inspect highlight and completion based on LSP client requests.
Just a bare-bone, hardwire, demo at this point… But concept does work at least to see something in VSCode. You can see it’s pull the real interfaces= from RouterOS, and knows about bad syntax from “highlight” (which are “Problems” in VSCode terms).

I disabled any RouterOS plugin, so graphic is using ONLY my experimental LSP (with minimal VSCode extension in same code to cause LSP to be used).

Now where the concept start leave you hanging is type definitions/constaints. For example, “:beep” takes frequency= and duration= - both take “num” types, but “inspect’s completion” get you some text describing what allow (and it NOT always consistent):

{
    completion: '<value>',
    offset: '16',
    preference: '-1',
    show: 'false',
    style: 'none',
    text: 'literal value that consists only of digits, letters and characters -.,:<>/|+_*&^%#@!~',
    type: 'completion'
  }

The “inspect highlight” will catch errors, but even CLI does NOT enforce those as “num” (and waits for error), so at some level at least on par with CLI.

Also, I didn’t deal with coloring – there are actually a few ways to handle them - so skipped it for now*. And the existing RouterOS editor plugins do most of that pretty well. * Colors come from either being explicitly setting color on specific text range (which is hackish, since colors would be fixed), or would require mapping “highlight” token to LSP’s known tokens (which is not one-to-one nor perfect aligned & more complex).

It actually be a lot more work to package up even my LSP sample+REST-API code… One problem is SOMEHOW the connection to RouterOS needs to be configured (include username/password) – which is not something LSP typically do & opens up WHERE to store username/password. I have them hardcoded for testing. Additionally, it not clear how to know if REQUIRE attributes are provide – that I don’t know. But it is sum of all the “little things” that make this a bit more tricky than just GIF to be usable by others.

Supports Neovim

And did a test tonight, also works in neovim. This one show it’s actually using 7.20beta2 commands from the connect router… since nvim knows about EVPN:

I’ll checkin the code I have tomorrow, even if it’s far from complete.

Code is posted on GitHub and VSIX file can be downloaded from there with the RouterOS LSP for VSCode. See post #1 and README.md on GitHub




At this point, it should not break anything – since it just reads routeros & can be easily disabled/removed from VSCode. And does a few common “LSP things” - at least to some degree:

  • completion (like “tab completion”)
  • semantic tokens (for colorization and based on “highlights” from /console/inspect)
  • diagnostics (shown in a few places, mainly “Problems” in VSCode terms)
  • configuration (required since it controls the connection to RouterOS used by LSP)

Feel free to post any problems, questions, or suggestions with the RouterOS LSP below…

1 Like

routeros-colors-in-lsp-on-alpine-armv7-container.png

Now Published as VSCode Extension!

The newest (v0.3.4) release now appears in VSCode’s Extension ‘Marketplace’, to allow for easy install and update. To install – in Visual Studio Code — use Shift + ⌘ + X to bring up “Extensions” in VSCode, then search for “RouterOS LSP” and this plugin will appear. Next, hit the “Install” button, where it will prompt you it’s pre-release from an unverified developer. This method allows the extension to be updated as fixes and new features are add.

Once the LSP is nstalled to VSCode in all case it must configured to use a RouterOS device via REST API to get

/console/inspect

data. In VSCode you can bring up “Settings” using ⌘ + , then search for “RouterOS LSP”. There the “baseUrl” (https:// — without trailing / or path), “username”, and “password” must be set. The README.md explains more on configuration — like using a RouterOS account with less permissions since only

/console/inspect

is used and NO writes are done by the LSP.

Improvements

Made some improvements from the prototype (v0.1.x), so we’re at the 0.3.x chain. As described in #1 post, the RouterOS LSP can be downloaded from GitHub Releases, and the README.md for the project has installation and other details. Also there is new CHANGES.md that details the known issues and has a version changelog.

While other fixes to various behaviors… some notable changes include:

Fidelity to RouterOS Colors

LSP’s semantic tokens are now matched one-to-one with RouterOS “highlights” used in CLI. So the default color scheme (at least for NeoVim and VSCode) should roughly match RouterOS color scheme. The semantic tokens also let the user customize specific parts like a “cmd”, “dir”, “arg”, etc. as desired. It does not yet colorize types like “ip”, “num”, “array”, etc yet –

 /console/inspect

does not provide those, so they’d need to be inferred by regex (not done).[/i]

Better NeoVim Support

The provided example Lua script (nvim-routeros-lsp-init.lua) to load LSP into NeoVim has been expanded and “corrected” in a few places. The example now adds semantic tokens support to use same coloring as VSCode (and RouterOS). It also by default enables “autocomplete”, which always show the completion options if / : = are typed. This makes NeoVim (nvim) act a bit like RouterOS “hotlock” feature. As such, it can be disabled by changing the “hotlock” to false in “settings” at top of the Lua script. Since example setup Lua avoids person preferences (so there is no ⌘+Space configured), to let user customize as needed, completion can always be done using ⌘X then ⌘O (for “omnicomp”) even when “hotlock” is disabled.

Logging and Performance Improvements

Completions, syntax checking, and more all make a REST call to RouterOS’s /console/inspect. So reducing the number of calls needed has a big impact on performance. In the current release, the number of calls are reduced, more could be done like caching or avoiding duplicative calls. Also the new release does “cleanup” the logs to avoid using large “dump”, so easier to trace (including # REST calls). To access the logs in VSCode, use Shift + ⌘ + X to bring up the “Output” panel, and select “RouterOS LSP” from dropdown. For NeoVim, see their docs on accessing LSP logs
routeros-colors-in-lsp-on-alpine-armv7-container.png

1 Like

Looks great, I’ll try it out later this weekend!

If you do a “hover” when the LSP loaded, you’ll see the tokens /console/inspect request=highlight returns after parsing script code in input= uses — thought of your BNF crusade when I added that. And it may get hidden behind an options

Really interesting! Thanks for sharing this

Great Work!

I would love to be able to access cache size and hit/miss stats as I test the VS extension.

Do you think it’d make sense to combine static syntax highlighter with this extension?

You can totally combine them… I think. I don’t provide any regex — kinda expecting that existing extension are still installed. From VSCode POV, the LSP does a language-configuration.json but it just the basic brackets no syntax things.

What the LSP actually adds is “semantic tokens” like “dir”, “cmd”, “arg” - even tokens for global and local vars. Now I’m not an expert in “normal” way so how to integrate those “token” if available, but there assigned a default color which can overriden elsewhere.
The NeoVim configuration actually shows the set of tokens used pretty clear.

And one thing RegEx do provide – that that LSP can’t get from /console/inspect – is type information like “ip” or “num”. So those have no “semantic tokens” right now, but that is where some RegEx be needed in LSP.

But even error checking is really just looking the “error”, “obj-inactive”,etc semantic token found by LSP in rsc file to create “Problems” in VSCode or red text in nvim when in command mode. So these “semantic tokens” are kinda important in this scheme - which all come from /console/inspect request=highlight.

Anyway I have done little optimization of things — kinda just want to see it work in some coherent way first. The “hover” event is handled by LSP but it mainly for debugging and certainly it’s not efficient since it generates a shit ton of requests (which are cached in LSP code if file has not changed)

This was suppose to be my first post in new forum. However it was put into “awaiting approval” for some reason - no attachments, maybe links, dunno…

And with forum upgrade outage, I got hours of time back…

So made some smaller updated the RouterOS LSP VSCode/NeoVim extensions above with the new found time, see CHANGES for recent updates to the RouterOS LSP

And for fun added a “companion” extension – TikBook for RouterOS – that provides a VSCode Notebook interface to RouterOS script. This was actually pretty straigtforward & nifty. The TikBook extensions uses the LSP but provides a “notebook format” to allow mixing Markdown and RouterOS into one .rsc file - that works as both as valid RouterOS script but visualized/edited as notebook in VSCode. It even lets you run code in “cells” (or entire notebook) via the REST API with a :execute, if credentials are provided in Settings.

tikbook-screenshot

Both the RouterOS LSP and Notebook VSCode Extensions are now installable directly from VSCode (search TIKOCI in extensions) and also via Microsoft’s VSCode Extension Marketplace.

If any “testers” have noticed, there are few updated builds. There is not really much of change in LSP itself actually. I’m trying to get VSCode for Web to work – at least with CORS proxy (see below). But if you see message about a “new version” (that does nothing new), it’s because it hard to test VSCode for Web Extensions, so just been pushing update to the “web” side of the code.

VSCode for Web support…

What is it?

VSCode for Web is a version that runs in web browser, but support some extensions as desktop. You can try it at https://vscode.dev. VSCode for Web is also used on GitHub when you type . whin a repo and that brings the repo up. Or, on desktop, normal VSCode can launch it via code serve-web.

WIP for Router LSP

I have started the “plumbing” need to build extension for VSCode for Web. So while RouterOS LSP loads in Web edition, there are still some TS/JS/ESM/CJS “packaging” issues to work out. These are fixable. But the larger issue with running RouterOS LSP in VSCode for Web is CORS.

The effect of CORS is that the RouterOS LSP (and TikBook cannot make any calls to RouterOS’s REST API when running in VSCode for Web. The REST calls are absolutely needed LSP, and also if you want to “run cells” in TikBook (which can read/write RouterOS TikBook without REST). There is no client-side workaround to it, the server must return the right headers in OPTIONS preflight request — it just how it works. And RouterOS does not send the right things for CORS support in browser for REST API.

Using CORS proxy as workaround

I have an open feature request for CORS support in REST API for “some time”, as it blocks the modern SPA approach. But one “workaround” is to use a “CORS Proxy” – which really just another web server that sits between VSCode for Web clients and RouterOS, that “fakes” the need response to CORS request from browser. I’ve tried a few approach to CORS proxies as /container before like NGNIX, Traefik, and recently Caddy:

X.509 client certs for REST API authentication be ideal…

The “original” approach using NGNIX has ability to use a client certificate for authentication – instead of just passing username/password through proxy to RouterOS. This is useful approach since storing credentials in VSCode configuration is far from ideal. However, add even more complexity to CORS since now there is cert management task too - also something not so simple.

But eventually I’d like the RouterOS LSP to have the ability to “Use X.509 Client Authentication” as alternative to asking for username/password. I still need to try X.509 with Caddy, but if that work I’ll probably publish a /container pre-configured. Now the container still needs a username/password to proxy, but having the RouterOS password on itself is less problematic than random browsers.

Ironic side note…

In discussion about the new forum complaints about “syntax coloring” in RouterOS script on the forum came up quick – so that was good to know that people do care about color highlight. But irony comes when my post about “Caddy as CORS proxy” was sent to “pending approval” by new forum software & now MT forum admin now dealing with CORS to enable coloring in this forum - just can’t make this up:

Which is different CORS problem than here. I suspect the problem on forum is that Discourse comes with a restricted CORS setting out-of-box. But enabling languages in highlight.js library used by forum software, it likely gets the “extra languages” like RouterOS by downloading them from GitHub/NPM/CDN – but since that is a different domain the “forum.mikrotk.com”, CORS on your browser blocks the download for highlight.js’s version of RouterOS syntax. Which is actually a different CORS problem but point is #CORSucks

2 Likes

Open Questions & Scripting Mysteries

In developing the LSP, I have some “open issues” that limit potential features. If anyone had ideas on any of these topics, feel free to reply.

So to answer @Kentzo question:

TL;DR answer is Yes. But exactly the needed RegEx start to get complicated since it not just looking for simple keywords and tokens…

Now could be JavaScript/TypeScript code too, since that what the LSP is implemented in.

But some “static” analysis is actually need to “fill gaps” in /console/inspect data.

Specifically the following things…

  1. Regular Expression for Types?

Anyone have a list of regular expressions for RouterOS types - like “str”, “array”, “ip-prefix” etc?

This be useful to produce more semantic tokens in LSP with the RouterOS types. /console/inspect highlight= only get variables (and dir, cmd, etc) - not types. So this limited what “validation” can do in VSCode’s “Problems” (“diagnostics” in LSP world) and LSP “missing” some script error that it could find. e.g. since some commands RouterOS LSP can know from request=syntax what the allowed types and for numbers allowed ranges (at least to some degree).

  1. Computing “cwd” is needed to validate args and provide help and more… – but hard

The LSP currently does not find commands, which is needed to use request=syntax, that allow more “schema” information for found tokens - or do thing like link to “help”. The problem is while “dir” and “cmd” can be identify via request=highlight – how they relate back the “current working directory” of RouterOS is not so easy to parse out, since it computed as it goes.

An example may help explain, RouterOS often uses syntax where “dir” is on separate line, or even using { } blocks:

/ip address
set address=1.1.1.1 interface=ether5
/container {
add remote-image=ghcr.io/tikoci/netinstall interface=veth1
mount add src=/ dst=/ list=mymounts
set netinstall mounts=mymounts
 }

The problem is if, for example, use the “Hover” editor feature, and hovered over “mount” above… the LSP would need to know the path= is “/container” to lookup “explanation” for help info for mount… but that single line has no idea what the parent path is…

I’m really not sure of an algo/heuristic/rule that find the path even… So ideas on doing this be welcomed. Since determine the path is critical to using /console/inspect request=syntax — but it needs to know a path=ip,address for example to do that…

I wrote up what /console/inspect request=syntax can on the readme:

But request=syntax only usable if the “current directory” is known to the LSP is the underlying.

Plan right now to at least look for for simple sequences of dir+cmd+[arg] until within a line (and until a “dir” or “cmd” is found after “cmd”)

Theoretical Approach: using [:parse] and the code types…

I have thought to use the[:parse] command (via REST), but I’ve never written a parser for it and even less is known about it’s format… But that is one idea I have in my head about this, not sure it will work… it likely has markers for variables and does know the stack. Now problem is linking editors “position” with the parse command is where this approach, even if possible, set even more tricky. i.e. these no .map file like JS has between RSC transpiled into it’s :parse/“code” type.

  1. Finding :local/:global variable scope - all { } blocks are not created equal?

Another thing the LSP does not know about is scope. Again, some RegEx be useful to identify “scoped blocks” but it beyond just regex match { with another } is the problem.

And being able to do be useful since the LSP can offer a “rename variable” refactoring option. For :global, I believe, within a single script it “syntaxly safe” to just rename all found since request=highlight can identify global variables in script.

The issue is :local variable is the scope is not entire clear from the { and } usage and require way more nuance. And I’m limited by docs and while I know most of the scoping rules when I see script but reducing it a RegEx is not so easy.

If anyone had ideas on how to identify a local scope in RegEx that be very useful to the LSP development.

Some example where it’s pretty clear:

{
  :local outside "outside"
  /ip address { 
    :local inside "inside"
    :put "ip seeing '$outside' (outside)"
    /interface {
        :local reallyinside "reallyinside"
        :put "my parent is '$inside', but I can see '$outside'"
        ethernet {
            :put "ethernet is '$reallyinside' (reallyinstead)"
        }
    }
  }
  :put "looking from outside i cannot see '$inside' (inside)"
}

which returns the follow, with local variable inhered into the /ip/address.

ip seeing 'outside' (outside)
my parent is 'inside', but I can see 'outside'
ethernet is 'reallyinside' (reallyinstead)
looking from outside i cannot see '' (inside)

Now once functions involved – local scope does not inherit the parents for local function. So with this:

{
    :local outside "outside"
    :local fn do={
        :put "local function can see '$outside' (outside)"
    }
    :global FN do={ 
        :put "but global cannot see '$outside' (outside)"
    }
    $fn
    $FN
}

you’d get nothing, since function do= blocks get a new “stack” AFAIK.

local function cannot see '' (outside)
nor can global see '' (outside) either

I’ll probably add a “global variable rename” support from LSP. But the :local is trickier.

And I’m not sure “local variable rename” is all that useful… But just from a BNF/syntax view, it would be good to identify “local scope”, somehow.

1 Like

I think best you can do for “relative” commands is to have a static definition of a command as a union of all known signatures. Or provide multiple signatures, if editor / LSP supports it.

I think the approach here is to describe the language via PEG and generate an AST.

Never heard of PEG grammars before. Little cleaner than BNF…

That might be better approach than RegEx, especially for types detection (…coloring types).

Going to mull on it. But Claude was hopeful on the concept, with a couple prompt:

PEG grammer example
https://claude.ai/public/artifacts/0ab650e8-5344-4611-96f9-ed3816a9a47f

TS implementation example
https://claude.ai/public/artifacts/772d4de1-4582-4f7c-a556-777f16e287c9

and, yes, I can already spot error/issues in LLM generated code/grammar - but boy in 5 minutes, I can see the concept.

I’m going to mull on this, but would like to avoid RegEx parsing and have a more normalized approach when /console/inspect fails.

I kinda want to get the “VSCode for Web” plumbing figured out for the LSP first, since I do use GitHub.dev a lot, so I kinda want these tools to work there first ;). And connection failures/errors are handled very poorly too - that should be fixed.

I just never bothered to research how to use AST for highlighting in VSCode / Sublime Text. The extent of my knowledge is grammar based on RegEx (TextMate’s legacy). Is LSP flexible enough?

In LSP is also about give some control of UI things (hover, “problems”, context menus, etc.) using a generic interface - why this works in neovim. So you can add all the various nicety like a context menu or nvim shortcut to see all the options for a found command. And it can this efficiently since it not dozens of regex’s.

Some AST help with what LSP calls “symbols”, which the currently LSP does not provide. But that’s what allow things like “Rename Variable” in an editor, or “Goto to Definition”. For example, clicking on :put $myvar could take you to some :global myvar elsewhere in file. And for :local, if you have multiple :local myvar, the LSP still need some what to which are related - symbols provide that info.

Importantly, for the “current directory problem”, symbols track a “call hierarchy” so imagine that could be used to find a “dir” path (e.g. /ip/address) from some line/char in the script – with that more is possible in the LSP.

Now… to get symbols, you something that parses the script to build them. so if some AST did exist, it could be parsed by the LSP code to come up with these answer without much work. But the LSP still needs custom code to bind any AST (in whatever form) to the LSP API — that’s not automatic.

What the LSP does provide today is “semantic tokens”. There are related to symbols, but the tokens are what is used for colors. And each character in a script is assign ONE token. Right now the LSP matches semantic tokens with the output of /console/inspect request=highligh so it 100% match with what the CLI provide in WinBox/etc. Except your editor is allowed to define what color maps to the “semantic tokens”.

So for colors, no AST is strictly needed. But for “symbols” require much deeper upstanding of the structure, why if there was some AST existing that be very helpful. But merely creating an AST alone is not the whole store, since it still needs to be map to what an LSP needed to return to editor somehow.

Some halfway house be start by trying PEG to detect RouterOS types, so they can be colorable “semantic token”.

I assume some sort of static grammar (AST or RegEx) is still needed to highlight cases like:

:local myFunc do={
    set interface=...
}

My current understanding is that tokenization and syntax highlighting can be mostly done statically and you only need connection to an actual device for completion. That being said, I think access to /console/inspect is key to figure out the grammar.

The LSP currently does only “dynamic analysis” (making up terms now) using /console/inspect to extract meaning for an arbitrary script provided as input=.

It does not need an AST since, essentially, request=highlight is providing the results of using an AST — which is really what’s needed to get some coloring.

Problem is request=highlight only tell half the story… if I modify your example, and show a letter that represents the /console/inspect obtained “highlight token” you can see what we get today “dynamically”:

:global myFunc do={ :local myLocal ether1; /ip/address/set [find name=bridge] comment=test }
Dcccccc-VVVVVV-^^@@-Dccccc-vvvvvvv-------@-DDDDDDDDDDDDccc-@cccc-vvvv@vvvvvv@-^^^^^^^@-----@

where:
D = dir
c = cmd
^ = arg
- = none
V = global variable
v = local variable
@ = meta / syntax

and each of those become semantic tokens, that can be colorized. There are about a dozen of possible results, so just show one string and how it’s parsed to match RouterOS. The “highlight” data is presumable what MikroTik uses to generate ANSI code in CLI – why this LSP perfectly matchs it.

But you can see there is not real “arg value” type, and but stuff like an =bridge is treated like a variable.

Critically, the is also “error” type that’s possible. So your simpler function uses =... but that’s actually an “error” (I get it’s example, now example-in-example):

/console/inspect request=highlight input=":local myFunc do={ set interface=... }"

       dir           
       cmd           
       cmd           
       cmd           
       cmd           
       cmd           
       none          
       variable-local
       variable-local
       variable-local
       variable-local
       variable-local
       variable-local
       none          
       arg           
       arg           
       syntax-meta   
       syntax-meta   
       none          
       none          
       none          
       none          
       cmd           
       cmd           
       cmd           
       none          
       <mark>error</mark>         
       none          
       none          
       none          
       none          
       none          
       none          
       none          
       none          
       none          
       none          
       none          
       none          
       none

Anyhow, that’s how /console/inspect dynamic analysis works. We’re letting /console/inspect maintain the AST, and just using it to get parsed results from that AST.