🤔 Discussion: Lint'ing and "Scoring" RouterOS configuration

RouterOS
1

Spin-off from Important security update

I've been thinking about how to "lint" RouterOS for a while. I already build RouterOS LSP, and have Rosetta MCP for AI. While these tools can "check" if a config is "valid" or has errors, and AI more easily find RouterOS fact from Rosetta... Nothing in my toolchain actually did any meaningful analysis of a configuration, which is important if you want to "surface" important configuration details. My general focus has been on offering "help" (for example linking docs to some hover in VSCode with a config). But the recent discussions highlight that some security-related and likely "config mistakes" also depend on deconstructing config to "cananocialize" into a normalized form that allow lookup/checks on the various path/cmd/attributes. In other words, some "abstract syntax tree" (AST) suggested by @Kenzo a couple years ago:

I've loosely been working on this as part of:

Which essentially designed as a library that has a CLI that matches the library, with the idea that it "safely" executes/fetches RouterOS things, including multiple to run a command against a set of routers (defined in centrs by CDB file's groups, which centrs uses to store creds).

But the "safely" part is tricky... So a critical part of tikoci/centrs is centrs explain which essentially builds an AST... since things like determining if a command "writes" before trying it is useful (and centrs has a option so read-only mode so even if account allowed write, centrs could still block itself).

explain is actually near a more completed state. The clunkers tell me the offline parsing understands 67% of syntax. And the AST generator inside explain is "scored" against a large collection of @rextended, @eworm, @tangent public scripts measured on live router to compare our offline parser... which is how I know what % is ~done. I have not measure the parser WITH a connected router, but the AST generation only get more complete, since with a real router, the syntax can be exactly checked to know if it valid (and router has needed package/architecture/etc).

But this is plumbing needed for some of the suggestions from the discussion in "Important security update", start with:

And already been at that for a while, so the centrs work is essentially the core idea suggested here:

... since I use the LLM to generate the code needed to static/offline/non-AI RouterOS anylsis. So that no LLM is needed to "lint" RouterOS configs... And if you do use AI, then the LLM can help more with by starting with linting and AST-like information.

Now one part I have not looked at as much — thus this post — is "linting rules"... So like ask for help for what some potential rule sets look like... with the idea you can run centrs (or some future tool that uses centrs' AST) on your own computer without AI. My thought is to essentially create a RouterOS version of biome linter for JS/TS/CSS/etc, https://biomejs.dev as a layer above the centrs explain (which provide the facts needed to then check against some set of rules).

Which is essential core idea from @optio's post:

For example, one potentially "@rextended approved" rule might be

style/no-unnecesary-semicolons - warn if semi-colons are used when not needed

which use centrs explain AST to look for semi-colons at the end of RouterOS script line... but you need the AST since some multiline KV arrays do need semi-colons since they are an array separator not command separator....

But the whole security patch discussion got me thinking perhaps some more security focus "linting" checks might be good. While I could eventually come up with some set, if folks has some suggested rules — both ones that could be checked against a parsed config file using more conditional logic, and ideally phased like biome things with some no-XXX / use-XXX. Like some theoritical security/use-https and security/no-unsecure-http that check www/www-ssl config, etc., and like biome offer a fix/suggestion to enable /ip/cloud+/certification/enable-ssl-certificate. In this example, it actually flag the default configuration.

With the idea being some are opinions and possible rules do conflict... and someone using my linter config disable/enable sets of rules (perhaps organized as set for "switch"/"home"/"ISP") etc.

Anyway I'm still working on the core code for AST... but thought some discussion of possible "linting rules" (which could feed into any kinda "config scoring" we want to conceive). So if folks have suggests on potentials rules or even better how to structure rules so they are usable, please comment below.

2

Can you start from Monaco ?
So it's all web based.

So maybe you can even make a container in RouterOS with it.

Maybe I am on the wrong way of thinking, i didn't get what your goal is.

3

My contribution is uno, and then only if you ignore the minor detail that it's an ROS 6 script someone else wrote, which I then hacked to work on ROS 7 in the early days, when I could find no canned solutions.

But since you have implicitly solicited my comment by pinging me…

AI more easily find RouterOS fact from Rosetta

Yes. Much of the carping about how bad LLMs are at writing RouterOS configs comes down to not having a formal grammar for generating them, or a FOSS compiler to test the resulting code.

Fully closing the loop would involve automated CHR setups with RSC injection, using a lightweight VM system like KVM on Linux or applehv on macOS. A brief LLM "rule" would then teach it how to test its spew on RouterOS before offering it as a working solution.

Not a trivial undertaking, admittedly, but with a tremendous payoff, if only because it gives undeniable feedback about whether a given configuration parses. Add an /export round-trip, and you will squish hallucination early and fast.

if folks has some suggested rules

Not really my area of expertise, but I'd start with the dead obvious:

  • error hard if TELNET is enabled
  • gripe bitterly if you see PPTP enabled

That type of thing. Basically ones where if someone is arguing against it, you know they're a troll or one of those can't-help-themselves quibblers.

Thank you for doing something positive about all this, and not merely joining the carp-and-moan squad.

4

I’m trying to better understand what you are aiming for in terms of practical architecture.

Is it some sort of symbolic execution for configuration files and scripts that you can query later on based on different rules/outcomes?

5

Fair question. For several years, I maintain a set of RouterOS "tools". Most development before AI, and with a few new ones almost entirely coded by AI. So for context here, I already have projects that build a static "schema" for RouterOS, and other code that use /console/inspect that allow verification that a script is "valid" (in the sense the syntax will be accepted, not that it's correct) in an LSP for VSCode/other-editors.

While the existing OpenAPI schema in restraml is already useful as-is for AI agent doing REST API. And Rosetta allows searching of RouterOS docs/changelogs/videos/product-catalog/test-results/etc/etc. Plus there is quickchr that brings up CHR for unit/intagration tests. So these already help for AI agents.

The "gap", as clunkers say, is that most relevant is actual configuration script, but nothing in my tool chain specifically help much with even checking if a configuration is "good" (or not bad, how bad, etc). There are no "best practices" it recommends... only pure facts from either `/console/inspect or some *.mikrotik.com page.

Basically nothing provides an JUDGEMENT. While the forum is good for that one, it's often the same judgement repeated in a new thread. And while some list like @jaclaz various 99 thesis's are close to "rules" but they more helpful for a SKILL.md, as many all actionable/checkable "mechanically" (e.g. with code/script plus an AST of config).

All my existing tools can help with syntax and attribute knowledge, nothing applies an "common sense" to a RouterOS configuration file... so more can be "flagged" (in the general sense, but akin to soft flagged=yes verdict). So some set of deterministic rules to check the actual config that routes/switches is one "missing link", since today all I can say is that "you got all the attributes right" and/or "command won't error". But if you want to enable telnet.

centrs explain does the lex'ing and tokenization in TypeScript without a router first... but if one is available it "enriches" the AST results since more can be resolve with a live router that matches the config file. But already explain beyond the data from /console/inspect (which basis for much of my work in TIKOCI).

centrs is a bit broader than just the "explain" part that knows about an AST, since it also deals with running commands (including always running through the "explain" code so that bad commands are prevented). But the issue is "bad command" is merely that the command won't run... not a judgement that it may not be a "good idea".

Centrs already has an existing error/warn/tip system for all of its commands. So linting errors map into that from the "explain" results. But the rules themselves likely follow the data structured used by biome, see https://biomejs.dev/linter/#configure-the-linter. So the rules don't have to be universal... as "RouterOS linter" might have some domain (borrowing biome's term) for a core router vs a home router vs PtP link.

From an overall architecture POV, centrs focus is as a TypeScript library that anyone/thing can use, that has a solid base of api/rest/mac-telnet/etc protocols supports, and for security uses WinBox's CDB (which does NOT have to your main CDB used by WinBox) to make it easier to handle auth needed. There is API Reference on what centrs does: @tikoci/centrs. The CLI is usable "surface" for the core code... but centrs itself focus is being the "shared library" for my various projects. So the "explain" AST stuff is needed since part of the concept is "safer" way to interact with RouterOS... but to be safer you the code needs to be able to unwind a config file.

A "alpha" prototype is already on NPM. So if you https://bun.sh installed, you can see what the offline support produces by using bunx @tikoci/centrs@next explain --help. So anyone can see what's found today.... either provide a --file <filename> (or pipe using -), or you can provide a string (with single-quotes to avoid POSIX escaping):

bunx @tikoci/centrs@next explain '/ip/address/add address=1.1.1.1/24 interface=\[/interface/find default-name=ether1\]' --tokens

verdict: pass — 1 statement(s), 0 diagnostic(s)

gate: script · 0 arg(s)

write: true

statements:
  \[0,82)       resolved  command  path=/ip/address verb=add  via=execute
    centrs: centrs execute '<router>' '/ip/address/add address=1.1.1.1/24 interface=\[/interface/find default-name=ether1\]'

subcommands:
  \[45,82)      resolved  command  path=/interface verb=find  (in /ip/address, depth 0)

symbols:
  \[62,74)      unknown   field       name="default-name"  (S8 filter-field: schema-dependent (local if a real menu field, undefined if not))

values:
  \[24,34)      attribute  name=address shapes=ip-prefix

tokens: 8 token(s), 40/82 byte(s) classified (48.8%), class provisional
  \[0,12)       dir
  \[12,15)      cmd
  \[15,24)      unclassified
  \[24,34)      value
  \[34,46)      unclassified
  \[46,57)      dir
  \[57,61)      cmd
  \[61,82)      unclassified

evidence: e0 canonicalizeExecuteCommand (direct), e1 analyzeCoordinates (direct), e12 pathSpans (heuristic), e4 resolveVerbs (heuristic), e5 resolveDocument + resolveVerb (heuristic), e6 containsWrite (heuristic), e7 resolveSymbols (heuristic), e8 classifyExplainTransport (heuristic), e9 valueShapeHints (heuristic)

runtimeAcceptance: not-proven

Tips:
  - \[tip/explain-offline-only\] No router was given, so this is the canonicalizer's reading alone.
    fix: Completion, schema, and \`:parse\` evidence come from a live target (\`centrs explain <router> '<input>'\`), which is phase 2 and not accepted yet.

That's possible. I already have Monaco-based /app YAML editor, RouterOS /app YAML Editor - so it's possible. But I'm still working on the core code for some offline checking.

My main focus is to get the newer centrs explain support back into my existing RouterOS LSP to make it better.

I have also thought to might be possible wrap it as Discourse plugin, and see if MikroTik use, so that some special ```routeros check in a post would render colors, and errors (or ideally more "lint" things) via a plugin with the centrs explain code.

Today, you can theorically use the RouterOS LSP with https://vscode.dev which uses Monoco... But this gets complex to setup since it requires CORS proxy setup in-between). And also the /app code-server let's you install the RouterOS LSP, so if you connect it a router, that also give with syntax checked of RouterOS - that works without proxy server, since essentially the code-server container is running the RouterOS LSP.

So basically one some "offline support" make it into my existing RouterOS LSP, it be trivial to wrap in Monaco, since there be no need for a live router. Now results would not be as accurate without a live router. But any "rule violations" are exactly what I want to show in some VSCode/monoco "Problems" view. But a few more steps to get there. One of which is ideally expanding the "warnings" and "tips" for actually RouterOS config items, not just the structure is valid.

Well, actually your collection of defconf script is part of the RouterOS "test corpus", see lsp-routeros-ts/test-data at main · tikoci/lsp-routeros-ts · GitHub which then has a DB of /console/inspect and [:parse] results accross versions on the set.... which is how the centrs explain AST lex'ing code is validated/tested.

6

Apologize for the delay, I wanted to actually look at the projects before replying.
It looks like you indeed already laid out most of the infrastructure required to build something more advanced on top.

I think the goal itself needs to be better defined in terms of the audience intended to use it, rather than trying to build a "one tool fits all", which would probably alienate either one or the other.

For example:
For Home users I wouldn’t invest much more than static analysis, with some interpretation of possible WAN to router exposure paths, a quick evaluation of which built in services are enabled/reachable, and then issue warnings based on that. In my opinion this should be fairly easy to implement given the constraints, and wouldn’t require firing up a VM, nor using LLMs.

For advanced users however, that’s where I would draw a fairly clear line between "good practices" and "self imposed requirements that need validation", which would better fit more complex networks.

Instead of trying to evaluate these using generalized rules, I suggest implementing this kind of validation similar to automated software tests, where the code is executed with predetermined input and the result is than asserted against the expected output.

This would essentially allow an advanced user to build a collection of different tests that check whether a configuration implements this or that rule and actually behaves as expected.

The implementation of this kind of test framework could be based on existing software test frameworks such as pytest/XCTest (or rather the js/ts equivalent), while using another layer on top of centrs and something like Scapy to facilitate and observe traffic flows from different points in the topology.

A test would look something like:

  1. Setup - fire up a CHR instance (or more than one if needed), wire the interfaces, load the configuration to the test machine and bring up a local server.
  2. Open a WireGuard connection to port XX on the test machine.
  3. Open a connection to port YY on the local server.
  4. Observe the incoming connection externally on the local server, and assert the input to the output, validate it using a traffic capture or inspect routeros directly to assert a certain expected indication - or do then all at once.

Of course, a test can be expected to produce either a positive or negative result. For example, observing a rejection or timeout when that is the expected behavior would result in a passed test.

The reason why I think this kind of framework makes more sense is that the moment you step outside of what is considered standard usage, the rules become far more complex to validate them statically.

Once you start mixing NAT, policy routing, tunnels, connection tracking, VRFs, dynamic state, etc, you eventually end up trying to reimplement quite a bit of routeros behaviour in the analyzer to be able to predict what will happen.

With an end to end test, ROS itself becomes the thing answering that question.
You provide the input, observe the actual outcome, and compare it against what was expected.

Moreover, every configuration change could then be retested to make sure none of the expected outcomes have been broken. The same would apply to ROS updates too, which could make this useful for spotting regressions.

Given the configuration and the integration with centrs/LSP mcps, LLMs should be able to suggest edge cases and generate tests for them, or generate tests based on the expected behavior, something like "management should only be reachable through wireguard from ip XX", or even a full description of the network which would then generate all the required tests at once.

The important part here is that the LLM only helps to decide what to test and generate it, and is not required either (you could write your own test manually using the APIs). The actual validation stays deterministic and is based on what ROS did in practice, so if the LLM fails to generate the appropriate test, the assertion would fail and the resulting artifacts could be examined.

You could also include some built in "common tests" that aren't inherently dependent on the exact configuration being tested, but rather on the outcome. At that point, it would be rather cheap to include a few examples, or broad use cases, similar to what I suggested for the Home User like "WAN must not reach management services", "guest network must not reach to LAN", etc.