Picking up the thread on why LLMs struggle with RouterOS — and what actually helps — I've spent the last while turning the hand-waving into measurements. It's all open at tikoci/bench-routeros-tools, a small benchmark that compares six ways of helping an agent: a bare model, curated skills, a docs-retrieval layer, an MCP "do everything" server, and combinations. Same prompts, same scoring, run against a live CHR so the numbers reflect what a router actually accepts, not what looks plausible. The write-ups are in the REPORT and the forward-looking AGENTIC_FUTURES if you want to poke holes in the methodology.
A few things fell out that are directly relevant to what @Larsa and others raised above:
-
/console/inspectis necessary but not sufficient. The thread's instinct to validate generated config before showing it is exactly right — but inspect is a static gate. Our sharpest example:/ip route add dst-address=… blackhole(a bare flag) is the device-valid form, yet/console/inspecthappily accepts the invalidblackhole=yes, andtype=blackholeis a pure hallucination that 30 of 36 model runs produced. Only live device execution catches that gap. So you really want two gates: parse/inspect and a dry run on hardware. (My hope is that the new manual.mikrotik.com grows a proper CLI reference with more data than/console/inspectexposes — enums, value constraints, deprecations — which would close a lot of this on the static side.) -
This isn't a weak-model problem you can scale away. We ran the same trap up a ladder from small models to the current frontier. The best model still scored 0/9 on it. A grounding/validate→run layer is a permanent correctness floor, not a crutch for cheap models.
-
More tools is not more help. This is the gentle part re: the mikrotik-mcp server that's been recommended around here — it's a genuinely useful project and the only one of the six that can execute. But measured honestly, its ~166 SSH-backed tools cost roughly 28K tokens of always-on context, create real tool-selection ambiguity, and ship ~27 destructive operations with no dry-run. By contrast, a read-only docs layer buys most of the planning accuracy for a fraction of the context. The data points away from "stack one giant executor" and toward an explain → validate → run split.
That split is what the TIKOCI projects are built around, and it's all public:
-
rosetta — RouterOS docs (pages, properties, the command tree, device specs, changelogs) as a SQLite/FTS RAG exposed over MCP. This is the cheap, read-only "explain/plan" layer.
-
routeros-skills — curated, grounded skills (firewall, scripting, containers, netinstall, …) usable from Claude Code or Copilot. That includes a MAC-Telnet skill, which grew out of the wire-format work a few of us did upthread — so an agent can reach a device by MAC on L2 with no IP yet.
-
quickchr — CLI + library to download, launch and manage CHR VMs under QEMU. This is what makes the "dry-run on hardware" step cheap: spin a throwaway 7.x router, validate against it, throw it away.
-
centrs — the in-progress scoped-verb MCP that realizes the recommendation: a handful of verbs (
explain,validate,retrieve,execute) over a canonicalize→validate→run core, withvalidaterunning both:parseand/console/inspect, and execution gated per-device read-only/read-write. It's the deliberate alternative to a 166-tool firehose — and MAC-Telnet is being folded in as one of its transports, so the same scoped verbs work whether the box is reachable by API or only by MAC.
None of this replaces MikroTik publishing a formal grammar — that would help everyone, and I'm still firmly in that camp. But you don't have to wait for it to get reliable agent help today: ground the model in real docs, validate against the command tree, and dry-run on a CHR before anything touches a production box. Happy to share the full report and methodology with anyone who wants to dig in.
Drafted and final reviewed by me, edited by Opus 4.8