I have another project, GitHub - tikoci/bench-routeros-tools: Benchmark RouterOS agent-support strategies across MCPs, skills, and retrieval. · GitHub that I've been testing various models and tools against "problematic for AI" RouterOS tasks. It a bit of mess if your try to read the docs, but the there are actually a few different tests performed, and importantly I'm still trying to structure them into proper benchmarks. The difficult part if identify potential test than can be "scored".
I added the "steering to manual.mikrotik.com" recently so see how it compares to "baseline" (no prompting, no tools), using Rosetta MCP, and my routeros-skills. So tikoci/bench-routeros-tools now add using the prompt in top post as test cases. I have run it only on Claude Haiku (there "cheap" model).
The raw data is stored in the GitHub, so more analysis can be done on past runs in future. _I'll re-run on Sonnet and/or Opus, but that burns credits/hits limits, so plan to review the current set to tweak tests. But I had Opus (high-end model) to review the results of the tests, quoted in part below (and see CORPUS.md linked below to convert the "short-hand names" for the test cases)
Claude Opus's Report on Haiku-based test
The sharpest cross-cut is vendordoc-steer ("Steering AI to use new manual.mikrotik.com"): it has the best fabrication discipline of any column (31/33 syntax-valid — reading the real manual kills invented args) and it almost always actually fetched the page — yet it still misses all three device-truth traps (blackhole, unreachable, dhcp-default). So reading the vendor manual ≠ knowing device truth: the page doesn't make the bare-flag-vs-type=, the deleted-unreachable, or the disabled=yes default legible enough to change behavior.
Net: at Haiku the ranking is rosetta > steer > baseline > skills, but the headline isn't the ranking — it's that the four real discriminators (two route traps + dhcp default + wg) all point at the same gap, and only the validate → run device tier closes the part that grounding can't. Honest-matrix bonus: the nine false route-unreachable "perfect"s are gone, so the per-column totals above are now real.
HELP WANTED — Looking for suggestions for tests where AI "gets it wrong"...
Please comment below with what prompt you tried, the results, and what you'd expected. I'll try to add more cases to my benchmarks. The current benchmark have the following test cases:
These can be run against a variety of models/harness. But most recents tests are the "live results" series using claude -p, with options to control how the context gets loaded so as not pollute it with any local learns/skills/~CLAUDE/etc.