7.26beta [development] is released!

I'd like to share a few thoughts and suggestions about CMR.

  • Automatic backups before upgrades: As you know, every changelog recommends taking a backup before upgrading. However, "/cmr/device/upgrade" currently doesn't do this. A workaround would be to run "/cmr/device/run-script" before the upgrade. But while upgrades can be scheduled, I don't think the same is possible for scripts.

  • Provisioning preview: WiFi and especially VLAN provisioning can cause problems if something goes wrong. The documentation already includes a warning about VLAN provisioning. Wouldn't it be useful to have a preview showing what changes will be applied to the selected devices before actually applying them? This could also include some basic sanity checks and display warnings if potential problems are detected.

  • Safe mode for CMR: As mentioned above, "/cmr/vlan" can potentially cause serious problems. Not everyone enjoys playing Russian roulette. :slight_smile: It would be great if provisioning actions could automatically enable Safe Mode. For example, if a device cannot reconnect to the CMR server within 60 seconds after provisioning, it could automatically roll back the changes.

  • Post upgrade checks combined with run-script. When set of checks fail, then a ROS downgrade to previous version is done.

I think these features would make CMR provisioning and upgrades much safer and easier to use.

Contingency plan?

https://www.youtube.com/watch?v=_B7MzBmjaJ8

Thank you for the report.
Working on the fix for it.

did onyone managed to assign IP to cmr_bridge interface then using vlan rules ?
whenever I try to set a vlan interface (and dhcp client) on cmr_bridge, router establish connection with controller and modify interface, making vlan interface invalid.

Anyone having a ip address conflict with netinstall package installed?

Any chance it can run on a vlan?

In have my lan on 192.168.1.2 but winbox is sending me 192.168.88.10.

Netinstall - 192.168.88.10/16

First 192.168.0.2
Last 192.168.255.254

It's dynamic and can't be changed.

On 7.24.5 I don't have that problem.

TIA

Also, in windox. The cache packages are no longer displayed if I reboot the router. Saved on the usb drive but no longer displayed. Can we fix that?

Thanks for the CMR feedback on beta4, a few updates:

@andrejtom, @Amm0 – the "UI file parsing error" and the missing CMR icons are fixed in WinBox 4.5rc1; please use it with 7.26beta4 and CMR. However most likely they will disappear in stable version.

@infabo – port-labels with several ports work in beta4: port-labels=ether1:trunk,ether2:uplink. Separate the ports with , and the labels of one port with /, for example ether1:uplink/trunk,ether5:office.

@DanMos79 – we could not reproduce the autosupout on device removal in the lab so far. Thanks for the ticket, we'll follow up there. The "Access Points" column label is noted.

A word of caution for VLAN provisioning: always give a /cmr/vlan rule explicit port-labels, and don't leave them empty or unset them on an existing rule. A rule without port selectors applies to all ports of the selected devices, including their uplink, and can disconnect them from the controller.

Thank you for fixing! Already noticed the fix :+1:

@sergejs Thank you very much. Perhaps the ugly rate of 1000.0 kbps can be corrected in one go. :wink:

Since the last Beta when I go to the controller (runnig test setup for CMR) and then settings I cannot make any changes and the controller IP is blank and I cannot save anything. If I do this via Winbox it looks correct and I can make changes. Is this a known issue, and no I have not created a case to MT about this as I feel it is very low priority.

CRS812-8DS-2DQ-2DDQ, 7.26beta3, l3-hw-offloading=yes.
Transit box. Default route via BGP from upstream on qsfp56-1-1, ~35 Gbit/s / ~5 Mpps at evening peak. One /24 directly connected on a VLAN for customer VMs (~180 entries in ARP), servers and downstream customers on sfp56 ports.

We had floods on Oct 20 and Oct 26, the one on Oct 26 was UDP to that /24. Upstream port went to 80-96 Gbit/s, ~10 Mpps. The switch chip handled it, egress didn't grow, the extra traffic was dropped in hardware. But CPU went to 98-99%, on Oct 26 almost all of it IRQ. REST stops answering and our monitoring loses the box. On Oct 20 this happened twice, ~35 min each, and the box rebooted in between.

From our graphs (switch1-cpu counters + cpu load over REST): 600-700 kpps to the CPU already gives 75-80% load. On Oct 26 tx-drop-queue0-packet on switch1-cpu went from ~5k to ~218M in one hour. So even a small part of a flood reaching the CPU kills the control plane.

No supout from the attacks, we were busy keeping things running.

First we thought packets to unused IPs in the /24 (no ARP entry) get punted to CPU for ARP resolution. So we made a script: it reads ARP every second and keeps a /32 permit rule per live IP, then a /24 drop, on the upstream port:

/interface ethernet switch rule
add switch=switch1 ports=qsfp56-1-1 mac-protocol=ip dst-address=198.51.100.10/32 comment="USED 198.51.100.10"
add switch=switch1 ports=qsfp56-1-1 mac-protocol=ip dst-address=198.51.100.11/32 comment="USED 198.51.100.11"
(... ~230 rules like this)
add switch=switch1 ports=qsfp56-1-1 mac-protocol=ip dst-address=198.51.100.0/24 new-dst-ports="" comment="UNUSED drop"

Then we tested with UDP flood to the whole /24, about 5 Gbit/s / 500 kpps. Routes to our DNS servers were withdrawn for the test, they are behind a GRE tunnel which goes through CPU anyway. CPU went up ~5% in both cases: without the script from ~10-11% idle, with the script from ~16-17% to ~22%. No difference. (Idle ~10% is mostly our own monitoring polling REST every second.)

What does work: one switch rule on the upstream port that drops all UDP to the /24 (protocol=udp dst-address=198.51.100.0/24 new-dst-ports=""). We put it first so it hits earlier, but it also worked lower in the list. With it, CPU load went back to normal right away. On Oct 26 the 98-99% was before we added it. The flood peaked at ~96 Gbit/s / ~9.5 Mpps after the rule was in place, and CPU stayed at 10-15%, only a couple of short spikes to 50-80%. Of course it also cuts all legit UDP, so it's only for emergencies.

So drop by dst /24 + protocol works, but /32 permits for live IPs + /24 drop for the rest doesn't change anything. Looks like the CPU load comes from flood traffic to live IPs, the ones the allowlist lets through, not from unused IPs. Maybe it's related to question 5 below, but we don't understand how. The script itself also costs ~6% CPU, ~350 ACL entries, new VMs are not reachable until they appear in ARP, and there is nothing for IPv6.

Simplified config, IPs and VLANs changed:

/interface ethernet switch
set switch1 l3-hw-offloading=yes qos-hw-offloading=yes
/interface ethernet switch l3hw-settings
set icmp-reply-on-error=no ipv6-hw=yes
/interface vlan
add interface=bridge name=vlan900-transit vlan-id=900
add interface=bridge name=vlan300-vms vlan-id=300
/ip address
add address=203.0.113.1/31 interface=vlan900-transit
add address=198.51.100.1/24 interface=vlan300-vms
/routing bgp connection
add name=UPSTREAM-V4 remote.address=203.0.113.0 local.role=ebgp-customer input.accept-nlri=default-only ...
/interface ethernet switch rule
add switch=switch1 ports=qsfp56-1-1 mac-protocol=arp comment="arp permit"
add switch=switch1 ports=qsfp56-1-1 src-address=198.51.100.0/24 new-dst-ports="" comment="antispoof"
add switch=switch1 ports=qsfp56-1-1 dst-address=203.0.113.1/32 protocol=tcp dst-port=179 src-address=203.0.113.0/32 comment="bgp"
add switch=switch1 ports=qsfp56-1-1 dst-address=203.0.113.1/32 protocol=icmp rate=1Mbps comment="icmp to router"
add switch=switch1 ports=qsfp56-1-1 dst-address=198.51.100.1/32 protocol=icmp rate=1Mbps comment="icmp to gw"
(+ USED/UNUSED rules from above)

Questions:

  1. What exactly goes to CPU when UDP flood hits a directly connected /24? Dropping all UDP to the /24 on ingress fixes it, the allowlist doesn't. So is traffic to live, resolved hosts being punted too? Or does the CPU load come from something else (ARP/FDB updates, neighbor discovery)?

  2. In beta4 there is "switch - add a high-priority CPU queue for control traffic on Marvell Prestera switches". What goes to this queue, BGP/ARP/ICMP to router? Will it keep control plane alive when queue 0 is full of punted packets, or it's still the same IRQ/RX path?

  3. If it is ARP-miss, is there a plan for an option to not send unresolved hosts to CPU and just drop in hardware? Router can still resolve ARP with its own requests. Or at least a rate limit for this trap, like glean throttling on Cisco/Arista. If it's not ARP-miss, what is it and will it be fixed? And in general, per trap reason rate limit to CPU, is it planned?

  4. Everything to CPU goes to queue 0 on switch1-cpu, queues 1-7 are always 0. So control traffic and flood are in the same queue and dropped together. tx-drop counters show how much, but not why packet went to CPU (ARP miss, to-router, TTL, ACL copy-to-cpu). Any plans for per-reason counters?

  5. l3hw advanced monitor shows ipv4-hosts ~34, but ARP on that VLAN has ~180 reachable/stale entries. Is it normal or most hosts are not offloaded? If they are not, does traffic to them go through CPU? That would explain why the allowlist doesn't help.

  6. Different issue, same box. Two times (Oct 20 and Oct 25) "/interface ethernet switch set switch1 l3-hw-offloading=no" with live traffic crashed the router. Oct 25: cpss signal=11. Oct 20: cpss and then /nova/bin/net signal=11, two reboots in a row. Is this the "l3hw - improve stability on CRS8xx series switches" fix in beta4? We have autosupout from these crashes, can send to support.

We don't want to post the real production config here. If it helps, we can open a support ticket with the full unmodified config, autosupouts and graphs.

@Tint6603, But wait, you’re actually running 7.26beta3 in full production on a transit box!? Seriously, why? I’m genuinely interested.

Open a ticket, its your best bet to get anything fixed and assessed by actual people who work at mikrotik, remmember the more detailed the report is with steps to reproduce and your exact issues stated, the better.

https://mikrotik.com/support

@Larsa yeah, I know :slight_smile: We needed VXLAN offload on CRS8xx and it only showed up in 7.26beta3, so the decision was made to go with the beta. Not my favorite idea either. Planning to move to stable as soon as it has VXLAN offload.

@drotz will definitely open a ticket with the full config, autosupouts and graphs. Just posted here first since MT devs actively read the beta threads - figured if it's already a known limitation or behaviour on 98DX7335, someone might point it out right away. Plus our previous CRS812 L3HW ticket (SUP-222890, opened Sep 3) still hasn't got a single reply, so the forum feels like a much faster way to get eyes on it.

Gotcha, you don't have to really give them your config when you send them autosupout btw; other than that hope you get a response or a fix for it soon!

Wish you luck :slight_smile:

I get it, but still... a beta on a transit box. Well, you'd better keep that rollback plan handy! Hope stable gets VXLAN offload soon. :smiley:

@drotz yeah, I know supout includes the config, just figured attaching a plain .rsc alongside the autosupouts might save them some time grepping through it (plus the autosupouts are from Oct 20/25, so a current export won't hurt). Thanks!

@Larsa thanks! Trust me, I'm really not happy about running beta on our prod transit box either. Unfortunately the requirement to deliver VXLAN transit came from higher up, so we just had to make it work somehow. Definitely keeping that rollback plan ready.

Opened a ticket with full config, autosupouts and graphs: SUP-227266.

The documentation currently has no implementation examples:

This needs at least a documented configuration example, and some indication of "easymesh" consumer hardware for testing