CRS812-8DS-2DQ-2DDQ, 7.26beta3, l3-hw-offloading=yes.
Transit box. Default route via BGP from upstream on qsfp56-1-1, ~35 Gbit/s / ~5 Mpps at evening peak. One /24 directly connected on a VLAN for customer VMs (~180 entries in ARP), servers and downstream customers on sfp56 ports.
We had floods on Oct 20 and Oct 26, the one on Oct 26 was UDP to that /24. Upstream port went to 80-96 Gbit/s, ~10 Mpps. The switch chip handled it, egress didn't grow, the extra traffic was dropped in hardware. But CPU went to 98-99%, on Oct 26 almost all of it IRQ. REST stops answering and our monitoring loses the box. On Oct 20 this happened twice, ~35 min each, and the box rebooted in between.
From our graphs (switch1-cpu counters + cpu load over REST): 600-700 kpps to the CPU already gives 75-80% load. On Oct 26 tx-drop-queue0-packet on switch1-cpu went from ~5k to ~218M in one hour. So even a small part of a flood reaching the CPU kills the control plane.
No supout from the attacks, we were busy keeping things running.
First we thought packets to unused IPs in the /24 (no ARP entry) get punted to CPU for ARP resolution. So we made a script: it reads ARP every second and keeps a /32 permit rule per live IP, then a /24 drop, on the upstream port:
/interface ethernet switch rule
add switch=switch1 ports=qsfp56-1-1 mac-protocol=ip dst-address=198.51.100.10/32 comment="USED 198.51.100.10"
add switch=switch1 ports=qsfp56-1-1 mac-protocol=ip dst-address=198.51.100.11/32 comment="USED 198.51.100.11"
(... ~230 rules like this)
add switch=switch1 ports=qsfp56-1-1 mac-protocol=ip dst-address=198.51.100.0/24 new-dst-ports="" comment="UNUSED drop"
Then we tested with UDP flood to the whole /24, about 5 Gbit/s / 500 kpps. Routes to our DNS servers were withdrawn for the test, they are behind a GRE tunnel which goes through CPU anyway. CPU went up ~5% in both cases: without the script from ~10-11% idle, with the script from ~16-17% to ~22%. No difference. (Idle ~10% is mostly our own monitoring polling REST every second.)
What does work: one switch rule on the upstream port that drops all UDP to the /24 (protocol=udp dst-address=198.51.100.0/24 new-dst-ports=""). We put it first so it hits earlier, but it also worked lower in the list. With it, CPU load went back to normal right away. On Oct 26 the 98-99% was before we added it. The flood peaked at ~96 Gbit/s / ~9.5 Mpps after the rule was in place, and CPU stayed at 10-15%, only a couple of short spikes to 50-80%. Of course it also cuts all legit UDP, so it's only for emergencies.
So drop by dst /24 + protocol works, but /32 permits for live IPs + /24 drop for the rest doesn't change anything. Looks like the CPU load comes from flood traffic to live IPs, the ones the allowlist lets through, not from unused IPs. Maybe it's related to question 5 below, but we don't understand how. The script itself also costs ~6% CPU, ~350 ACL entries, new VMs are not reachable until they appear in ARP, and there is nothing for IPv6.
Simplified config, IPs and VLANs changed:
/interface ethernet switch
set switch1 l3-hw-offloading=yes qos-hw-offloading=yes
/interface ethernet switch l3hw-settings
set icmp-reply-on-error=no ipv6-hw=yes
/interface vlan
add interface=bridge name=vlan900-transit vlan-id=900
add interface=bridge name=vlan300-vms vlan-id=300
/ip address
add address=203.0.113.1/31 interface=vlan900-transit
add address=198.51.100.1/24 interface=vlan300-vms
/routing bgp connection
add name=UPSTREAM-V4 remote.address=203.0.113.0 local.role=ebgp-customer input.accept-nlri=default-only ...
/interface ethernet switch rule
add switch=switch1 ports=qsfp56-1-1 mac-protocol=arp comment="arp permit"
add switch=switch1 ports=qsfp56-1-1 src-address=198.51.100.0/24 new-dst-ports="" comment="antispoof"
add switch=switch1 ports=qsfp56-1-1 dst-address=203.0.113.1/32 protocol=tcp dst-port=179 src-address=203.0.113.0/32 comment="bgp"
add switch=switch1 ports=qsfp56-1-1 dst-address=203.0.113.1/32 protocol=icmp rate=1Mbps comment="icmp to router"
add switch=switch1 ports=qsfp56-1-1 dst-address=198.51.100.1/32 protocol=icmp rate=1Mbps comment="icmp to gw"
(+ USED/UNUSED rules from above)
Questions:
-
What exactly goes to CPU when UDP flood hits a directly connected /24? Dropping all UDP to the /24 on ingress fixes it, the allowlist doesn't. So is traffic to live, resolved hosts being punted too? Or does the CPU load come from something else (ARP/FDB updates, neighbor discovery)?
-
In beta4 there is "switch - add a high-priority CPU queue for control traffic on Marvell Prestera switches". What goes to this queue, BGP/ARP/ICMP to router? Will it keep control plane alive when queue 0 is full of punted packets, or it's still the same IRQ/RX path?
-
If it is ARP-miss, is there a plan for an option to not send unresolved hosts to CPU and just drop in hardware? Router can still resolve ARP with its own requests. Or at least a rate limit for this trap, like glean throttling on Cisco/Arista. If it's not ARP-miss, what is it and will it be fixed? And in general, per trap reason rate limit to CPU, is it planned?
-
Everything to CPU goes to queue 0 on switch1-cpu, queues 1-7 are always 0. So control traffic and flood are in the same queue and dropped together. tx-drop counters show how much, but not why packet went to CPU (ARP miss, to-router, TTL, ACL copy-to-cpu). Any plans for per-reason counters?
-
l3hw advanced monitor shows ipv4-hosts ~34, but ARP on that VLAN has ~180 reachable/stale entries. Is it normal or most hosts are not offloaded? If they are not, does traffic to them go through CPU? That would explain why the allowlist doesn't help.
-
Different issue, same box. Two times (Oct 20 and Oct 25) "/interface ethernet switch set switch1 l3-hw-offloading=no" with live traffic crashed the router. Oct 25: cpss signal=11. Oct 20: cpss and then /nova/bin/net signal=11, two reboots in a row. Is this the "l3hw - improve stability on CRS8xx series switches" fix in beta4? We have autosupout from these crashes, can send to support.
We don't want to post the real production config here. If it helps, we can open a support ticket with the full unmodified config, autosupouts and graphs.