Varying ICMP echo RTT to cAP AX

pjck · Fri Dec 22, 2023 1:19 am

Hi.

I just bought two cAP AX. I configured one of the and enabled wifi for 5 GHz AX and AC, and 2.4 GHz AX, N and G. I can easily get a couple of 100 Mbps Internet traffic when I use Speedtest or similar, but I see weird ICMP echo RTT on my local network.

When I ping the AP from my laptop, I get RTT's varying from 4 ms all the way up to 200+ ms. Same goes for ping to my router on the other end of the AP's wired connection, and to my ISP respectively.

If I login via SSH on the AP and ping the router over the wire, I get RTT's of around 300 us or so. Very consistent RTT's for my ISP's next hop too. The varying RTT's is only seen on wifi, so I assume that's where the latency happens.

I have disabled NAT, DHCP client, etc. Ethernet interfaces and wifi interfaces are bridged. STP is disabled on the bridge.
I have tested different combinations of channels and channel width, different tx power and antenna gain settings, etc.

Is this normal behaviour? It feels odd to me, but I also know that ICMP traffic gets lower prio sometimes, so I've learned not to rely on it for troubleshooting purposes.

The AP came with RouterOS 7.8 but the behaviour is still there in 7.12.

/interface bridge
add admin-mac=78:9A:18:xx:xx:Xx auto-mac=no comment=defconf name=bridge \
    protocol-mode=none
/interface wifiwave2
set [ find default-name=wifi1 ] channel.band=5ghz-ax .skip-dfs-channels=all \
    .width=20/40/80mhz configuration.country=Sweden .mode=ap .ssid=yeah \
    datapath.bridge=bridge disabled=no security.authentication-types=\
    wpa2-psk,wpa3-psk .connect-priority=0
set [ find default-name=wifi2 ] channel.band=2ghz-ax .skip-dfs-channels=all \
    .width=20/40mhz configuration.country=Sweden .mode=ap .ssid=yeah \
    datapath.bridge=bridge disabled=no security.authentication-types=\
    wpa2-psk,wpa3-psk .connect-priority=0
add channel.band=5ghz-ac .skip-dfs-channels=all .width=20/40/80mhz \
    configuration.mode=ap .ssid=yeah datapath.bridge=bridge disabled=no \
    mac-address=7A:9A:18:xx:xx:xx master-interface=wifi1 name=wifi3 \
    security.authentication-types=wpa2-psk,wpa3-psk
add channel.band=2ghz-n .skip-dfs-channels=all .width=20/40mhz \
    configuration.mode=ap .ssid=yeah datapath.bridge=bridge disabled=no \
    mac-address=7A:9A:18:xx:xx:xx master-interface=wifi2 name=wifi4 \
    security.authentication-types=wpa2-psk,wpa3-psk
add channel.band=2ghz-g .skip-dfs-channels=all .width=20mhz \
    configuration.mode=ap .ssid=yeah datapath.bridge=bridge disabled=no \
    mac-address=7A:9A:18:xx:xx:xx master-interface=wifi2 name=wifi5 \
    security.authentication-types=wpa2-psk,wpa3-psk
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/ip address
add address=192.168.88.2/24 interface=bridge network=192.168.88.0
/ip dhcp-client
add comment=defconf disabled=yes interface=ether1
/ip dns
set servers=192.168.88.1
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" disabled=yes \
    ipsec-policy=out,none out-interface-list=WAN
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.88.1 \
    pref-src="" routing-table=main suppress-hw-offload=no
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=\
    33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/system clock
set time-zone-name=Europe/Stockholm
/system note
set show-at-login=no
/system ntp client
set enabled=yes
/system ntp client servers
add address=mmo1.ntp.se
add address=mmo2.ntp.se
/system routerboard mode-button
set enabled=yes on-event=dark-mode
/system script
add comment=defconf dont-require-permissions=no name=dark-mode owner=*sys \
    policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    source="\r\
    \n   :if ([system leds settings get all-leds-off] = \"never\") do={\r\
    \n     /system leds settings set all-leds-off=immediate \r\
    \n   } else={\r\
    \n     /system leds settings set all-leds-off=never \r\
    \n   }\r\
    \n "
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

mkx · Fri Dec 22, 2023 8:39 am

Check the channel occupancy ... if any other AP is seen on same channel (or even adjacent channels on 2.4GHz, there channels partly overlap), then activity on the other wireless network does affect operations on your own wireless network. If the other wifi AP operates on same channel, then activity on the other channel is detected by your AP (and stations) and your radios pause for a while to see clear spectrum (and your activity affects the other wireless network equally). If the other AP operates on adjacent 2.4GHz channel[*], then your wireless devices will not detect activity, rather their transmissions will be heavily interfered (requiring retransmissions over air which increases latency).

[*] In 2.4GHz spectrum, channel spacing is 5MHz while channel width is 20MHz. Which means that 3 channels higher and 3 channels lower than operating channel are interfering each other. E.g. if AP operates on channel 6, then channels 3, 4, 5, 7, 8, and 9 at least partially overlap the 20MHz frequency space of channel 6 and thus cause (higher or lower) interference. Hence the "cookbook recipe" of only using channels 1-6-11 (or 1-5-9-13, depending on the part of the globe). Some device vendors have built the recipe into their frequency selection algorithms, MT doesn't seem to be doing it.

pjck · Fri Dec 22, 2023 11:17 am

Thanks, mkx.

I spent quite a lot of time analyzing the spectrum, and I can only see two other AP's/networks on 5 GHz. I chose a channel that appeared to be empty, but I'm not sure if that channel had any AP's on adjacent channels. Will try to get a bit more spacing.

I am indoors in a newly built house with drywall, and 4-5 meters from the AP I get a RSSI of around -65 dBm on 802.11ax. Isn't that quite low for that range? Not sure if that can be a factor, but could possibly be an indicator that something is wrong, either with the hardware or with my configuration.

The behaviour is present both on 2.4 and 5 GHz, so I'm going to focus on getting it solved on 5 GHz primarily.

mkx · Fri Dec 22, 2023 11:32 am

IMO signal strength of -65dBm 4 metres away from AP is expectable. And according to cAP ax specs it should (just) suffice to get maximum speed. If there's no interference.

The problem with wireless network scanners (available on phones and APs) is that it only shows 802.11 use of particular frequency channel. Other uses are not reported, but still cause interference. 2.4GHz band is particularly infested with other uses (ZigBee, microwave owens, etc.), 5GHz band should be much better in this regard ... and higher loss of signal (both due to higher free air attenuation as well as attenuation when passing any obstacles) actually helps in this regard (because interference will be attenuated more as well).

There's another factor which does affect wireless performance: most devices put their radios to sleep after some inactivity period to conserve power. It then takes a while to wake up radio and that causes high delay jitter. Which is more prominent if traffic source (e.g. ping sender) is on the AP side. If sender is wireless device itself, then it can wake up radio as soon as Tx buffer gets anything in it (but some devices may be slow even with that). But if sender is AP, then it simply has to wait for station to wake up (it does so periodically) and that can take longer time. It might be possible to play with power saving options of wireless card on your laptop and see if things get any better.

Varying ICMP echo RTT to cAP AX

Varying ICMP echo RTT to cAP AX

Re: Varying ICMP echo RTT to cAP AX

Re: Varying ICMP echo RTT to cAP AX

Re: Varying ICMP echo RTT to cAP AX

Who is online