Community discussions

MikroTik App
 
DrRob
just joined
Topic Author
Posts: 10
Joined: Tue Oct 06, 2015 8:15 pm

packet loss between CPU and switch

Tue Mar 16, 2021 9:23 pm

Hi,

I help to set up and manage the network at our church using Mikrotik devices, with this structure:

Image

Wi-fi routers #1 and #2 (both 951G-2HnD) are located in the church building and are connected to a separate office via a pair of SXT G-5HPacD and a third 951G-2HnD located there. These five Mikrotik devices together share the 192.168.11.0/24 network. Only router #3 has a WAN side, which connects to a LAN port of the office broadband router. The ISP provides an IP address that is shared with other customers so inbound connections aren't possible, so instead I have configured router #3 to establish a L2TP connection to my home router (also a 951G-2HnD), with firewalls set up at both ends to allow only ssh connections. I have ssh client config on my laptop that allows me to ssh to any of the five Mikrotik devices, direct for Router #3, and by tunnelling via #3 to the other four. Routers #1, #2 and #3 all offer addresses via DHCP, all in the same network but with non-overlapping pools. Routers #1, #2 and #3 all offer staff wifi connections in that same network, and guest wifi via hotspots in a different network.

All of this works fine apart from one thing - when I ssh to Router #2, the connection regularly stalls for many seconds and then catches up. If I ping to or from router #2 to/from any of the other devices, it periodically times out, then returns. For example:
[rob@router.north] /ip address> /ping address=192.168.11.21
  SEQ HOST                                     SIZE TTL TIME  STATUS
    0 192.168.11.21                              56  64 4ms
    1 192.168.11.21                              56  64 2ms
    2 192.168.11.21                              56  64 3ms
    3 192.168.11.21                              56  64 2ms
    4 192.168.11.21                              56  64 2ms
    5 192.168.11.21                              56  64 4ms
    6 192.168.11.21                              56  64 2ms
    7 192.168.11.21                              56  64 3ms
    8 192.168.11.21                              56  64 2ms
    9 192.168.11.21                              56  64 2ms
   10 192.168.11.21                              56  64 3ms
   11 192.168.11.21                              56  64 2ms
   12 192.168.11.21                              56  64 3ms
   13 192.168.11.21                              56  64 2ms
   14 192.168.11.21                              56  64 3ms
   15 192.168.11.21                              56  64 2ms
   16 192.168.11.21                              56  64 2ms
   17 192.168.11.21                              56  64 3ms
   18 192.168.11.21                              56  64 2ms
   19 192.168.11.21                              56  64 2ms
    sent=20 received=20 packet-loss=0% min-rtt=2ms avg-rtt=2ms max-rtt=4ms
  SEQ HOST                                     SIZE TTL TIME  STATUS
   20 192.168.11.21                              56  64 4ms
   21 192.168.11.21                              56  64 2ms
   22 192.168.11.21                                           timeout
   23 192.168.11.21                                           timeout
   24 192.168.11.21                                           timeout
   25 192.168.11.21                                           timeout
   26 192.168.11.21                                           timeout
   27 192.168.11.21                                           timeout
   28 192.168.11.21                                           timeout
   29 192.168.11.21                                           timeout
   30 192.168.11.21                              56  64 2ms
   31 192.168.11.21                              56  64 3ms
   32 192.168.11.21                              56  64 3ms
   33 192.168.11.21                              56  64 2ms
   34 192.168.11.21                              56  64 4ms
   35 192.168.11.21                              56  64 3ms
   36 192.168.11.21                              56  64 2ms
   37 192.168.11.21                              56  64 2ms
   38 192.168.11.21                              56  64 3ms
   39 192.168.11.21                              56  64 3ms
    sent=40 received=32 packet-loss=20% min-rtt=2ms avg-rtt=2ms max-rtt=4ms
  SEQ HOST                                     SIZE TTL TIME  STATUS
   40 192.168.11.21                              56  64 4ms
   41 192.168.11.21                              56  64 3ms
   42 192.168.11.21                              56  64 4ms
   43 192.168.11.21                              56  64 3ms
   44 192.168.11.21                              56  64 2ms
   45 192.168.11.21                              56  64 2ms
    sent=46 received=38 packet-loss=17% min-rtt=2ms avg-rtt=2ms max-rtt=4ms
If I ssh to Router #1 (a connection which has to pass through Router #2), it is fine - it does not stall and there is no packet loss. I guess this is because the packets are relayed from the SXT G-5HPacD to Router #1 via the hardware switch of Router #2.

Here is the (redacted) config of Router #2:
[rob@router.church] > /export hide-sensitive compact
# mar/16/2021 19:09:57 by RouterOS 6.47.4
# software id = V5PK-5YIJ
#
# model = 951G-2HnD
# serial number = REDACTED as I'm not sure if it's safe to reveal that?
/interface bridge
add admin-mac=E4:8D:8C:41:A9:72 auto-mac=no fast-forward=no name=bridge-local
/interface wireless
set [ find default-name=wlan1 ] band=2ghz-b/g/n country="united kingdom" disabled=no distance=indoors mode=ap-bridge ssid=All-Saints-Staff station-roaming=\
    enabled wireless-protocol=802.11
/interface ethernet
set [ find default-name=ether1 ] name=ether1-gateway speed=100Mbps
set [ find default-name=ether2 ] name=ether2-master-local speed=100Mbps
set [ find default-name=ether3 ] name=ether3-slave-local speed=100Mbps
set [ find default-name=ether4 ] name=ether4-slave-local speed=100Mbps
set [ find default-name=ether5 ] name=ether5-slave-local speed=100Mbps
/interface wireless security-profiles
set [ find default=yes ] authentication-types=wpa2-psk eap-methods="" mode=dynamic-keys supplicant-identity=MikroTik
add authentication-types=wpa2-psk eap-methods="" management-protection=allowed mode=dynamic-keys name=guest-profile supplicant-identity=""
/interface wireless
add disabled=no keepalive-frames=disabled mac-address=E6:8D:8C:41:A9:76 master-interface=wlan1 multicast-buffering=disabled name=guest-wlan security-profile=\
    guest-profile ssid=All-Saints-Guest station-roaming=enabled wds-cost-range=0 wds-default-cost=0
/ip hotspot profile
add hotspot-address=172.16.1.1 http-cookie-lifetime=2w6d login-by=cookie,http-chap,trial,mac-cookie name=hotspot-profile trial-uptime-limit=3h \
    trial-uptime-reset=3h
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc
/ip pool
add name=lan-pool ranges=192.168.11.176-192.168.11.253
add name=guest-pool ranges=172.16.1.2-172.16.1.254
/ip dhcp-server
add address-pool=lan-pool disabled=no interface=bridge-local lease-time=1h name=default
add address-pool=guest-pool disabled=no interface=guest-wlan lease-time=1h name=dhcp1
/ip hotspot
add address-pool=guest-pool disabled=no interface=guest-wlan name=church-hotspot profile=hotspot-profile
/snmp community
set [ find default=yes ] addresses=0.0.0.0/0
/interface bridge port
add bridge=bridge-local interface=ether2-master-local
add bridge=bridge-local interface=wlan1
add bridge=bridge-local interface=ether1-gateway
add bridge=bridge-local interface=ether3-slave-local
add bridge=bridge-local interface=ether4-slave-local
add bridge=bridge-local interface=ether5-slave-local
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/interface detect-internet
set detect-interface-list=all
/ip address
add address=192.168.11.21/24 comment="default configuration" interface=ether2-master-local network=192.168.11.0
add address=172.16.1.1/24 comment="hotspot network" interface=guest-wlan network=172.16.1.0
/ip dhcp-client
add comment="default configuration" interface=ether1-gateway
/ip dhcp-server lease
add address=192.168.11.236 always-broadcast=yes mac-address=B8:27:EB:72:7E:4D server=default
add address=192.168.11.5 mac-address=F0:DE:F1:7A:35:35 server=default
add address=192.168.11.4 mac-address=A0:88:B4:C0:24:18 server=default
/ip dhcp-server network
add address=172.16.1.0/24 comment="hotspot network" gateway=172.16.1.1
add address=192.168.11.0/24 comment="default configuration" gateway=192.168.11.1 netmask=24 ntp-server=192.168.11.21
/ip dns
set allow-remote-requests=yes servers=192.168.11.1
/ip dns static
add address=192.168.11.21 name=router.church
add address=192.168.11.5 comment="Projection laptop wired" name=slides.church
add address=192.168.11.1 name=router.north
add address=192.168.11.31 name=link.north
add address=192.168.11.41 name=link.church
/ip firewall filter
add action=passthrough chain=unused-hs-chain comment="place hotspot rules here" disabled=yes
add action=fasttrack-connection chain=forward comment="fasttrack established/related forward" connection-state=established,related
add action=accept chain=input comment="accept established/related input" connection-state=established,related
add action=accept chain=forward comment="accept established/related forward" connection-state=established,related
add action=accept chain=forward comment="accept forward to projection pc web" dst-address=192.168.11.5 dst-port=4316 protocol=tcp
add action=drop chain=forward comment="drop forward guest wi-fi to local subnets" dst-address=192.168.0.0/16 in-interface=guest-wlan
add action=drop chain=forward comment="drop invalid forward" connection-state=invalid
add action=drop chain=forward comment="drop new and !dstnat forward" connection-nat-state=!dstnat connection-state=new in-interface=bridge-local
/ip firewall nat
add action=passthrough chain=unused-hs-chain comment="place hotspot rules here" disabled=yes
add action=masquerade chain=srcnat comment="default configuration" disabled=yes out-interface=ether1-gateway
add action=masquerade chain=srcnat comment="masquerade hotspot network" src-address=172.16.1.0/24
/ip route
add distance=1 gateway=192.168.11.1
/ip smb
set allow-guests=no comment=RecordingShare
/ip smb shares
set [ find default=yes ] disabled=yes
add directory=/disk1 max-sessions=50 name=CHURCH
/ip smb users
add name=office read-only=no
add name=recorder read-only=no
/ip ssh
set forwarding-enabled=both
/system clock
set time-zone-name=Europe/London
/system identity
set name=router.church
/system leds
set 0 interface=wlan1
/system ntp client
set enabled=yes primary-ntp=185.53.93.157 secondary-ntp=162.159.200.1
/system ntp server
set enabled=yes manycast=no
/system scheduler
add interval=1d name=set-ntp-ip policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=oct/20/2018 start-time=03:04:05
/system script
add dont-require-permissions=no name=up.8.8.8.8 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put true\" file=\"hotspot/ping/8.8.8.8\""
add dont-require-permissions=no name=up.192.168.11.1 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put true\" file=\"hotspot/ping/192.168.11.1\""
add dont-require-permissions=no name=up.192.168.11.21 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put true\" file=\"hotspot/ping/192.168.11.21\""
add dont-require-permissions=no name=up.192.168.11.31 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put true\" file=\"hotspot/ping/192.168.11.31\""
add dont-require-permissions=no name=up.192.168.11.41 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put true\" file=\"hotspot/ping/192.168.11.41\""
add dont-require-permissions=no name=up.192.168.1.254 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put true\" file=\"hotspot/ping/192.168.1.254\""
add dont-require-permissions=no name=down.8.8.8.8 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put false\" file=\"hotspot/ping/8.8.8.8\""
add dont-require-permissions=no name=down.192.168.11.1 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put false\" file=\"hotspot/ping/192.168.11.1\""
add dont-require-permissions=no name=down.192.168.11.21 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put false\" file=\"hotspot/ping/192.168.11.21\""
add dont-require-permissions=no name=down.192.168.11.31 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put false\" file=\"hotspot/ping/192.168.11.31\""
add dont-require-permissions=no name=down.192.168.11.41 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put false\" file=\"hotspot/ping/192.168.11.41\""
add dont-require-permissions=no name=down.192.168.1.254 owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive source=\
    ":execute script=\":put false\" file=\"hotspot/ping/192.168.1.254\""
add dont-require-permissions=no name=set-ntp-ip owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="{\r\
    \n:local ntpServer \"uk.pool.ntp.org\"\r\
    \n:local primary [resolve \$ntpServer]\r\
    \n:local secondary [resolve \$ntpServer]\r\
    \n\r\
    \n/system ntp client set primary-ntp \$primary\r\
    \n/system ntp client set secondary-ntp \$secondary\r\
    \n}"
/tool graphing interface
add interface=ether2-master-local
/tool graphing resource
add
/tool netwatch
add comment=8.8.8.8 disabled=yes down-script=down.8.8.8.8 host=8.8.8.8 interval=10s up-script=up.8.8.8.8
add comment=192.168.11.1 disabled=yes down-script=down.192.168.11.1 host=192.168.11.1 interval=10s up-script=up.192.168.11.1
add comment=192.168.11.21 disabled=yes down-script=down.192.168.11.21 host=192.168.11.21 interval=10s up-script=up.192.168.11.21
add comment=192.168.11.31 disabled=yes down-script=down.192.168.11.31 host=192.168.11.31 interval=10s up-script=up.192.168.11.31
add comment=192.168.11.41 disabled=yes down-script=down.192.168.11.41 host=192.168.11.41 interval=10s up-script=up.192.168.11.41
add comment=192.168.1.254 disabled=yes down-script=down.192.168.1.254 host=192.168.1.254 interval=10s up-script=up.192.168.1.254
/tool sniffer
set filter-interface=all filter-port=smb streaming-server=192.168.11.2
The config for the other devices is similar except for Router #3 which has a WAN side, NAT, L2TP, etc.

What could be causing this packet loss? Misconfiguration or hardware fault?

Thanks,
Rob.
 
DrRob
just joined
Topic Author
Posts: 10
Joined: Tue Oct 06, 2015 8:15 pm

Re: packet loss between CPU and switch

Sun Mar 21, 2021 12:42 pm

A bit more info - there's a regular pattern to the packet loss:
  • ping requests arriving at router #2 from router #3 or either of the SXT devices, arriving via the neighbouring SXT device, work for 22 seconds, then fail for 8 seconds, then repeat
  • ping requests arriving at router #2 from router #1 work all the time
Perhaps the problem only affects the port of router #2 that the SXT device is connected to. When I can get on site I'll try moving the SXT connection to a different port on router #2.

Would could trigger this pattern of failure that repeats every 30 seconds? Seems unlikely to me that a hardware fault would repeat with this sort of regularity. Seems much more likely to be triggered by something that happens on a timer in the software...?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: packet loss between CPU and switch  [SOLVED]

Sun Mar 21, 2021 8:17 pm

It should help to set detect-interface-list=none under /interface detect-internet on all routers.
 
DrRob
just joined
Topic Author
Posts: 10
Joined: Tue Oct 06, 2015 8:15 pm

Re: packet loss between CPU and switch

Sun Mar 21, 2021 9:45 pm

It should help to set detect-interface-list=none under /interface detect-internet on all routers.
Thank you! I did that on the router with the problem and the periodic packet loss has stopped! (I haven't applied the same setting to the other devices yet but will do.)

It would be good to understand exactly what that function does and why it caused the problem?

Thanks,
Rob.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: packet loss between CPU and switch

Sun Mar 21, 2021 9:49 pm

It would be good to understand exactly what that function does and why it caused the problem?
I agree it would be good.

Unfortunately, no one but Mikrotik knows that, and they didn't bother to explain in detail. Worse than that, I haven't heard a single person so far to say that they've found that function useful, yet it comes enabled by default.

Who is online

Users browsing this forum: ekinsl, ips, mkx and 83 guests