Help troubleshooting latency crash on RB4011

dchardy · December 12, 2022, 4:56pm

[Edit: the RB4011 is not doing routing, just bridging]

Greetings,

We are a small ISP (~500 clients) and we are testing a backup upstream connection to the Internet, as follows:
Clients - openBSD router (NAT) - AirFiber bridge - RB4011 bridge (with queuing) - Internet gw (1Gbps fiber)

The RB4011 has queues set up such that we get 250Mbps and can utilize the entire 1Gbps if not used by other higher priority queues.

The connection works fine for hours or days passing up to 500Mbps, then suddenly we are seeing a huge spike in latency to 1000-2000ms between our router and the RB4011, and loss of connection to the internet. This doesn’t seem to be related to amount of traffic - it has happened while passing 200Mbps, and 500Mbps. It does not resolve itself unless all client traffic is stopped, and the issue returns immediately with clients traffic is resumed, unless stopped for 30-60 minutes, which seems to reset things.

Observations during the “episode”:
-pings from our router to either of the Airfiber bridge devices are unaffected.
-pings from our router to the RB4011, or another host on LAN side of the the RB4011 (same subnet) are unaffected.
-CPU utilization on the RB4011 is max 28% with traffic, and memory usage is minimal.

My question is what to look at on the RB4011 to determine where things are going south. My feeling is that something is going on with the queue tree set up on the RB4011, or something else is getting overloaded (connection tracking/state table?). There is very little other usage by other clients on the LAN side of the RB4011, but it’s possible that something happening there is triggering this.

If anyone has any suggestions regarding how to troubleshoot this and where to look, I would greatly appreciate it.

David

pcunite · December 20, 2022, 3:02pm

What firmware and show an export between CODE tags here.

dchardy · December 23, 2022, 3:36pm

v6.48.6 (long-term)
We upgraded to a CCR1009 with the same config, and still see the issue. We don't see the issue when bypassing the Mikrotik and going straight into the upstream router via a switch (the 1009 is actually not doing routing as I originally stated, just bridging). We disabled RSTP, which enabled HW offloading (thus disabling fw/queueing) , and still saw the issue. Thanks!

# dec/23/2022 08:24:52 by RouterOS 6.48.6
# software id = E9P3-3ZK1
#
# model = CCR1009-7G-1C-1S+
# serial number = HD10823VJY3
/interface bridge
add ageing-time=5m arp=enabled arp-timeout=auto auto-mac=yes dhcp-snooping=no disabled=no fast-forward=yes forward-delay=15s igmp-snooping=no max-message-age=20s mtu=auto name=bridge1 priority=0x8000 protocol-mode=rstp transmit-hold-count=6 vlan-filtering=no
/interface ethernet
set [ find default-name=combo1 ] advertise=10M-full,100M-full,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited combo-mode=auto disabled=no full-duplex=yes l2mtu=1580 loop-protect=default loop-protect-disable-time=5m \
    loop-protect-send-interval=5s mac-address=18:FD:74:89:18:8D mtu=1500 name=combo1 orig-mac-address=18:FD:74:89:18:8D rx-flow-control=off sfp-rate-select=high sfp-shutdown-temperature=95C speed=1Gbps tx-flow-control=off
set [ find default-name=ether1 ] advertise=10M-full,100M-full,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1580 loop-protect=default loop-protect-disable-time=5m \
    loop-protect-send-interval=5s mac-address=18:FD:74:89:18:8E mtu=1500 name=ether1 orig-mac-address=18:FD:74:89:18:8E rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether2 ] advertise=10M-full,100M-full,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1580 loop-protect=default loop-protect-disable-time=5m \
    loop-protect-send-interval=5s mac-address=18:FD:74:89:18:8F mtu=1500 name=ether2 orig-mac-address=18:FD:74:89:18:8F rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether3 ] advertise=10M-full,100M-full,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1580 loop-protect=default loop-protect-disable-time=5m \
    loop-protect-send-interval=5s mac-address=18:FD:74:89:18:90 mtu=1500 name=ether3 orig-mac-address=18:FD:74:89:18:90 rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether4 ] advertise=10M-full,100M-full,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1580 loop-protect=default loop-protect-disable-time=5m \
    loop-protect-send-interval=5s mac-address=18:FD:74:89:18:91 mtu=1500 name=ether4 orig-mac-address=18:FD:74:89:18:91 rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether5 ] advertise=10M-full,100M-full,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1580 loop-protect=default loop-protect-disable-time=5m \
    loop-protect-send-interval=5s mac-address=18:FD:74:89:18:92 mtu=1500 name=ether5 orig-mac-address=18:FD:74:89:18:92 rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether6 ] advertise=10M-full,100M-full,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1580 loop-protect=default loop-protect-disable-time=5m \
    loop-protect-send-interval=5s mac-address=18:FD:74:89:18:93 mtu=1500 name=ether6 orig-mac-address=18:FD:74:89:18:93 rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=ether7 ] advertise=10M-full,100M-full,1000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1580 loop-protect=default loop-protect-disable-time=5m \
    loop-protect-send-interval=5s mac-address=18:FD:74:89:18:94 mtu=1500 name=ether7 orig-mac-address=18:FD:74:89:18:94 rx-flow-control=off speed=1Gbps tx-flow-control=off
set [ find default-name=sfp-sfpplus1 ] advertise=1000M-full,10000M-full arp=enabled arp-timeout=auto auto-negotiation=yes bandwidth=unlimited/unlimited disabled=no full-duplex=yes l2mtu=1580 loop-protect=default loop-protect-disable-time=5m \
    loop-protect-send-interval=5s mac-address=18:FD:74:89:18:8C mtu=1500 name=sfp-sfpplus1 orig-mac-address=18:FD:74:89:18:8C rx-flow-control=off sfp-rate-select=high sfp-shutdown-temperature=95C speed=10Gbps tx-flow-control=off
/queue interface
set bridge1 queue=no-queue
/interface list
set [ find name=all ] comment="contains all interfaces" exclude="" include="" name=all
set [ find name=none ] comment="contains no interfaces" exclude="" include="" name=none
set [ find name=dynamic ] comment="contains dynamic interfaces" exclude="" include="" name=dynamic
set [ find name=static ] comment="contains static interfaces" exclude="" include="" name=static
add exclude="" include="" name=LAN
/interface lte apn
set [ find default=yes ] add-default-route=yes apn=internet default-route-distance=2 name=default use-peer-dns=yes
/interface wireless security-profiles
set [ find default=yes ] authentication-types="" disable-pmkid=no eap-methods=passthrough group-ciphers=aes-ccm group-key-update=5m interim-update=0s management-protection=disabled mode=none mschapv2-username="" name=default radius-called-format=mac:ssid \
    radius-eap-accounting=no radius-mac-accounting=no radius-mac-authentication=no radius-mac-caching=disabled radius-mac-format=XX:XX:XX:XX:XX:XX radius-mac-mode=as-username static-algo-0=none static-algo-1=none static-algo-2=none

static-algo-3=none
static-sta-private-algo=none static-transmit-key=key-0 supplicant-identity=MikroTik tls-certificate=none tls-mode=no-certificates unicast-ciphers=aes-ccm
/ip dhcp-client option
set clientid_duid code=61 name=clientid_duid value="0xff$(CLIENT_DUID)"
set clientid code=61 name=clientid value="0x01$(CLIENT_MAC)"
set hostname code=12 name=hostname value="$(HOSTNAME)"
/ip hotspot profile
set [ find default=yes ] dns-name="" hotspot-address=0.0.0.0 html-directory=hotspot html-directory-override="" http-cookie-lifetime=3d http-proxy=0.0.0.0:0 login-by=cookie,http-chap name=default rate-limit="" smtp-server=0.0.0.0 split-user-domain=no use-radius=no
/ip hotspot user profile
set [ find default=yes ] add-mac-cookie=yes address-list="" idle-timeout=none !insert-queue-before keepalive-timeout=2m mac-cookie-timeout=3d name=default !parent-queue !queue-type shared-users=1 status-autorefresh=1m transparent-proxy=no
/ip ipsec mode-config
set [ find default=yes ] name=request-only responder=no use-responder-dns=exclusively
/ip ipsec policy group
set [ find default=yes ] name=default
/ip ipsec profile
set [ find default=yes ] dh-group=modp2048,modp1024 dpd-interval=2m dpd-maximum-failures=5 enc-algorithm=aes-128,3des hash-algorithm=sha1 lifetime=1d name=default nat-traversal=yes proposal-check=obey
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha1 disabled=no enc-algorithms=aes-256-cbc,aes-192-cbc,aes-128-cbc lifetime=30m name=default pfs-group=modp1024
/port
set 0 baud-rate=auto data-bits=8 flow-control=none name=serial0 parity=none stop-bits=1
set 1 baud-rate=auto data-bits=8 flow-control=none name=serial1 parity=none stop-bits=1
/ppp profile
set *0 address-list="" !bridge !bridge-horizon bridge-learning=default !bridge-path-cost !bridge-port-priority change-tcp-mss=yes !dns-server !idle-timeout !incoming-filter !insert-queue-before !interface-list !local-address name=default on-down="" on-up=""
only-one=default !outgoing-filter !parent-queue !queue-type !rate-limit !remote-address !session-timeout use-compression=default use-encryption=default use-mpls=default use-upnp=default !wins-server
set *FFFFFFFE address-list="" !bridge !bridge-horizon bridge-learning=default !bridge-path-cost !bridge-port-priority change-tcp-mss=yes !dns-server !idle-timeout !incoming-filter !insert-queue-before !interface-list !local-address name=default-encryption
on-down="" on-up="" only-one=default !outgoing-filter !parent-queue !queue-type !rate-limit !remote-address !session-timeout use-compression=default use-encryption=yes use-mpls=default use-upnp=default !wins-server
/queue type
set 0 kind=pfifo name=default pfifo-limit=50
set 1 kind=pfifo name=ethernet-default pfifo-limit=50
set 2 kind=sfq name=wireless-default sfq-allot=1514 sfq-perturb=5
set 3 kind=red name=synchronous-default red-avg-packet=1000 red-burst=20 red-limit=60 red-max-threshold=50 red-min-threshold=10
set 4 kind=sfq name=hotspot-default sfq-allot=1514 sfq-perturb=5
set 5 kind=pcq name=pcq-upload-default pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-classifier=src-address pcq-dst-address-mask=32 pcq-dst-address6-mask=128 pcq-limit=50KiB pcq-rate=0 pcq-src-address-mask=32 pcq-src-address6-mask=128
pcq-total-limit=6000KiB
set 6 kind=pcq name=pcq-download-default pcq-burst-rate=0 pcq-burst-threshold=0 pcq-burst-time=10s pcq-classifier=dst-address pcq-dst-address-mask=32 pcq-dst-address6-mask=128 pcq-limit=50KiB pcq-rate=0 pcq-src-address-mask=32 pcq-src-address6-mask=128
pcq-total-limit=6000KiB
set 7 kind=none name=only-hardware-queue
set 8 kind=mq-pfifo mq-pfifo-limit=50 name=multi-queue-ethernet-default
set 9 kind=pfifo name=default-small pfifo-limit=10
/queue interface
set combo1 queue=only-hardware-queue
set ether1 queue=only-hardware-queue
set ether2 queue=only-hardware-queue
set ether3 queue=only-hardware-queue
set ether4 queue=only-hardware-queue
set ether5 queue=only-hardware-queue
set ether6 queue=only-hardware-queue
set ether7 queue=only-hardware-queue
set sfp-sfpplus1 queue=only-hardware-queue
/queue tree
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=0 name="Global Queue" packet-mark="" parent=global priority=8 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=0 name=Download packet-mark="" parent="Global Queue" priority=2 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=0 max-limit=0 name=Upload packet-mark="" parent="Global Queue" priority=3 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=250M max-limit=850M name="Nednet Upload" packet-mark=NednetUpload parent=Upload priority=8 queue=pcq-upload-default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=250M max-limit=850M name="Nednet Download" packet-mark=NednetDownload parent=Download priority=8 queue=pcq-download-default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=200M max-limit=1G name="OC Upload" packet-mark=OCUpload parent=Upload priority=8 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=200M max-limit=1G name="OC Download" packet-mark=OCDownload parent=Download priority=8 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=350M max-limit=1G name="Hub Upload" packet-mark=HubUpload parent=Upload priority=8 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=350M max-limit=1G name="Hub Download" packet-mark=HubDownload parent=Download priority=8 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=100M max-limit=1G name="Tenant Upload" packet-mark=TenantUpload parent=Upload priority=8 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=100M max-limit=1G name="Tenant Download" packet-mark=TenantDownload parent=Download priority=8 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=100M max-limit=1G name="AS Upload" packet-mark=ASUpload parent=Upload priority=8 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=no limit-at=100M max-limit=1G name="AS Download" packet-mark=ASDownload parent=Download priority=8 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=yes limit-at=20M max-limit=1G name="WB Upload" packet-mark=WBUpload parent=Upload priority=8 queue=default
add bucket-size=0.1 burst-limit=0 burst-threshold=0 burst-time=0s disabled=yes limit-at=20M max-limit=1G name="WB Download" packet-mark=WBDownload parent=Download priority=8 queue=default
/routing bgp instance
set default as=65530 client-to-client-reflection=yes !cluster-id !confederation disabled=no ignore-as-path-len=no name=default out-filter="" redistribute-connected=no redistribute-ospf=no redistribute-other-bgp=no redistribute-rip=no redistribute-static=no
router-id=0.0.0.0 routing-table=""
/routing ospf instance
set [ find default=yes ] disabled=no distribute-default=never !domain-id !domain-tag in-filter=ospf-in metric-bgp=auto metric-connected=20 metric-default=1 metric-other-ospf=auto metric-rip=20 metric-static=20 !mpls-te-area !mpls-te-router-id name=default
out-filter=ospf-out redistribute-bgp=no redistribute-connected=no redistribute-other-ospf=no redistribute-rip=no redistribute-static=no router-id=0.0.0.0 !routing-table !use-dn
/routing ospf area
set [ find default=yes ] area-id=0.0.0.0 disabled=no instance=default name=backbone type=default
/snmp community
set [ find default=yes ] addresses=::/0 authentication-protocol=MD5 disabled=no encryption-protocol=DES name=public read-access=yes security=none write-access=no
/system logging action
set 0 memory-lines=1000 memory-stop-on-full=no name=memory target=memory
set 1 disk-file-count=2 disk-file-name=log disk-lines-per-file=1000 disk-stop-on-full=no name=disk target=disk
set 2 name=echo remember=yes target=echo
set 3 bsd-syslog=no name=remote remote=0.0.0.0 remote-port=514 src-address=0.0.0.0 syslog-facility=daemon syslog-severity=auto syslog-time-format=bsd-syslog target=remote
/user group
set read name=read policy=local,telnet,ssh,reboot,read,test,winbox,password,web,sniff,sensitive,api,romon,tikapp,!ftp,!write,!policy,!dude skin=default
set write name=write policy=local,telnet,ssh,reboot,read,write,test,winbox,password,web,sniff,sensitive,api,romon,tikapp,!ftp,!policy,!dude skin=default
set full name=full policy=local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,tikapp,!dude skin=default
/caps-man aaa
set called-format=mac:ssid interim-update=disabled mac-caching=disabled mac-format=XX:XX:XX:XX:XX:XX mac-mode=as-username
/caps-man manager
set ca-certificate=none certificate=none enabled=no package-path="" require-peer-certificate=no upgrade-policy=none
/caps-man manager interface
set [ find default=yes ] disabled=no forbid=no interface=all
/certificate settings
set crl-download=no crl-store=ram crl-use=no
/interface bridge port
add auto-isolate=no bpdu-guard=no bridge=bridge1 broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=ether2 internal-path-cost=10 learn=auto multicast-router=temporary-query path-cost=10
point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes unknown-unicast-flood=yes
add auto-isolate=no bpdu-guard=no bridge=bridge1 broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=ether3 internal-path-cost=10 learn=auto multicast-router=temporary-query path-cost=10
point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes unknown-unicast-flood=yes
add auto-isolate=no bpdu-guard=no bridge=bridge1 broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=ether4 internal-path-cost=10 learn=auto multicast-router=temporary-query path-cost=10
point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes unknown-unicast-flood=yes
add auto-isolate=no bpdu-guard=no bridge=bridge1 broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=ether5 internal-path-cost=10 learn=auto multicast-router=temporary-query path-cost=10
point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes unknown-unicast-flood=yes
add auto-isolate=no bpdu-guard=no bridge=bridge1 broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=ether6 internal-path-cost=10 learn=auto multicast-router=temporary-query path-cost=10
point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes unknown-unicast-flood=yes
add auto-isolate=no bpdu-guard=no bridge=bridge1 broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=ether7 internal-path-cost=10 learn=auto multicast-router=temporary-query path-cost=10
point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes unknown-unicast-flood=yes
add auto-isolate=no bpdu-guard=no bridge=bridge1 broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=sfp-sfpplus1 internal-path-cost=10 learn=auto multicast-router=temporary-query
path-cost=10 point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes unknown-unicast-flood=yes
add auto-isolate=no bpdu-guard=no bridge=bridge1 broadcast-flood=yes disabled=no edge=auto fast-leave=no frame-types=admit-all horizon=none hw=yes ingress-filtering=no interface=ether1 internal-path-cost=10 learn=auto multicast-router=temporary-query path-cost=10
point-to-point=auto priority=0x80 pvid=1 restricted-role=no restricted-tcn=no tag-stacking=no trusted=no unknown-multicast-flood=yes unknown-unicast-flood=yes
/interface bridge port-controller

disabled

set bridge=none cascade-ports="" switch=none
/interface bridge port-extender

disabled

set control-ports="" excluded-ports="" switch=none
/interface bridge settings
set allow-fast-path=yes use-ip-firewall=yes use-ip-firewall-for-pppoe=no use-ip-firewall-for-vlan=no
/ip firewall connection tracking
set enabled=auto generic-timeout=10m icmp-timeout=10s loose-tcp-tracking=yes tcp-close-timeout=10s tcp-close-wait-timeout=10s tcp-established-timeout=1d tcp-fin-wait-timeout=10s tcp-last-ack-timeout=10s tcp-max-retrans-timeout=5m tcp-syn-received-timeout=5s
tcp-syn-sent-timeout=5s tcp-time-wait-timeout=10s tcp-unacked-timeout=5m udp-stream-timeout=3m udp-timeout=10s
/ip neighbor discovery-settings
set discover-interface-list=LAN lldp-med-net-policy-vlan=disabled protocol=cdp,lldp,mndp
/ip settings
set accept-redirects=no accept-source-route=no allow-fast-path=yes arp-timeout=30s icmp-rate-limit=10 icmp-rate-mask=0x1818 ip-forward=yes max-neighbor-entries=8192 route-cache=yes rp-filter=no secure-redirects=yes send-redirects=yes tcp-syncookies=no
/interface detect-internet
set detect-interface-list=none internet-interface-list=none lan-interface-list=none wan-interface-list=none
/interface l2tp-server server
set allow-fast-path=no authentication=pap,chap,mschap1,mschap2 caller-id-type=ip-address default-profile=default-encryption enabled=no keepalive-timeout=30 max-mru=1450 max-mtu=1450 max-sessions=unlimited mrru=disabled one-session-per-host=no use-ipsec=no
/interface list member
add disabled=no interface=bridge1 list=LAN
/interface ovpn-server server
set auth=sha1,md5 cipher=blowfish128,aes128 default-profile=default enabled=no keepalive-timeout=60 mac-address=FE:85:D3:87:55:86 max-mtu=1500 mode=ip netmask=24 port=1194 require-client-certificate=no
/interface pptp-server server
set authentication=mschap1,mschap2 default-profile=default-encryption enabled=no keepalive-timeout=30 max-mru=1450 max-mtu=1450 mrru=disabled
/interface sstp-server server
set authentication=pap,chap,mschap1,mschap2 certificate=none default-profile=default enabled=no force-aes=no keepalive-timeout=60 max-mru=1500 max-mtu=1500 mrru=disabled pfs=no port=443 tls-version=any verify-client-certificate=no
/interface wireless align
set active-mode=yes audio-max=-20 audio-min=-100 audio-monitor=00:00:00:00:00:00 filter-mac=00:00:00:00:00:00 frame-size=300 frames-per-second=25 receive-all=no ssid-all=no
/interface wireless cap
set bridge=none caps-man-addresses="" caps-man-certificate-common-names="" caps-man-names="" certificate=none discovery-interfaces="" enabled=no interfaces="" lock-to-caps-man=no static-virtual=no
/interface wireless sniffer
set channel-time=200ms file-limit=10 file-name="" memory-limit=10 multiple-channels=no only-headers=no receive-errors=no streaming-enabled=no streaming-max-rate=0 streaming-server=0.0.0.0
/interface wireless snooper
set channel-time=200ms multiple-channels=yes receive-errors=no
/ip accounting
set account-local-traffic=no enabled=no threshold=256
/ip accounting web-access
set accessible-via-web=no address=0.0.0.0/0
/ip address
add address=192.168.88.1/24 comment=defconf disabled=no interface=combo1 network=192.168.88.0
add address=65.xxx.xxx.xxx/28 disabled=no interface=bridge1 network=65.xxx.xxx.xxx
add address=65.xxx.xxx.xxx/28 disabled=no interface=bridge1 network=65.xxx.xxx.xxx
/ip cloud
set ddns-enabled=no ddns-update-interval=none update-time=yes
/ip cloud advanced
set use-local-address=no
/ip dhcp-server config
set accounting=yes interim-update=0s store-leases-disk=5m
/ip dns
set allow-remote-requests=yes cache-max-ttl=1w cache-size=2048KiB max-concurrent-queries=100 max-concurrent-tcp-sessions=20 max-udp-packet-size=4096 query-server-timeout=2s query-total-timeout=10s servers=1.1.1.1,8.8.8.8 use-doh-server="" verify-doh-cert=no
/ip firewall address-list
add address=63.xxx.xxx.xxx/29 disabled=no list=Trusted
add address=65.xxx.xxx.xxx/28 disabled=no list=Trusted
add address=65.xxx.xxx.xxx/24 disabled=no list=Trusted
/ip firewall filter
add action=accept chain=input src-address-list=Trusted
add action=accept chain=forward src-address-list=Trusted
add action=accept chain=input connection-state=established,related,untracked
add action=drop chain=input
/ip firewall mangle
add action=mark-packet chain=prerouting comment=HubNed dst-address=65.xxx.xxx.xxx new-packet-mark=HubDownload passthrough=yes
add action=mark-packet chain=prerouting comment=HubNed new-packet-mark=HubUpload passthrough=yes src-address=65.xxx.xxx.xxx
add action=mark-packet chain=prerouting comment=WB dst-address=65.xxx.xxx.xxx new-packet-mark=WBDownload passthrough=yes
add action=mark-packet chain=prerouting comment=WB new-packet-mark=WBUpload passthrough=yes src-address=65.xxx.xxx.xxx
add action=mark-packet chain=prerouting comment=AMB dst-address=65.xxx.xxx.xxx new-packet-mark=OCDownload passthrough=yes
add action=mark-packet chain=prerouting comment=AMB new-packet-mark=OCUpload passthrough=yes src-address=65.xxx.xxx.xxx
add action=mark-packet chain=prerouting comment=Tenant dst-address=65.xxx.xxx.xxx new-packet-mark=TenantDownload passthrough=yes
add action=mark-packet chain=prerouting comment=Tenant new-packet-mark=TenantUpload passthrough=yes src-address=65.xxx.xxx.xxx
add action=mark-packet chain=prerouting comment=AS dst-address=65.xxx.xxx.xxx new-packet-mark=ASDownload passthrough=yes
add action=mark-packet chain=prerouting comment=AS new-packet-mark=ASUpload passthrough=yes src-address=65.xxx.xxx.xxx
add action=mark-packet chain=prerouting comment=NedNet dst-address=65xxx.xxx.xxx new-packet-mark=NednetDownload passthrough=yes
add action=mark-packet chain=prerouting comment=NedNet dst-address=65.xxx.xxx.xxx new-packet-mark=NednetDownload passthrough=yes
add action=mark-packet chain=prerouting comment=NedNet new-packet-mark=NednetUpload passthrough=yes src-address=65.xxx.xxx.xxx
add action=mark-packet chain=prerouting comment=NedNet new-packet-mark=NednetUpload passthrough=yes src-address=65.xxx.xxx.xxx
add action=mark-packet chain=prerouting disabled=yes dst-address=65.xxx.xxx.xxx new-packet-mark=TenantDownload passthrough=yes
add action=mark-packet chain=prerouting disabled=yes dst-address=65xxx.xxx.xxx new-packet-mark=TenantDownload passthrough=yes
/ip firewall service-port
set ftp disabled=yes ports=21
set tftp disabled=yes ports=69
set irc disabled=yes ports=6667
set h323 disabled=yes
set sip disabled=yes ports=5060,5061 sip-direct-media=yes sip-timeout=1h
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip hotspot service-port
set ftp disabled=no ports=21
/ip hotspot user
set [ find default=yes ] comment="counters and limits for trial users" disabled=no name=default-trial
/ip ipsec policy
set 0 disabled=no dst-address=::/0 group=default proposal=default protocol=all src-address=::/0 template=yes
/ip ipsec settings
set accounting=yes interim-update=0s xauth-use-radius=no
/ip proxy
set always-from-cache=no anonymous=no cache-administrator=webmaster cache-hit-dscp=4 cache-on-disk=no cache-path=web-proxy enabled=no max-cache-object-size=2048KiB max-cache-size=unlimited max-client-connections=600 max-fresh-time=3d max-server-connections=600
parent-proxy=:: parent-proxy-port=0 port=8080 serialize-connections=no src-address=::
/ip route
add !bgp-as-path !bgp-atomic-aggregate !bgp-communities !bgp-local-pref !bgp-med !bgp-origin !bgp-prepend !check-gateway disabled=no distance=1 dst-address=0.0.0.0/0 gateway=65.xxx.xxx.xxx !route-tag !routing-mark scope=30 target-scope=10
add !bgp-as-path !bgp-atomic-aggregate !bgp-communities !bgp-local-pref !bgp-med !bgp-origin !bgp-prepend !check-gateway disabled=no distance=1 dst-address=0.0.0.0/0 gateway=65.xxx.xxx.xxx !route-tag !routing-mark scope=30 target-scope=10
add !bgp-as-path !bgp-atomic-aggregate !bgp-communities !bgp-local-pref !bgp-med !bgp-origin !bgp-prepend !check-gateway disabled=no distance=1 dst-address=0.0.0.0/0 gateway=65.xxx.xxx.xxx !route-tag !routing-mark scope=30 target-scope=10
/ip service
set telnet address="" disabled=no port=23
set ftp address="" disabled=no port=21
set www address="" disabled=no port=80
set ssh address="" disabled=no port=22
set www-ssl address="" certificate=none disabled=yes port=443 tls-version=any
set api address="" disabled=no port=8728
set winbox address="" disabled=no port=8291
set api-ssl address="" certificate=none disabled=no port=8729 tls-version=any
/ip smb
set allow-guests=yes comment=MikrotikSMB domain=MSHOME enabled=no interfaces=all
/ip smb shares
set [ find default=yes ] comment="default share" directory=/pub disabled=no max-sessions=10 name=pub
/ip smb users
set [ find default=yes ] disabled=no name=guest read-only=yes
/ip socks
set auth-method=none connection-idle-timeout=2m enabled=no max-connections=200 port=1080 version=4
/ip ssh
set allow-none-crypto=no always-allow-password-login=no forwarding-enabled=no host-key-size=2048 strong-crypto=no
/ip tftp settings
set max-block-size=4096
/ip traffic-flow
set active-flow-timeout=30m cache-entries=256k enabled=no inactive-flow-timeout=15s interfaces=all
/ip traffic-flow ipfix
set bytes=yes dst-address=yes dst-address-mask=yes dst-mac-address=yes dst-port=yes first-forwarded=yes gateway=yes icmp-code=yes icmp-type=yes igmp-type=yes in-interface=yes ip-header-length=yes ip-total-length=yes ipv6-flow-label=yes is-multicast=yes
last-forwarded=yes nat-dst-address=yes nat-dst-port=yes nat-events=no nat-src-address=yes nat-src-port=yes out-interface=yes packets=yes protocol=yes src-address=yes src-address-mask=yes src-mac-address=yes src-port=yes sys-init-time=yes tcp-ack-num=yes
tcp-flags=yes tcp-seq-num=yes tcp-window-size=yes tos=yes ttl=yes udp-length=yes
/ip upnp
set allow-disable-external-interface=no enabled=no show-dummy-rule=yes
/lcd
set backlight-timeout=30m color-scheme=light default-screen=main-menu enabled=yes flip-screen=no read-only-mode=no time-interval=hour touch-screen=enabled
/lcd pin
set hide-pin-number=no pin-number=1234
/lcd interface
set sfp-sfpplus1 disabled=no max-speed=auto timeout=10s
set combo1 disabled=no max-speed=auto timeout=10s
set ether1 disabled=no max-speed=auto timeout=10s
set ether2 disabled=no max-speed=auto timeout=10s
set ether3 disabled=no max-speed=auto timeout=10s
set ether4 disabled=no max-speed=auto timeout=10s
set ether5 disabled=no max-speed=auto timeout=10s
set ether6 disabled=no max-speed=auto timeout=10s
set ether7 disabled=no max-speed=auto timeout=10s
/lcd interface pages
set 0 interfaces=sfp-sfpplus1,combo1,ether1,ether2,ether3,ether4,ether5,ether6,ether7
/lcd screen
set 0 disabled=no timeout=10s
set 1 disabled=no timeout=10s
set 2 disabled=no timeout=10s
set 3 disabled=no timeout=10s
set 4 disabled=no timeout=10s
set 5 disabled=no timeout=10s
/mpls
set dynamic-label-range=16-1048575 propagate-ttl=yes
/mpls interface
set [ find default=yes ] disabled=no interface=all mpls-mtu=1508
/mpls ldp
set distribute-for-default-route=no enabled=no hop-limit=255 loop-detect=no lsr-id=0.0.0.0 path-vector-limit=255 transport-address=0.0.0.0 use-explicit-null=no
/port firmware
set directory=firmware ignore-directip-modem=no
/ppp aaa
set accounting=yes interim-update=0s use-circuit-id-in-nas-port-id=no use-radius=no
/radius incoming
set accept=no port=3799
/routing bfd interface
set [ find default=yes ] disabled=no interface=all interval=0.2s min-rx=0.2s multiplier=5
/routing mme
set bidirectional-timeout=2 gateway-class=none gateway-keepalive=1m gateway-selection=no-gateway origination-interval=5s preferred-gateway=0.0.0.0 timeout=1m ttl=50
/routing rip
set distribute-default=never garbage-timer=2m metric-bgp=1 metric-connected=1 metric-default=1 metric-ospf=1 metric-static=1 redistribute-bgp=no redistribute-connected=no redistribute-ospf=no redistribute-static=no routing-table=main timeout-timer=3m
update-timer=30s
/snmp
set contact="" enabled=no engine-id="" location="" trap-community=public trap-generators=temp-exception trap-target="" trap-version=1
/system clock
set time-zone-autodetect=yes time-zone-name=America/Denver
/system clock manual
set dst-delta=+00:00 dst-end="jan/01/1970 00:00:00" dst-start="jan/01/1970 00:00:00" time-zone=+00:00
/system console
set [ find port=serial0 ] channel=0 disabled=no port=serial0 term=vt102
/system health
set cpu-overtemp-check=yes cpu-overtemp-startup-delay=1m cpu-overtemp-threshold=100C fan-mode=auto use-fan=main
/system identity
set name=RouterOS
/system leds
set 0 disabled=no interface=sfp-sfpplus1 leds=sfp-sfpplus1-led1 type=interface-speed
set 1 disabled=no interface=sfp-sfpplus1 leds=sfp-sfpplus1-led2 type=interface-activity
set 2 disabled=no interface=combo1 leds=combo1-led type=interface-activity
/system leds settings
set all-leds-off=never
/system logging
set 0 action=memory disabled=no prefix="" topics=info
set 1 action=memory disabled=no prefix="" topics=error
set 2 action=memory disabled=no prefix="" topics=warning
set 3 action=echo disabled=no prefix="" topics=critical
/system note
set note="" show-at-login=yes
/system ntp client
set enabled=yes primary-ntp=0.0.0.0 secondary-ntp=0.0.0.0 server-dns-names=us.pool.ntp.org
/system resource irq
set 0 cpu=auto
set 1 cpu=auto
set 2 cpu=auto
set 3 cpu=auto
set 4 cpu=auto
set 5 cpu=auto
set 6 cpu=auto
set 7 cpu=auto
set 8 cpu=auto
set 9 cpu=auto
/system resource irq rps
set sfp-sfpplus1 disabled=yes
set combo1 disabled=yes
set ether1 disabled=yes
set ether2 disabled=yes
set ether3 disabled=yes
set ether4 disabled=yes
set ether5 disabled=yes
set ether6 disabled=yes
set ether7 disabled=yes
/system routerboard settings
set auto-upgrade=no baud-rate=115200 boot-delay=2s boot-device=nand-if-fail-then-ethernet boot-protocol=bootp enable-jumper-reset=yes enter-setup-on=any-key force-backup-booter=no protected-routerboot=disabled reformat-hold-button=20s reformat-hold-button-max=10m
silent-boot=no
/system scheduler
add disabled=no interval=0s name=Reboot on-event="/system reboot" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-date=dec/13/2022 start-time=07:30:00
/system upgrade mirror
set check-interval=1d enabled=no primary-server=0.0.0.0 secondary-server=0.0.0.0 user=""
/system watchdog
set auto-send-supout=no automatic-supout=yes ping-start-after-boot=5m ping-timeout=1m watch-address=none watchdog-timer=yes
/tool bandwidth-server
set allocate-udp-ports-from=2000 authenticate=yes enabled=yes max-sessions=100
/tool e-mail
set address=0.0.0.0 from=<> port=25 start-tls=no user=""
/tool graphing
set page-refresh=300 store-every=5min
/tool graphing interface
add allow-address=0.0.0.0/0 disabled=no interface=bridge1 store-on-disk=yes
add allow-address=0.0.0.0/0 disabled=no interface=ether2 store-on-disk=yes
/tool graphing resource
add allow-address=0.0.0.0/0 disabled=no store-on-disk=yes
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool mac-server ping
set enabled=yes
/tool romon
set enabled=no id=00:00:00:00:00:00
/tool romon port
set [ find default=yes ] cost=100 disabled=no forbid=no interface=all
/tool sms
set allowed-number="" auto-erase=no channel=0 port=none receive-enabled=no
/tool sniffer
set file-limit=1000KiB file-name="" filter-cpu="" filter-direction=any filter-interface="" filter-ip-address="" filter-ip-protocol="" filter-ipv6-address="" filter-mac-address="" filter-mac-protocol="" filter-operator-between-entries=or filter-port=""
filter-size="" filter-stream=no memory-limit=100KiB memory-scroll=yes only-headers=no streaming-enabled=no streaming-server=0.0.0.0:37008
/tool traffic-generator
set latency-distribution-max=100us measure-out-of-order=no stats-samples-to-keep=100 test-id=0
/user aaa
set accounting=yes default-group=read exclude-groups="" interim-update=0s use-radius=no
[/code]

pcunite · December 24, 2022, 5:09am

I was curious as to what would make the arm based RB4011 crash. But you have given an export of the CCR1009-7G-1C-1S+, which is a Tile based design. Very different architecture and maybe there is a bug somewhere.

With so many new connections, reconsider changing tcp-established-timeout=1d to something that doesn’t use up so many resources. Applications that need to keep a connection open will send out a small packet to do so. You don’t have to do this, especially as an ISP. This really only needs to set to something like 5 minutes.

You mentioned that it takes 30 minutes to fix the issue. Might it have something to do with how your tracking connections above in combination with how you’re creating Traffic Flow graphs as well? Maybe turn this all off and see how it goes.

/ip traffic-flow
set active-flow-timeout=30m cache-entries=256k enabled=no inactive-flow-timeout=15s interfaces=all

/tool graphing set page-refresh=300 store-every=5min

/tool graphing interface
add allow-address=0.0.0.0/0 disabled=no interface=bridge1 store-on-disk=yes
add allow-address=0.0.0.0/0 disabled=no interface=ether2 store-on-disk=yes

dchardy · December 25, 2022, 3:36pm

Thanks for the suggestions, I will try those changes and see how it goes.
While observing an “episode” yesterday, I noticed some odd ping behavior that might offer a clue to what’s going on. I had a few concurrent pings running: our router->8.8.8.8, our router->gateway, a client->8.8.8.8. When the latency spiked to 1000+ ms, it did so in the ping running to the gateway, but not in the pings I had running from a client behind out router, nor in the ping from our router to 8.8.8.8. However, if I started a up new ping to the internet, it would exhibit the latency, both from our router, and from a client behind it. But the pings I started before the “episode” would continue unaffected, unless I stopped them and started them again. This seems to indicate to me that some new connection limit is being reached somewhere. Perhaps changing the “tcp-established-timeout=1d” as mentioned will address this?

David