Is the following normal operation for the 4011? It is doing basic routing, simple queues and firewall/nat.

You can see from the first graph on Friday all went to hell, it was barely accessible via MAC-telnet but managed to reboot.

Export config. Hide sensitive

After it crashes or reboots, generate and send supout to support@mikrotik.com - that's the ONLY way how to get any official support and to help them fix this issue.
Posting here on forum does nothing, except maybe informs other users about it.

Who says I haven't sent the sup out info in already regarding the crash?

That was merely one part of the issue and informing the community helps if anyone experiences the same in the future.

The second part which is open to the community, including the Tik bods, was the question about normal operation of a multicore router.

I have emailed Mikrotik many a time before and had no response so reaching out to the forum is my second port of call.

If you don't have anything to add, please don't respond at all.

Who says I haven't sent the sup out info in already regarding the crash?

You haven't said you did...and it does make sense to give the advice.

Based on the information you supplied...this is not normal operation.
Could you please share your config (as per requested by Dude2048)?
What version are you running?

Interesting! Not to hijack, with PRTG you using? I like those graphs.

Do you have any ipsec tunnels? Are you using bridge vlan configuration? do your config export.... /export hide-sensitive

Thanks peeps, will post config as soon as I get back to the office.

Graphs are from Librenms, yes to a single IPSEC tunnel back to our NOC, very little traffic. No VLANs.

It might be the IPsec tunnel (HW Offload) with the type your using (Encryption type). Unsure if a bug and its loading up on single CPU core. I wouldnt worry if only single core being loaded up.
DH Group
SHA, MD5?
SHA-128, SHA-256?

What routerOS release? Try the 6.45 long term.

# aug/22/2019 17:27:49 by RouterOS 6.45.3
# software id = FDK9-ISF2
#
# model = RB4011iGS+
# serial number = AAAF0A1A68A6
/interface bridge
add name=********inn-bridge
add name=loopback
/interface ethernet
set [ find default-name=ether1 ] comment=\
set [ find default-name=ether2 ] comment=
set [ find default-name=ether3 ] comment=\
set [ find default-name=ether4 ] comment=\
set [ find default-name=ether5 ] comment=\
set [ find default-name=ether6 ] comment="L2 MGMT"
set [ find default-name=ether7 ] disabled=yes
set [ find default-name=ether8 ] disabled=yes
set [ find default-name=ether9 ] disabled=yes
set [ find default-name=ether10 ] disabled=yes
set [ find default-name=sfp-sfpplus1 ] disabled=yes
/interface pppoe-client
add add-default-route=yes comment= \
default-route-distance=2 interface=ether5 name=pppoe-out-******** \
password=******** use-peer-dns=yes user=********inn-********
add add-default-route=yes comment="PPPoE Client to ********" disabled=no \
interface=ether3 name=pppoe-out-******** password=******** use-peer-dns=\
yes user=mw_********_router
/interface l2tp-client
add allow=mschap2 comment="VPN to ******** CHR / Network Monitoring" connect-to=\
******** disabled=no ipsec-secret=******** name=l2tp-out-******** \
password=******** use-ipsec=yes user=********-********
add allow=chap,mschap2 comment="VPN to ******** / RADIUS" connect-to=\
gw1-********.********.co.uk disabled=no name=l2tp-out-******** password=\
******** user=fn-m********
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=********inn-pppoe-nat ranges=10.10.8.130-10.10.8.254
add name=********inn-dhcp-infrastructure ranges=10.10.8.100-10.10.8.126
add name=********-dhcp-infrastructure ranges=10.103.98.2-10.103.98.254
add name=********-infrastructure ranges=10.103.8.200-10.103.8.254
add name=********-pppoe-nat ranges=10.103.40.2-10.103.40.254
add name=********-pppoe-nat ranges=10.103.80.2-10.103.80.254
/ip dhcp-server
add address-pool=********inn-dhcp-infrastructure disabled=no interface=\
********inn-bridge name=********inn-infrastructure
add address-pool=********-dhcp-infrastructure disabled=no interface=ether4 \
name=********-infrastructure
add address-pool=********-infrastructure disabled=no interface=ether5 name=\
********-infrastructure
/ppp profile
add dns-server=10.10.8.129 local-address=10.10.8.129 name=\
********inn-pppoe-profile queue-type=default remote-address=********inn-pppoe-nat
add dns-server=10.103.40.1 local-address=10.103.40.1 name=\
********-pppoe-profile queue-type=default remote-address=\
********-pppoe-nat
add dns-server=10.103.80.1 local-address=10.103.80.1 name=********-pppoe-profile \
queue-type=default remote-address=********-pppoe-nat
/routing ospf area
set [ find default=yes ] disabled=yes
add area-id=0.0.0.8 disabled=yes name=pppoe
/routing ospf instance
set [ find default=yes ] disabled=yes router-id=10.0.0.8
add name=******** router-id=10.0.0.8
/routing ospf area
add area-id=0.0.0.1 instance=******** name=********
/snmp community
set [ find default=yes ] read-access=no
add addresses=10.200.200.0/24 name=********snmp
/interface bridge port
add bridge=********inn-bridge interface=ether1
add bridge=********inn-bridge interface=ether2
/interface pppoe-server server
add default-profile=********inn-pppoe-profile disabled=no interface=\
********inn-bridge service-name=********inn-pppoe-server
add authentication=pap default-profile=********-pppoe-profile disabled=no \
interface=ether4 service-name=********-pppoe-server
add default-profile=********-pppoe-profile disabled=no interface=ether5 \
service-name=********-pppoe-server
/ip address
add address=10.0.0.8 comment="Loopback Address" interface=loopback network=\
10.0.0.8
add address=10.103.98.1/24 comment="OLD Config / ******** DHCP" interface=\
ether4 network=10.103.98.0
add address=10.103.4.1/24 comment="OLD Config / ******** Infrastructure" \
interface=ether4 network=10.103.4.0
add address=10.103.8.1/24 comment="OLD Config / ******** Infrastructure & DHCP" \
interface=ether5 network=10.103.8.0
add address=10.10.8.1/25 comment="NEW Config / ******** Inn Infrastructure" \
interface=********inn-bridge network=10.10.8.0
add address=10.100.0.102/29 comment="******** Inn > ******** (OSPF)" interface=\
ether3 network=10.100.0.96
add address=10.103.5.1/24 comment="OLD Config / ******** Infrastructure" \
disabled=yes interface=ether5 network=10.103.5.0
/ip dhcp-client
add default-route-distance=11 dhcp-options=hostname,clientid interface=ether3
add default-route-distance=10 dhcp-options=hostname,clientid interface=ether5
/ip dhcp-server network
add address=10.10.8.0/25 dns-server=10.10.8.1 gateway=10.10.8.1
add address=10.103.8.0/24 dns-server=10.103.8.1 gateway=10.103.8.1
add address=10.103.98.0/24 dns-server=10.103.98.1 gateway=10.103.98.1
/ip dns
set allow-remote-requests=yes cache-size=4096KiB servers=1.1.1.1,9.9.9.9
/ip dns static
add address=10.200.200.5 name=unms.********-uk.com
add address=10.200.200.4 name=unmsx.********-uk.com
add address=******** name=unifi.********-uk.com
/ip firewall filter
add action=accept chain=input comment="Allow input/NAT 'established'" \
connection-state=established
add action=accept chain=forward comment="Allow input/NAT 'established'" \
connection-state=established
add action=accept chain=input comment="Allow input/NAT 'related'" \
connection-state=related
add action=accept chain=forward comment="Allow input/NAT 'related'" \
connection-state=related
add action=accept chain=input comment="Allow Winbox" dst-port=18291 protocol=\
tcp
add action=accept chain=input comment="Allow VPN (L2TP)" dst-port=\
1701,4500,500 protocol=udp
add action=drop chain=input comment="Drop Invalid connections" \
connection-state=invalid in-interface=pppoe-out-******** protocol=tcp
add action=drop chain=input comment="Drop external DNS requests" dst-port=53 \
in-interface=pppoe-out-******** protocol=tcp
add action=drop chain=input comment="Drop external DNS requests" dst-port=53 \
in-interface=pppoe-out-******** protocol=udp
add action=add-src-to-address-list address-list=spammers \
address-list-timeout=3h chain=forward comment=\
"Add Spammers to the list for 3 hours" connection-limit=30,32 dst-port=\
25,587,465 limit=30/1m,0:packet log=yes log-prefix=SPAMMERS----> \
protocol=tcp
add action=drop chain=forward comment="Avoid spammers action" dst-port=\
25,587,465 log=yes log-prefix=SPAMMERS----> protocol=tcp \
src-address-list=spammers
add action=add-src-to-address-list address-list=Port_Scanner \
address-list-timeout=1w chain=input comment="Port Scanner Detect" \
protocol=tcp psd=21,3s,3,1
add action=drop chain=input comment="Drop to port scan list" \
src-address-list=Port_Scanner
add action=drop chain=input comment="drop ssh brute forcers" dst-port=22 \
protocol=tcp src-address-list=ssh_blacklist
add action=add-src-to-address-list address-list=ssh_blacklist \
address-list-timeout=1w3d chain=input connection-state=new dst-port=22 \
protocol=tcp src-address-list=ssh_stage3
add action=add-src-to-address-list address-list=ssh_stage3 \
address-list-timeout=1m chain=input connection-state=new dst-port=22 \
protocol=tcp src-address-list=ssh_stage2
add action=add-src-to-address-list address-list=ssh_stage2 \
address-list-timeout=1m chain=input connection-state=new dst-port=22 \
protocol=tcp src-address-list=ssh_stage1
add action=add-src-to-address-list address-list=ssh_stage1 \
address-list-timeout=1m chain=input connection-state=new dst-port=22 \
protocol=tcp
add action=drop chain=input comment="drop telnet brute forcers" dst-port=23 \
protocol=tcp src-address-list=telnet_blacklist
add action=add-src-to-address-list address-list=telnet_blacklist \
address-list-timeout=1w3d chain=input connection-state=new dst-port=23 \
protocol=tcp src-address-list=telnet_stage3
add action=add-src-to-address-list address-list=telnet_stage3 \
address-list-timeout=1m chain=input connection-state=new dst-port=23 \
protocol=tcp src-address-list=telnet_stage2
add action=add-src-to-address-list address-list=telnet_stage2 \
address-list-timeout=1m chain=input connection-state=new dst-port=23 \
protocol=tcp src-address-list=telnet_stage1
add action=add-src-to-address-list address-list=telnet_stage1 \
address-list-timeout=1m chain=input connection-state=new dst-port=23 \
protocol=tcp
add action=drop chain=input comment="drop winbox brute forcers" dst-port=8291 \
protocol=tcp src-address-list=winbox_blacklist
add action=add-src-to-address-list address-list=winbox_blacklist \
address-list-timeout=1w3d chain=input connection-state=new dst-port=8291 \
protocol=tcp src-address-list=winbox_stage3
add action=add-src-to-address-list address-list=winbox_stage3 \
address-list-timeout=1m chain=input connection-state=new dst-port=8291 \
protocol=tcp src-address-list=winbox_stage2
add action=add-src-to-address-list address-list=winbox_stage2 \
address-list-timeout=1m chain=input connection-state=new dst-port=8291 \
protocol=tcp src-address-list=winbox_stage1
add action=add-src-to-address-list address-list=winbox_stage1 \
address-list-timeout=1m chain=input connection-state=new dst-port=8291 \
protocol=tcp
add action=drop chain=input comment="Drop everything else from ether8" \
in-interface=pppoe-out-********
/ip firewall nat
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** PPPoE Clients" out-interface=\
pppoe-out-******** src-address=10.103.40.0/24
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** PPPoE Clients" out-interface=pppoe-out-******** \
src-address=10.103.80.0/24
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** Inn PPPoE Clients" out-interface=\
pppoe-out-******** src-address=10.10.8.128/25
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** Infrastructure" out-interface=\
pppoe-out-******** src-address=10.103.4.0/24
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** Infrastructure" out-interface=pppoe-out-******** \
src-address=10.103.8.0/24
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** Inn Infrastructure" out-interface=\
pppoe-out-******** src-address=10.10.8.0/25
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** Infrastructure" out-interface=\
pppoe-out-******** src-address=10.103.98.0/24
# pppoe-out-******** not ready
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** PPPoE Clients" out-interface=\
pppoe-out-******** src-address=10.103.40.0/24
# pppoe-out-******** not ready
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** Inn PPPoE Clients" out-interface=\
pppoe-out-******** src-address=10.10.8.128/25
# pppoe-out-******** not ready
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** Infrastructure" out-interface=\
pppoe-out-******** src-address=10.103.4.0/24
# pppoe-out-******** not ready
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** Inn Infrastructure" out-interface=\
pppoe-out-******** src-address=10.10.8.0/25
# pppoe-out-******** not ready
add action=masquerade chain=srcnat comment=\
"NAT Masquerade / ******** Infrastructure" out-interface=\
pppoe-out-******** src-address=10.103.98.0/24
add action=masquerade chain=srcnat comment="TEMP TO BE REMOVED AFTER INSTALL" \
disabled=yes
/ip route
add comment="VPN Route to ******** / RADIUS Server 1" distance=1 dst-address=\
10.1.1.151/32 gateway=l2tp-out-********
add comment="VPN Route to ******** / RADIUS Server 2" distance=1 dst-address=\
10.1.1.152/32 gateway=l2tp-out-********
add distance=1 dst-address=10.200.200.2/32 gateway=l2tp-out-********
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set winbox port=********
set api-ssl disabled=yes
/ip ssh
set allow-none-crypto=yes forwarding-enabled=remote
/ppp aaa
set interim-update=10m use-radius=yes
/radius
add address=10.1.1.151 comment="******** RADIUS Server 1" secret=******** \
service=ppp src-address=10.0.0.1 timeout=3s
add address=10.1.1.152 comment="******** RADIUS Server 2" secret=******** \
service=ppp timeout=3s
/routing ospf network
add area=backbone disabled=yes network=10.10.8.0/25
add area=pppoe disabled=yes network=10.10.8.128/25
add area=******** network=10.0.0.8/32
add area=backbone disabled=yes network=10.100.0.96/29
add area=******** network=10.200.200.0/24
add area=******** network=10.103.4.0/24
add area=******** network=10.103.40.0/24
add area=******** network=10.103.80.0/24
add area=******** network=10.103.8.0/24
add area=******** network=10.103.98.0/24
/snmp
set contact=info@********-uk.com enabled=yes location="********" trap-community=\
********snmp trap-version=2
/system clock
set time-zone-name=Europe/London
/system identity
set name=INF_********
/system ntp client
set enabled=yes primary-ntp=85.199.214.102 secondary-ntp=109.74.192.97
/system script
add dont-require-permissions=no name=speedtest owner=******** policy=\
ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source=":\
local txc\r\
\n:local txcA\r\
\n:local txcB\r\
\n:local txcC\r\
\n\r\
\n:local rxc\r\
\n:local rxcA\r\
\n:local rxcB\r\
\n:local rxcC\r\
\n\r\
\n:local rxta\r\
\n:local rxtaA\r\
\n:local rxtaB\r\
\n:local rxtaC\r\
\n\r\
\n:local txta\r\
\n:local txtaA\r\
\n:local txtaB\r\
\n:local txtaC\r\
\n\r\
\n:local sysname [/system identity get name]\r\
\n:local datetime \"\$[/system clock get date] \$[/system clock get time]\
\"\r\
\n:local interfaces [/interface get 0 comment]\r\
\n\r\
\n:log info \"Performing Internet Connection Speed Test...\"\r\
\n\r\
\n/tool bandwidth-test ******** protocol=tcp direction=receive dura\
tion=30s user=******** password=******** do={\r\
\n\r\
\n:set txcA (\$\"tx-current\" / 1000)\r\
\n:set txcB (\$txcA / 1000 * 1000)\r\
\n:set txcC (\$txcA - \$txcB)\r\
\n:set txcB (\$txcB / 1000)\r\
\n:set txc \"\$txcB.\$txcC\"\r\
\n\r\
\n:set rxcA (\$\"rx-current\" / 1000)\r\
\n:set rxcB (\$rxcA / 1000 * 1000)\r\
\n:set rxcC (\$rxcA - \$rxcB)\r\
\n:set rxcB (\$rxcB / 1000)\r\
\n:set rxc \"\$rxcB.\$rxcC\"\r\
\n\r\
\n:set rxtaA (\$\"rx-total-average\" / 1000)\r\
\n:set rxtaB (\$rxtaA / 1000 * 1000)\r\
\n:set rxtaC (\$rxtaA - \$rxtaB)\r\
\n:set rxtaB (\$rxtaB / 1000)\r\
\n:set rxta \"\$rxtaB.\$rxtaC\"\r\
\n\r\
\n:set txtaA (\$\"tx-total-average\" / 1000)\r\
\n:set txtaB (\$txtaA / 1000 * 1000)\r\
\n:set txtaC (\$txtaA - \$txtaB)\r\
\n:set txtaB (\$txtaB / 1000)\r\
\n:set txta \"\$txtaB.\$txtaC\"\r\
\n\r\
\n}\r\
\n\r\
\n:log info \"Speed Test Complete: Sending report by e-mail\"\r\
\n\r\
\n/tool e-mail send to=\"m********@********-uk.com\" subject=\"Internet \
Bandwidth Speed Test Complete: \$sysname\" body=\"Site Name: \$sysname\\nT\
ime Conducted: \$datetime \\nInterfaces: \$interfaces \\n \\n Results: \\n\
\_\\n Upload speed \$txc Mbps/s \\n Download speed \$rxc Mbps/s \\n \\n Up\
load total average \$txta Mbps/s \\n Download total average \$rxta Mbps/s\
\""

which is your current firmware in system -->routerboard ??

/system routerboard print

(firmware not routeros)

I believe this bug has also bitten me. CPU goes to the extreme, and eventually, the system grinds to unresponsiveness. It doesn't reboot; it just sits there constipated. Maybe it could be a thread leak? It is syslogging to a remote server and there aren't any errors. Instead, it goes silent without warning. Is there a way I can better watch CPU/threads to find out what this is?

The only concrete symptoms/logs I can find/show is this CPU graph from Observium. You can see in the following charts that it stopped at about noon on Saturday, but that's not when I cold-rebooted. My network was broken for about 3 hours until someone complained. The system that went down is my home network, so not a big deal, but I also use several rb4011s at work to interconnect remote sites. I hope those rb4011s don't puke too.

After opening a ticket with Mikrotik Support, they determined that the cause was faulty hardware. They are aware that some RB4011 hardware has flaws that can cause this and the warranty should cover a replacement.

Is there any official guidance on this? I have an RB4011 non-wifi that locked up the first time after 75 days uptime, then again after another 90ish days but the second time it did not recover on a cold reboot. I am going to attempt to netinstall it back to life. It would be nice to know if there are known batches of defective hardware, or if there is a firmware/bootloader/routeros combination that mitigates the issue.

Followup - Looked closer at my logs and its about 60ish days between the lockups. Managed to unbrick the unit using Netinstall and move from Long Term 6.44.3 to Stable 6.46.2. We'll see what happens.

I believe this bug has also bitten me. CPU goes to the extreme, and eventually, the system grinds to unresponsiveness. It doesn't reboot; it just sits there constipated. Maybe it could be a thread leak? It is syslogging to a remote server and there aren't any errors. Instead, it goes silent without warning. Is there a way I can better watch CPU/threads to find out what this is?

The only concrete symptoms/logs I can find/show is this CPU graph from Observium. You can see in the following charts that it stopped at about noon on Saturday, but that's not when I cold-rebooted. My network was broken for about 3 hours until someone complained. The system that went down is my home network, so not a big deal, but I also use several rb4011s at work to interconnect remote sites. I hope those rb4011s don't puke too.

We have the very same Issue with aur RB4011:


I cant acces the Router with Winbox nor with ssh or over HTML.
Is it fix that this is a hardware failure?

I have removed 4 units now - is there an explanation for the lock up's yet??

Sigh, the new RB4011 that I received as a replacement for this problem started doing this today, too.

The profiler gives no clue as to which process is actually pegging a core.

It hadn't fully hung yet when I caught it, so I actually got some decent supouts as it was happening this time, so hopefully, Mikrotik Support can see what is going on. I'll try to remember to update here if they find anything new.

I have a hunch this might be caused by the L2TP client. Are any of you seeing this issue not running an L2TP server or client?

Latest lockup and reboot showed this: jul/22/2020 22:26:52 system,error,critical kernel failure in previous boot

Mikrotik Support analyzed my supouts and says that my problem is still hardware and wants me to RMA again. RMA roulette is not a very fun game to play. It is even more painful because I made the mistake of buying from a random Amazon Marketplace seller who had a great purchase price but who has pretty bad customer support.

I can't help wondering if this has something to do with the AL21400 SoC. I have never seen any other RouterBoard devices do this besides the RB4011. I have never seen a TILE or MMIPS RouterBoard hang like this.

I have an RB4011 running for about two weeks now on latest firmware and software. It's doing 4 IP tunnels and total traffic maxes out currently at 50k PPS.

I have 0 issues so far.