Hi,
I have one Mikrotik CCR1009 and two CCR1036.
two hours ago, the devices started to reboot randomly every 5 to 10 mins.
I closed all the IP services except Winbox. unchecked DNS "Allow remote Requests".
upgraded from 6.40.2 to 6.40.3.
without success to solve the problem. is it a hacker doing this? or software problem?

please help it is urgent.


THanks
Yasser

If you suspect kernel failure then I doubt if three device have same problem. You need to check logs after restart if router reports kernel failure.
I suspect network loop which makes processing packets saturate them to CPU 100% and then reboot as side effect.

PS.
Mikrotiks are rarely shutdowned with a command as it is easier to switch them off with no side effects.

BartoszP is right, highly unlikely three routers start to misbehave at the same time; check your network / routers first.

If you don't find 100% CPU or other anomalies on logs:

Make sure Current Firmware is the same version as Update firmware on System > Routerboard. If not, click on upgrade and reboot the router.

A preventive measure:

1. Save an export of the routers
2. netinstall them
3. Reset it to no defaults
4. Load back their configuration .rsc file you saved on step #1

Additionally, look for autosupout.rif file in files. If there's none, check that Automatic Supout is enabled on System > Watchdog.

Check supout file date to confirm it was auto generated when the reboot happened. If that's the case, write support attaching the autosupout.rif file.

the logs is showing this message, "router was rebooted without proper shutdown, probably kernel failure" that's why I named the subject of this post by that name.
[img][/img]
the number of restarting servers is 5 now.
no new configurations added or loop. the CPU is working normally before the restart.

BartoszP is right, highly unlikely three routers start to misbehave at the same time; check your network / routers first.

If you don't find 100% CPU or other anomalies on logs:

Make sure Current Firmware is the same version as Update firmware on System > Routerboard. If not, click on upgrade and reboot the router.

A preventive measure:

1. Save an export of the routers
2. netinstall them
3. Reset it to no defaults
4. Load back their configuration .rsc file you saved on step #1

Additionally, look for autosupout.rif file in files. If there's none, check that Automatic Supout is enabled on System > Watchdog.

Check supout file date to confirm it was auto generated when the reboot happened. If that's the case, write support attaching the autosupout.rif file.

the firmware of the servers is up to date (6.40.3)

A. Change admin passwords
B. Remove "strange/unknown" users from routers ....
C. Block SSH
D. Change 8291 port to eg 28291 to blocka access via WinBox
E. Have you been changing cables ?
F. Are you sure that someone in the building have not made a loop connecting cables to local switch?

My networking rule of thumb is from some time:

1. If you do not know what is going on ... check cables.
2. If you still do not know ... goto 1.


EDIT:
Firmware is not the ROS. You check ROS version in System/Packages but firmware in System/Routerboard.

BartoszP is right, highly unlikely three routers start to misbehave at the same time; check your network / routers first.

If you don't find 100% CPU or other anomalies on logs:

Make sure Current Firmware is the same version as Update firmware on System > Routerboard. If not, click on upgrade and reboot the router.

the firmware of the servers is up to date (6.40.3)

I wasn't referring to RouterOS version, but Board Firmware, see System > Routerboard.

E. Have you been changing cables ?
F. Are you sure that someone in the building have not made a loop connecting cables to local switch?.

I'd add:

G. Are you sure nobody has created a new bridge, added a new port to an existing one, or changed/set some ethernet port master port parameter?

If you had previously, working setup .rsc files from the routers, you can generate an export and then diff actual vs previous to spot easily those kind of changes.

You can do it manually, from some editor suppporting diffs like notepad++ (windows), Sublime Text (Mac) or a purposedly built utility like Milliscript.

I have observed in the past week that a Linux system with 3.16 kernel that is directly on internet
functioning only as a router (with iptables firewall of course) has had kernel panics several times,
after running for at least 2 years without any issue. Unfortunately, after setting up netconsole to
get the detailed panic info it stopped doing that. There may be some bug that is being scanned,
or maybe targeted.

Hi again,
After disabling the SSH and changing the default port number for the Winbox, 4 servers seems to be working fine for more than one and a half hour, but the only server remains unstable is the main server linking my network to the outside world wide web.
this server is restarting every 25 mins. still not able to diagnose the problem, but for sure it is not a loop problem.
any security suggestion to be applied on Mikrotik?
when I was writing this reply, the main server restarted and after that I noticed that the other 4 also restarted after 2 mins.

Do you have any capability of tracing the internet traffic outside of the routers (e.g. with a switch with monitoring port and attached PC running wireshark)?
It could be interesting to have a capture of the traffic at the time the crash occurs.
Do you have a syslog server configured on the routers? It can be a good idea, to maybe capture some log message that occurs before the crash.
(which you don't see anymore after the reboot that shows the kernel failure message as the first)

I applied some rules found it on the Internet:
/ip firewall filter
add action=accept chain=input dst-port="Winbox Port" in-interface=ether1 protocol=tcp
add action=accept chain=input dst-port=8728 in-interface=ether1 protocol=tcp
add action=drop chain=input in-interface=ether1 protocol=tcp
add action=drop chain=input in-interface=ether1 protocol=udp
add action=drop chain=input in-interface=ether1 protocol=icmp
add action=accept chain=input dst-port=1723 in-interface=ether1 protocol=tcp
add action=accept chain=input in-interface=ether1 protocol=udp src-port=1723
add action=accept chain=input in-interface=ether1 protocol=tcp src-port=1723

these rules seems to be working fine, the total traffic dropped more than 100MB over the past couple of hours.
do you have any suggestions?
these rules applied only on the main server, do I need to apply it on all servers? what the rules should look like if I have setup a hotspot on some of them?

Are you sure that attack from "outer space"? Can you have bright whizz inside company who is "testing" your net? You have disabled only WAN side with these rules.

If you have a router which reboots itself, then conversation in forum will only be a guessing game. If you want to find out for sure what is the problem, then send supout file (generated after reboot) to support@mikrotik.com. We are the only ones who can tell what the problem was. Tracing the cause of the issue might take a while, but still it is the best solution how to get rid of the problem.

I applied some rules found it on the Internet:

I advise you to find a consultant that can configure your network with better rules.

If you have a router which reboots itself, then conversation in forum will only be a guessing game. If you want to find out for sure what is the problem, then send supout file (generated after reboot) to support@mikrotik.com. We are the only ones who can tell what the problem was. Tracing the cause of the issue might take a while, but still it is the best solution how to get rid of the problem.

I am sending the support file now.
thanks

Are you sure that attack from "outer space"? Can you have bright whizz inside company who is "testing" your net? You have disabled only WAN side with these rules.

yes you are right.
we decided to apply these rules for only WAN port because the attack was not only on our servers, it was also on a broader range of servers.
I don't know if the traffic rejected by the rules is a good sign or bad sign, but the server still up and running since 17 hours ago: I can't tell if these rules solved the problem or something else.

Any update on this? I experienced a similar crash this morning on our main CCR1016. I sent the supout file to Mikrotik but any updates on what happened?

Any update on this? I experienced a similar crash this morning on our main CCR1016. I sent the supout file to Mikrotik but any updates on what happened?

Hi,
it seems like I was exposed to an external attack, please use the rules I posted on my previous reply, they solved the problem.

Awesome thanks for the reply.

Awesome thanks for the reply.

it is my pleasure. try also to update the router software to 6.40.5
it is also could be system failure.

Yeah I was already on 6.40.5 but what I hadnt done is upgraded the actual routerboard firmware itself. I upgraded that as well yesterday. I have only had a single crash yesterday morning so keeping an eye on things.

Hello,

i have same problem. Unfortunately, i cannot make supout.rif, because it crashes and restarts directly.
i use v6.41.3 in rb433AH

Configuration
----------------
PPP to radius
WDS wireless to APs
ethernet to APs
MDF for all APs
WAN for all APs

Total HDD Size: 512 MiB

i have checked CPU by Tools -> Profile.
cpu0 - 20% what is?
wireless - 9%
ovpn - 10 %
l2tp -5 %
pptp -5 %

Please answer.

Hello,

i have same problem. Unfortunately, i cannot make supout.rif, because it crashes and restarts directly.
i use v6.41.3 in rb433AH

Configuration
----------------
PPP to radius
WDS wireless to APs
ethernet to APs
MDF for all APs
WAN for all APs

Total HDD Size: 512 MiB

i have checked CPU by Tools -> Profile.
cpu0 - 20% what is?
wireless - 9%
ovpn - 10 %
l2tp -5 %
pptp -5 %

Please answer.

You have also upgrade the 6.41.3 BIOS (firmware)?

Install RouterOS 6.40.7 (bugfix) and update the bios (firmware) to the 3.41 (already ready to install inside 6.40.7)

cpu0: RB433AH do not have "switch" chip, all traffic go trought CPU and the CPU is used as "switch" between ether and wlan...

Hello,

i have same problem. Unfortunately, i cannot make supout.rif, because it crashes and restarts directly.
i use v6.41.3 in rb433AH

Configuration
----------------
PPP to radius
WDS wireless to APs
ethernet to APs
MDF for all APs
WAN for all APs

Total HDD Size: 512 MiB

i have checked CPU by Tools -> Profile.
cpu0 - 20% what is?
wireless - 9%
ovpn - 10 %
l2tp -5 %
pptp -5 %

Please answer.

You have also upgrade the 6.41.3 BIOS (firmware)?

Install RouterOS 6.40.7 (bugfix) and update the bios (firmware) to the 3.41 (already ready to install inside 6.40.7)

cpu0: RB433AH do not have "switch" chip, all traffic go trought CPU and the CPU is used as "switch" between ether and wlan...

Thanks for your answer and recommendations
yes, i have upgraded BIOS to 6.41.3
Now i'm trying your tip.
Best regards!

i have downgraded to v6.40.7 and updates the bios to the 3.41, but it still reboots again.

i have replaced miniPCI on ROS and RouterBoard (v4.60.7). Unfortunately, it repeatedly restarts again.

i have created supoit.rif and sent it to support team.

this is a bug?

maybe it is switch problem?

mikrotik 7.8 has this buge . after 250 online user has been rebooted .

i face this error :
router was rebooted without proper shutdown
kernel failure in previous boot