Community discussions

MikroTik App
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

CCR1009 Rebooting

Sun Nov 27, 2022 3:20 am

CCR1009-7G-1C-1S+ v7.6

Seeing this in the logs

router was rebooted without proper shutdown, probably kernel failure

then

kernel failure in previous boot

Any suggestions? Where should I look to get more information?
 
biomesh
Long time Member
Long time Member
Posts: 561
Joined: Fri Feb 10, 2012 8:25 pm

Re: CCR1009 Rebooting

Sun Nov 27, 2022 4:15 am

You might want to netinstall the device to make sure there isn't an issue with the flash or the os install. I would also check the capacitors to make sure none are failing. If you have a backup power supply, I would also test that as well.
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Sun Nov 27, 2022 4:24 am

You might want to netinstall the device to make sure there isn't an issue with the flash or the os install. I would also check the capacitors to make sure none are failing. If you have a backup power supply, I would also test that as well.
That mirrors my thoughts.. :)

Cheers
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Sun Nov 27, 2022 5:27 am

I got the exact same problem with my CCR1009-8G-1S-1S+

I will try Netinstall.

edit: Won't do Netinstall... as it factory reset the routeur, I do not want that. Maybe 7.7 Beta? Altrought I don't see anything "fixing" in the release.
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Sun Nov 27, 2022 5:42 am

You might want to netinstall the device to make sure there isn't an issue with the flash or the os install. I would also check the capacitors to make sure none are failing. If you have a backup power supply, I would also test that as well.
That mirrors my thoughts.. :)

Cheers
Update me if it fix it.
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Sun Nov 27, 2022 7:45 am

I got the exact same problem with my CCR1009-8G-1S-1S+
Wasn't a power supply issue.

Very interesting that you have the same issue... When did yours start?

I sent the supout to support, trying NetInstall now..
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Sun Nov 27, 2022 10:17 am

Update me if it fix it.
Been a couple hours and it hasn't rebooted..

First tried restoring a rsc export after the NetInstall, that didn't go well, didn't want to restore the full backup because I didn't want to restore the corruption.. But I did because the rsc import was not good..

Been a couple hours and still good.

Edit: Not anymore.. Rebooting again..
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Sun Nov 27, 2022 4:30 pm

I think it started at the beginning of the week. I'm not sure when I updated to 7.6, but I feel like it's related.

Tried 7.7Beta8 and it's still rebooting.

Wish I could go back to 7.5 and try, but the "Downgrade" options just reboot into 7.6 :(
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Sun Nov 27, 2022 4:49 pm

ok, fond how to downgrade, pretty simple in fact.

Get older version here : https://mikrotik.com/download/archive (you need "tile" .npk for the CCR1009)

Upload into "Files", then go into "Package" and click "Downgrade". Router will reboot and be into the older version.

Trying 7.5 for now.
edit #1 : 7.5 crashing, trying 7.4.1 (can't recall what version I had before 7.6).
 
hughccl
just joined
Posts: 2
Joined: Sun Nov 27, 2022 10:21 pm

Re: CCR1009 Rebooting

Sun Nov 27, 2022 10:24 pm

I've got the exact same issue on a CCR1009-8G-1S-1S+ that has just started in the last couple of days. I was on 7.3 when it started, upgraded to 7.6 and it has crashed one more time a couple of hours later. Seems a bit of a coincidence that that three of us have the same issue starting around the same time with similar models??
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Mon Nov 28, 2022 5:24 am

Trying 7.5 for now.
edit #1 : 7.5 crashing, trying 7.4.1 (can't recall what version I had before 7.6).
Yes, I did the same, 7.5 still crashed for me too.

I had a transparent firewall set on two ports, removed them because the CCR1009 was unstable, the reboots stopped..

Tomorrow will check for blown caps but unplugging the two ports stopped the crashing.

Hoping support responds and doesn't just fix the issue with an update without telling me like last time..
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Mon Nov 28, 2022 6:07 am

7.4.1 seems to be stable for me, I'll update here if it still crash.

edit: crashing.
Last edited by nka on Tue Nov 29, 2022 3:28 am, edited 1 time in total.
 
User avatar
nichky
Forum Guru
Forum Guru
Posts: 1275
Joined: Tue Jun 23, 2015 2:35 pm

Re: CCR1009 Rebooting

Mon Nov 28, 2022 7:51 am

i think is too early for netinstall.
what tool/profile showing?(keep it running)

on logical reason would be, that one of those parameters getting overloaded
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Mon Nov 28, 2022 8:12 am

i think is too early for netinstall.
what tool/profile showing?(keep it running)

on logical reason would be, that one of those parameters getting overloaded
Netinstall has already been done.
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Mon Nov 28, 2022 10:50 am

I've got the exact same issue on a CCR1009-8G-1S-1S+ that has just started in the last couple of days.

Seems a bit of a coincidence that that three of us have the same issue starting around the same time with similar models??
Indeed..
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Mon Nov 28, 2022 4:04 pm

i think is too early for netinstall.
what tool/profile showing?(keep it running)

on logical reason would be, that one of those parameters getting overloaded
didn't know about the tool/profile

It's currently running (I downgraded to 7.4.1), total is around 2 to 3.5, with "profiling" at around 2.5 and management 1.0, the rest is under 1.0 (most at 0.0).

Mine has been running fine for almost 24 hours now on 7.4.1. Might need to look at the changelog to see what is new on 7.5 that could cause that Kernel Panic.

I've also enabled syslog, hoping to see something in the log before it crash (if it does).
 
biomesh
Long time Member
Long time Member
Posts: 561
Joined: Fri Feb 10, 2012 8:25 pm

Re: CCR1009 Rebooting

Mon Nov 28, 2022 6:24 pm

Just FYI my CCR1009 was running 7.5 for over a month without issues. I replaced it with a CCR2004, so the 1009 is not currently in use.

The only issues I had with 7.x and the CCR1009 were/are:

- sd card no longer works
- in earlier 7.x version there was a bug with l2tp/ipsec where it would go into high utilization randomly. It has not happened since 7.3.x from what I remember.
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Mon Nov 28, 2022 8:22 pm

7.4.1 seems to be stable for me, I'll update here if it still crash.
7.4.1 just rebooted on me.

Less than 7.5 and 7.6 but still happening.
 
wispvt
Member Candidate
Member Candidate
Posts: 109
Joined: Tue May 10, 2011 4:20 pm

Re: CCR1009 Rebooting

Mon Nov 28, 2022 9:05 pm

I have a CCR1016-12S-1S+ doing the exact same thing. It had been fine for over a month on 7.6 but in the last week since 11/24/2022 it has rebooted around 10 times. Some of the times are shortly after a previous reboot. It can't be a power supply issue as it's a dual PS router and it's fed by a UPS which also feeds other devices without issue. Any way of getting Mikrotik to look at the supout files to figure out why?
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Mon Nov 28, 2022 9:09 pm

Any way of getting Mikrotik to look at the supout files to figure out why?
Open a support ticket and include the supout files..?

I just got a response that it might be related to connection tracking.
Last edited by kevinds on Mon Nov 28, 2022 9:22 pm, edited 1 time in total.
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Mon Nov 28, 2022 9:11 pm

7.4.1 just rebooted on me.

Less than 7.5 and 7.6 but still happening.
Every 30-60 minutes now, sometimes 5-10.
Last edited by kevinds on Mon Nov 28, 2022 9:17 pm, edited 1 time in total.
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Mon Nov 28, 2022 9:17 pm

what tool/profile showing?(keep it running)

on logical reason would be, that one of those parameters getting overloaded
Nothing of any significance, was watching it the last time a reboot happened.
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Mon Nov 28, 2022 9:22 pm

Just FYI my CCR1009 was running 7.5 for over a month without issues. I replaced it with a CCR2004, so the 1009 is not currently in use.

The only issues I had with 7.x and the CCR1009 were/are:

- sd card no longer works
- in earlier 7.x version there was a bug with l2tp/ipsec where it would go into high utilization randomly. It has not happened since 7.3.x from what I remember.
Ok...?

I've got a few of them, two of them are running very similar configurations, but only one is doing this.
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Mon Nov 28, 2022 9:41 pm

I have a CCR1016-12S-1S+ doing the exact same thing. It had been fine for over a month on 7.6 but in the last week since 11/24/2022 it has rebooted around 10 times. Some of the times are shortly after a previous reboot. It can't be a power supply issue as it's a dual PS router and it's fed by a UPS which also feeds other devices without issue. Any way of getting Mikrotik to look at the supout files to figure out why?
mine has been fine until last week, can't recall when exactly... pretty wierd. Look like it's something releated to a date (certificate?) or maybe an attack that target mikrotik router ?
 
tracert
just joined
Posts: 1
Joined: Mon Nov 28, 2022 10:32 pm

Re: CCR1009 Rebooting

Mon Nov 28, 2022 10:44 pm

Hi!

I have the same issue!
CCR1036-12G-4S
Rebooting probably kernel failure
SW, FW: 7.6
Rebooting random.
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Mon Nov 28, 2022 10:47 pm

7.4.1 seems to be stable for me, I'll update here if it still crash.
7.4.1 just rebooted on me.

Less than 7.5 and 7.6 but still happening.
crashed again for me too, about 35 hours later.
 
mjmabs
just joined
Posts: 17
Joined: Tue Feb 12, 2013 11:54 pm

Re: CCR1009 Rebooting

Mon Nov 28, 2022 11:37 pm

You can add another CCR1009-8G-1S-1S+ to the list, upgraded from 6.49.7 to 7.6 about two weeks ago and all the sudden today it has rebooted about 3 times in a 12 hour period. The CCR226's running 7.6 connected to same ups haven't rebooted at all. Tool/Profile shows very little usage currently last reboot was 1.5 hours ago.
 
hughccl
just joined
Posts: 2
Joined: Sun Nov 27, 2022 10:21 pm

Re: CCR1009 Rebooting

Mon Nov 28, 2022 11:43 pm

We switched over to a CCR-1009-8G-1S from our CCR-1009-8G-1S-1S+ and having the exact same issue on both 7.3 and 7.6.

Have downgraded to 6.49.7, will see how that goes...
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Mon Nov 28, 2022 11:52 pm

Have downgraded to 6.49.7, will see how that goes...
Going to try that tonight.
 
wispvt
Member Candidate
Member Candidate
Posts: 109
Joined: Tue May 10, 2011 4:20 pm

Re: CCR1009 Rebooting

Tue Nov 29, 2022 2:18 am

I rolled back to 6.48.6 to try as my reboots increased in frequency to every 5-15 minutes. I'll post results later.
 
kevinds
Long time Member
Long time Member
Topic Author
Posts: 575
Joined: Wed Jan 14, 2015 8:41 am

Re: CCR1009 Rebooting

Tue Nov 29, 2022 8:20 am

Which RouterBoot version are you all running?

My affected unit rebooted twice on the 24th, then started frequently rebooting on the 26th.
 
wispvt
Member Candidate
Member Candidate
Posts: 109
Joined: Tue May 10, 2011 4:20 pm

Re: CCR1009 Rebooting

Tue Nov 29, 2022 2:32 pm

Yup, the 24th was the trigger date for us as well and then saw the frequency of reboots increase from the 26th on. After rolling back to 6.48.6 we've had no issues for 12 hours. It seems like it's only affecting tile processor routers from the posts.
 
JnSchl
just joined
Posts: 8
Joined: Wed Sep 23, 2015 11:11 am

Re: CCR1009 Rebooting

Tue Nov 29, 2022 3:30 pm

Same happens on our device since the update to 7.6. It's CCR1009-8G-1S-1S+ (tile)

Log:
  • router was rebooted without proper shutdown, probably kernel failure
  • kernel failure in previous boot
Firmware version of the RouterBOARD always matched the installed firmware.

We tried: routeros-7.2.1-tile.npk
>> only stable for a few minutes

Now trying: routeros-tile-6.49.7.npk
 
tamisoft
just joined
Posts: 2
Joined: Tue Nov 29, 2022 4:53 pm

Re: CCR1009 Rebooting

Tue Nov 29, 2022 4:59 pm

Exact same issues with 7.6 on CCR1009-7G started about a week ago. Support suggested turning off the ip / firewall / service ports. Did that, no change. Waiting for a solution.
 
mjmabs
just joined
Posts: 17
Joined: Tue Feb 12, 2013 11:54 pm

Re: CCR1009 Rebooting

Tue Nov 29, 2022 6:44 pm

So the CCR1009-8G-1S-1S+ (7.6) that was effected by the reboots has been stable since yesterday around 1:15pm PST, I don't know if this is related or not but I noticed the ipv4 address of "36.156.28.131" attempting all types of connections with the router before I blocked it a short time later. The address is listed out of China and is on several blacklist for brute-force hacking attempts.
 
Jooos
just joined
Posts: 4
Joined: Tue Oct 08, 2013 2:06 am

Re: CCR1009 Rebooting

Tue Nov 29, 2022 7:18 pm

CCR1009-8G-1S-1S+ v7.6

Same problem here, random reboots with error "kernel failure in previous boot", upgraded from 7.1.1 to 7.6 but still the same issue.
My CCR was running for years with 1 PSU without any issue.
Last edited by Jooos on Tue Nov 29, 2022 8:24 pm, edited 1 time in total.
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Tue Nov 29, 2022 7:34 pm

So the CCR1009-8G-1S-1S+ (7.6) that was effected by the reboots has been stable since yesterday around 1:15pm PST, I don't know if this is related or not but I noticed the ipv4 address of "36.156.28.131" attempting all types of connections with the router before I blocked it a short time later. The address is listed out of China and is on several blacklist for brute-force hacking attempts.
I'm pretty sure this is what happen right now. Its very date-related, so it look like an attack that target Mikotik Routeur.

I added this IP to my blacklist too, just in case.
 
akschu
Frequent Visitor
Frequent Visitor
Posts: 57
Joined: Thu Mar 15, 2012 2:09 am

Re: CCR1009 Rebooting

Tue Nov 29, 2022 9:46 pm

I have a CCR1009-8G-1S here that I upgraded to 7.6 and immediately had reboot issues. After the 4th reboot in 2 hours I downgraded back to 6.49.7 and it's been stable since.
 
wispvt
Member Candidate
Member Candidate
Posts: 109
Joined: Tue May 10, 2011 4:20 pm

Re: CCR1009 Rebooting

Tue Nov 29, 2022 9:48 pm

It definately could be some sort of attack vector but the fact that it only affects tile version of routers running the exact same software version as other models makes it unlikely although not entirely impossible.
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Tue Nov 29, 2022 11:40 pm

I've sended two files (from two crash) to the support. Hope you have some news about that soon.
 
Ordghio
just joined
Posts: 5
Joined: Fri Oct 17, 2014 11:24 am

Re: CCR1009 Rebooting

Wed Nov 30, 2022 12:30 am

Same issue by one of my CCR1009-8G-1S-1S+, not funny at all
11 reboots for last 24 hours :shock:

Maybe it's some sort of attack, but i have other 1009/1036 routers with similar configurations (and latest stable firmware - 7.6) and no any issues
 
User avatar
JavierTF
just joined
Posts: 12
Joined: Thu Nov 19, 2015 12:15 pm
Location: Santa Cruz de Tenerife
Contact:

Re: CCR1009 Rebooting

Wed Nov 30, 2022 12:44 am

Hi all

We have exactly the same issue here with two CCR1009-8G-1S rigs. The first one was working for months with version 7.6 without any problem and suddenly, a few days ago it started to have reboots with the kernel failure message. We export it's configuration and reinstalled it on the seccond CCR (previusly upgraded to version 7.6) and immediately started experiencing the reboot issues. Try downgrading to version 7.5 and the same... I'm going to try downgrading to version 6 to see if something changes. From what I see in this forum, it seems to be a problem that is affecting devices with a Tilera processor... I'll update when I try version 6

Best regards
 
rememberme
just joined
Posts: 23
Joined: Fri Nov 13, 2015 10:13 pm
Location: Chicago, USA

Re: CCR1009 Rebooting

Wed Nov 30, 2022 6:06 am

I'm in the same boat. There was a CCR1009 running 7.3 or 7.4 and started crashing last week. It was replaced today with a CCR1016 on ROS v7.6. Still rebooting.

I think it might be related to NTP. What I did to make it kernel panic is the following: set the time to 00:00:00 then disable/enable NTP client. Router rebooted right away. Then I tried simply disabling/enabling NTP client - it crashed. Lastly just disabled NTP client - crashed in less than 10secs.
I will leave the NTP client disabled for now and will keep an eye on how it does without it.
 
julianho
just joined
Posts: 1
Joined: Wed Jan 06, 2021 4:31 am

Re: CCR1009 Rebooting

Wed Nov 30, 2022 8:53 am

The same problem occurred in the early morning of the 29th, Shanghai time zone. v7 versions of 1009 and 1036 had kernel errors, after downgrading to v6 version everything works fine.
BTW, all platforms such as mips, arm, and x86 had no issues, and it seems that there is only a problem with the tile.
 
User avatar
tslytsly
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Tue Oct 27, 2015 6:52 pm
Location: Nottingham
Contact:

Re: CCR1009 Rebooting

Wed Nov 30, 2022 9:36 am

Hi,
We have the same with a CCR1009-8g-1s-1s+
It's been installed and working for over 8 years.
Running 7.6 for a few weeks. then started rebooting on the 26th.
Cannot see a pattern, it can go for hours without a reboot, or reboot every 10 mins or so.
Swapped in a new CCR1009-8g-1s with config from RSC, it is doing the same thing.

I have a ticket open with support.

I have the original 1009 on my test bench, not connected to anything and it has been running over 10 hours with no failure.
So it seems to be some external factor.
 
User avatar
strods
MikroTik Support
MikroTik Support
Posts: 1616
Joined: Wed Jul 16, 2014 7:22 am
Location: Riga, Latvia

Re: CCR1009 Rebooting

Wed Nov 30, 2022 11:40 am

We have lately received multiple complaints regarding Kernel Failures while running RouterOS v7. First of all, please note that we can not ever be sure if multiple users are suffering from the same issue without seeing the supout file. We can talk much about concrete configuration and network structure, but supout file speaks louder than thousand words. Without supout files, it is just guessing.

However, it seems that we have managed to resolve a very popular Kernel Failure issue today and the fix is on the way. It will be included in RouterOS v7.7beta9 and later releases. If your router is rebooting, then try out this release and see how it goes. If issues persist, then send supout files to support@mikrotik.com.

Remember - if your router is rebooting due to a software issue, then support should know about it. That is why you must contact support. Nobody In the forum will be able to fix the software. And support will not be able to tell much (besides simple guessing) without supout file.
 
xavierbt
Frequent Visitor
Frequent Visitor
Posts: 57
Joined: Thu Jan 22, 2015 11:55 am

Re: CCR1009 Rebooting

Wed Nov 30, 2022 12:25 pm

I'm affected for tile router rebooting, too.

Model CCR1009-8G-1S-1S+
SW and FW 7.6
router was rebooted without proper shutdown, probably kernel failure
kernel failure in previous boot

Not sure when this begins but not many days ago. My router reboots a couple of times in a day, more or less.

:(
 
User avatar
tslytsly
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Tue Oct 27, 2015 6:52 pm
Location: Nottingham
Contact:

Re: CCR1009 Rebooting

Wed Nov 30, 2022 2:20 pm

We have lately received multiple complaints regarding Kernel Failures while running RouterOS v7. First of all, please note that we can not ever be sure if multiple users are suffering from the same issue without seeing the supout file. We can talk much about concrete configuration and network structure, but supout file speaks louder than thousand words. Without supout files, it is just guessing.

However, it seems that we have managed to resolve a very popular Kernel Failure issue today and the fix is on the way. It will be included in RouterOS v7.7beta9 and later releases. If your router is rebooting, then try out this release and see how it goes. If issues persist, then send supout files to support@mikrotik.com.

Remember - if your router is rebooting due to a software issue, then support should know about it. That is why you must contact support. Nobody In the forum will be able to fix the software. And support will not be able to tell much (besides simple guessing) without supout file.
Thank you for the update, I did log a support ticket as soon as we had the issue and provided supout files.
Can you give us an ETA on the fix? When will 7.6beta9 be released please?
 
User avatar
JavierTF
just joined
Posts: 12
Joined: Thu Nov 19, 2015 12:15 pm
Location: Santa Cruz de Tenerife
Contact:

Re: CCR1009 Rebooting

Wed Nov 30, 2022 2:26 pm

Hi again...

Finally I could not test with version 6 since we have configured eoip over Wireguard tunnels and there is no support for Wireguard in version 6. We have tested with a third CCR1009-8G-1S and everything is the same, we have tested version 7.7beta8 of RouterOS and the same... we have tried various power supplies with different power and current and the same...

What finally worked for me:
Finally we have formatted with NetInstall and version 7.6 one of the devices that were experiensing problems and we have completely reconfigured it by hand (omiting importing the rsc file with the configuration of the previous device). We already have all the main services operational (including VPN's and VLAN's) and the equipment has been working without problems for more than 2 hours (before it did not exceed 35 minutes without restarting) I still need to apply some dst_nat rules to map some ports and others minor rules but in principle, every critical services that we could suspect that could be generating the problem, are already active... I will try to activate the remaining rules y services one by one to see if something triggers the problem and we detect it
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Wed Nov 30, 2022 3:35 pm

Remember - if your router is rebooting due to a software issue, then support should know about it. That is why you must contact support. Nobody In the forum will be able to fix the software. And support will not be able to tell much (besides simple guessing) without supout file.
Yes, but I think it's a good idea to come here first and see if there is already a solution and have a discussion other too, it's nice to see you are not alone! :)


That said, I'm testing the fix in the nightly build as proposed by the support, it's 7.7-alpha255. Will report back if it's crashing again (here and on support).

No, I won't provide the file, if you want it, ask the support! :)
 
User avatar
tslytsly
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Tue Oct 27, 2015 6:52 pm
Location: Nottingham
Contact:

Re: CCR1009 Rebooting

Wed Nov 30, 2022 4:57 pm

I'm also testing the alpha version as provided by support.
Will report back results.
If you are having issues, make sure to open a support ticket, like the others have said, that is the only way to know if your issue is the same as the one they have a fix for.
 
tamisoft
just joined
Posts: 2
Joined: Tue Nov 29, 2022 4:53 pm

Re: CCR1009 Rebooting

Wed Nov 30, 2022 9:12 pm

The provided ROS7.7alpha255 nightly has fixed the issues for us. 7 hours no crashes. Thanks to the support's quick assist!
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Wed Nov 30, 2022 10:08 pm

The provided ROS7.7alpha255 nightly has fixed the issues for us. 7 hours no crashes. Thanks to the support's quick assist!
don't be too quick, I did more than 30 hours when I downgraded to 7.5 :P

But they said they found why, so it look like it will fix it.
 
w4rh0und
Member Candidate
Member Candidate
Posts: 107
Joined: Fri Oct 16, 2009 10:58 pm

Re: CCR1009 Rebooting

Wed Nov 30, 2022 11:36 pm

I am having this issue as well since a few days ago

We have been on version 7.4.1 since August, and now we started with the reboot.

I've raised a task with support.

routerboard: yes
model: CCR1009-8G-1S-1S+
serial-number: 4AB1041DBDBD
firmware-type: tilegx
factory-firmware: 3.18
current-firmware: 7.4.1
upgrade-firmware: 7.4.1
 
w4rh0und
Member Candidate
Member Candidate
Posts: 107
Joined: Fri Oct 16, 2009 10:58 pm

Re: CCR1009 Rebooting

Thu Dec 01, 2022 10:50 am

The provided ROS7.7alpha255 nightly has fixed the issues for us. 7 hours no crashes. Thanks to the support's quick assist!
Quick couple of questions:
I've received the file from support as well.

I just copy it on the router and reboot?

Do I also have to upgrade the firmware? ( /system/routerboard/upgrade)

Have you encountered any issues with this alpha build?
The routers are in production, so it is a big gamble

Thank you
 
User avatar
tslytsly
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Tue Oct 27, 2015 6:52 pm
Location: Nottingham
Contact:

Re: CCR1009 Rebooting

Thu Dec 01, 2022 11:10 am

The provided ROS7.7alpha255 nightly has fixed the issues for us. 7 hours no crashes. Thanks to the support's quick assist!
Quick couple of questions:
I've received the file from support as well.

I just copy it on the router and reboot?

Do I also have to upgrade the firmware? ( /system/routerboard/upgrade)

Have you encountered any issues with this alpha build?
The routers are in production, so it is a big gamble

Thank you
Hi,
I had to copy the file to the router then use the
/system/package/downgrade
function.
The router rebooted and then applied the package.
I have not updated the bootloader for now and it's been stable since.
So far no issues, but that is only on one router. so your mileage may vary.
 
User avatar
strods
MikroTik Support
MikroTik Support
Posts: 1616
Joined: Wed Jul 16, 2014 7:22 am
Location: Riga, Latvia

Re: CCR1009 Rebooting

Thu Dec 01, 2022 2:41 pm

The particular issue has been resolved in v7.7beta9. If you experience Kernel Failures even after applying the fix, please contact support@mikrotik.com and send a supout file.
 
User avatar
tslytsly
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Tue Oct 27, 2015 6:52 pm
Location: Nottingham
Contact:

Re: CCR1009 Rebooting

Thu Dec 01, 2022 3:08 pm

The particular issue has been resolved in v7.7beta9. If you experience Kernel Failures even after applying the fix, please contact support@mikrotik.com and send a supout file.
Thank you.
Was this the issue?
*) conntrack - improved system stability when processing SCTP connections on TILE
I have to explain to my boss what the issue was.

Thanks
 
User avatar
strods
MikroTik Support
MikroTik Support
Posts: 1616
Joined: Wed Jul 16, 2014 7:22 am
Location: Riga, Latvia

Re: CCR1009 Rebooting

Thu Dec 01, 2022 3:17 pm

Yes, that was the issue for this particular "sudden" increased number of TILE routers rebooting. The problem was introduced in RouterOS v7. Most likely affected routers simply did not process any SCTP traffic before. From where did this traffic appear in your networks? Hard to guess. Maybe some kind of a worldwide DDoS attack.
 
User avatar
tslytsly
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Tue Oct 27, 2015 6:52 pm
Location: Nottingham
Contact:

Re: CCR1009 Rebooting

Thu Dec 01, 2022 3:27 pm

Yes, that was the issue for this particular "sudden" increased number of TILE routers rebooting. The problem was introduced in RouterOS v7. Most likely affected routers simply did not process any SCTP traffic before. From where did this traffic appear in your networks? Hard to guess. Maybe some kind of a worldwide DDoS attack.
Thank you.

Would adding as RAW rule to drop SCTP work to prevent this if an update is not possible?

Thanks
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Thu Dec 01, 2022 3:49 pm

I can confirm 7.7-alpha255 (now Beta9) is stable on my end for about 24 hours now.
 
hex
just joined
Posts: 9
Joined: Wed Nov 10, 2010 4:32 am

Re: CCR1009 Rebooting

Thu Dec 01, 2022 4:27 pm

Yes, that was the issue for this particular "sudden" increased number of TILE routers rebooting. The problem was introduced in RouterOS v7. Most likely affected routers simply did not process any SCTP traffic before. From where did this traffic appear in your networks? Hard to guess. Maybe some kind of a worldwide DDoS attack.
Then maybe filtering SCTP is better mitigation then installing alpha quality software to production routers?
 
w4rh0und
Member Candidate
Member Candidate
Posts: 107
Joined: Fri Oct 16, 2009 10:58 pm

Re: CCR1009 Rebooting

Fri Dec 02, 2022 8:24 am

I've created a RAW sctp block rule
/ip firewall/raw/add action=drop chain=prerouting comment="Drop all SCTP" in-interface-list=WAN protocol=sctp
So far no reboot for the past 11 hours.

If nothing else changes I might delay the upgrade until 7.7 full is released
 
User avatar
tslytsly
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Tue Oct 27, 2015 6:52 pm
Location: Nottingham
Contact:

Re: CCR1009 Rebooting

Fri Dec 02, 2022 10:27 am

I've created a RAW sctp block rule
/ip firewall/raw/add action=drop chain=prerouting comment="Drop all SCTP" in-interface-list=WAN protocol=sctp
So far no reboot for the past 11 hours.

If nothing else changes I might delay the upgrade until 7.7 full is released
Me too.
I added this block rule to one of our other 1036 routers, still on 7.6, and it's not rebooted.
I also added a rule above it to add the IP address to an address list.
So far, only one address has been added after 10+ hits.
167.114.50.149
I looked it up, it's a European cloud hosting provider.
So its very low level and single source for me so far.
Suspect its someone doing initial scanning or scouting?
 
nka
newbie
Posts: 44
Joined: Tue Mar 22, 2011 7:48 pm
Location: Quebec, Canada

Re: CCR1009 Rebooting

Fri Dec 02, 2022 3:26 pm

7.7 - alpha255 has been very stable for me. Will run on this until 7.7 full release.
 
hex
just joined
Posts: 9
Joined: Wed Nov 10, 2010 4:32 am

Re: CCR1009 Rebooting

Fri Dec 02, 2022 6:20 pm

I've created a RAW sctp block rule
/ip firewall/raw/add action=drop chain=prerouting comment="Drop all SCTP" in-interface-list=WAN protocol=sctp
So far no reboot for the past 11 hours.

If nothing else changes I might delay the upgrade until 7.7 full is released
Me too.
I added this block rule to one of our other 1036 routers, still on 7.6, and it's not rebooted.
I also added a rule above it to add the IP address to an address list.
So far, only one address has been added after 10+ hits.
167.114.50.149
I looked it up, it's a European cloud hosting provider.
So its very low level and single source for me so far.
Suspect its someone doing initial scanning or scouting?
I have added logging now, let's see what comes out.
It's extremely strange that somebody is sending SCTP packets all over Internet.
I thought this bug must be time related, or somebody trying to attack.
 
w4rh0und
Member Candidate
Member Candidate
Posts: 107
Joined: Fri Oct 16, 2009 10:58 pm

Re: CCR1009 Rebooting

Fri Dec 02, 2022 6:59 pm

I've enabled logging yesterday as well, and all packets are received from 167.114.50.149 which is US
 
hex
just joined
Posts: 9
Joined: Wed Nov 10, 2010 4:32 am

Re: CCR1009 Rebooting

Fri Dec 02, 2022 7:24 pm

I received from same address.
167.114.50.149

Fun.
 
User avatar
tslytsly
Frequent Visitor
Frequent Visitor
Posts: 65
Joined: Tue Oct 27, 2015 6:52 pm
Location: Nottingham
Contact:

Re: CCR1009 Rebooting

Fri Dec 02, 2022 10:22 pm

I've enabled logging yesterday as well, and all packets are received from 167.114.50.149 which is US
NSA?
:lol:

I wonder if it's someone testing an exploit?
 
ofca
Member Candidate
Member Candidate
Posts: 228
Joined: Fri Aug 20, 2004 7:18 pm

Re: CCR1009 Rebooting

Sat Dec 03, 2022 12:54 am

whatever they're doing, one of our CCR1016s was hit by this; it started on last sunday. Started dumping SCTP traffic today, and turns out it originated at the same IP. 7.7b9 didn't crash. As one would expect, routers just forwarding traffic are not affected; only those doing some conntrack-related activities, like NAT, crash.

There's nearly no SCTP traffic inbound to our network. It's few packets per hour, and I observed those two IPs.
00:07:45.986661 IP 146.190.28.73.57723 > X.X.X.X.36444: sctp (1) [INIT] [init tag: 2621498272] [rwnd: 32768] [OS: 10] [MIS: 2048] [init TSN: 1176166205] 
        0x0000:  6cb3 1151 01a4 0022 83dd ce48 0800 4500  l..Q..."...H..E.
        0x0010:  0034 45de 0000 f784 e413 92be 1c49 XXXX  .4E..........I..
        0x0020:  XXXX e17b 8e5c 0000 0000 39cd cd80 0100  ...{.\....9.....
        0x0030:  0014 9c40 e3a0 0000 8000 000a 0800 461a  ...@..........F.
        0x0040:  df3d                                     .=
00:09:32.344132 IP 167.114.50.149.43472 > X.X.X.X.6120: sctp (1) [INIT] [init tag: 2953808970] [rwnd: 32768] [OS: 10] [MIS: 2048] [init TSN: 1176166205] 
        0x0000:  6cb3 1151 01a4 0022 83dd ce48 0800 4500  l..Q..."...H..E.
        0x0010:  0034 b37a 0000 f384 4ec1 a772 3295 XXXX  .4.z....N..r2...
        0x0020:  XXXX a9d0 17e8 0000 0000 2fa7 5573 0100  ........../.Us..
        0x0030:  0014 b00f 8c4a 0000 8000 000a 0800 461a  .....J........F.
        0x0040:  df3d                                     .=
 
faxxe
newbie
Posts: 39
Joined: Wed Dec 12, 2018 1:46 pm

Re: CCR1009 Rebooting

Sat Dec 03, 2022 8:06 am

I received from same address.
167.114.50.149

Fun.
Same packets arrived here to my CCR1009. Thanks god it was filtered in a raw rule but i got an log entry:
Not TCP protocol prerouting: in:ether1 out:(unknown 0), connection-state:invalid src-mac 00:01:5c:b5:dc:46, proto 132, 167.114.50.149->xxx.xxx.xxx.xxx, len 52 
I think there is an exploit being worked on

-faxxe
 
hex
just joined
Posts: 9
Joined: Wed Nov 10, 2010 4:32 am

Re: CCR1009 Rebooting

Mon Dec 05, 2022 1:35 am

I have written to OVH abuse team with link to this thread.
 
User avatar
yo1frenchtoast
just joined
Posts: 9
Joined: Thu Jul 09, 2015 3:48 pm
Location: Brittany, France

Re: CCR1009 Rebooting

Tue Dec 06, 2022 11:34 am

Here's my catches (GMT+1) :
 dec/02 12:20:20 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/02 12:20:20 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/02 17:48:59 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/02 17:48:59 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/03 09:53:23 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/03 09:53:23 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/03 14:27:02 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.148->86.215.xxx.y, len 52
 dec/03 14:27:02 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.148->86.215.xxx.y, len 52
 dec/03 15:45:53 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/03 15:45:53 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/04 00:33:44 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/04 00:33:44 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/04 00:33:57 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.148->86.215.xxx.y, len 52
 dec/04 00:33:57 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.148->86.215.xxx.y, len 52
 dec/04 03:54:16 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 146.190.28.73->86.215.xxx.y, len 52
 dec/04 03:54:16 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 146.190.28.73->86.215.xxx.y, len 52
 dec/04 06:23:49 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/04 06:23:49 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.149->86.215.xxx.y, len 52
 dec/04 10:34:59 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.148->86.215.xxx.y, len 52
 dec/04 10:34:59 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.148->86.215.xxx.y, len 52
 dec/04 14:38:42 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.148->86.215.xxx.y, len 52
 dec/04 14:38:42 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.148->86.215.xxx.y, len 52
 dec/04 18:25:08 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.148->86.215.xxx.y, len 52
 dec/04 18:25:08 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 167.114.50.148->86.215.xxx.y, len 52
 dec/04 23:21:05 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 146.190.28.73->86.215.xxx.y, len 52
 dec/04 23:21:05 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 146.190.28.73->86.215.xxx.y, len 52
 dec/05 23:17:46 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 146.190.28.73->86.215.xxx.y, len 52
 dec/05 23:17:46 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 146.190.28.73->86.215.xxx.y, len 52
 05:01:02 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 146.190.28.73->86.215.xxx.y, len 52
 05:01:02 firewall,info drop_sctp prerouting: in:vlan832 out:(unknown 0), connection-state:invalid src-mac 24:21:24:xx::yy::zz, proto 132, 146.190.28.73->86.215.xxx.y, len 52
Raw filter rule works perfectly fine for me
 
Ordghio
just joined
Posts: 5
Joined: Fri Oct 17, 2014 11:24 am

Re: CCR1009 Rebooting

Tue Dec 06, 2022 2:41 pm

My blacklist is the same:
146.190.28.73
167.114.50.148
167.114.50.149
 
User avatar
inteq
Member
Member
Posts: 402
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: CCR1009 Rebooting

Thu Dec 08, 2022 8:59 pm

Had a CCR1009 doing the same.
Weird thing is that is was doing the same even after I took it out of the rack and place it on my desk without internet.
A netinstall and the same config back and so far no reboots (not back in production tho). Mikrotik support stated they saw a problem from supout and I should try a beta.
Just replaced the router with a backup 1100AHx4 until they sort this stuff out in stable.
 
wispvt
Member Candidate
Member Candidate
Posts: 109
Joined: Tue May 10, 2011 4:20 pm

Re: CCR1009 Rebooting

Fri Dec 09, 2022 11:40 pm

Just had a CCR2004-1G-12S+2XS start doing the same thing. Filtered the SCTP traffic in RAW but to no avail so it's something else. Downgraded the router to 6.48.6 for the fix. So if it is an attack it has evolved to now include arm64 processors and possibly different protocol.
 
emisionlocal
just joined
Posts: 1
Joined: Wed Jan 16, 2019 2:21 pm

Re: CCR1009 Rebooting

Tue Dec 13, 2022 6:01 pm

Hello everyone, I have read all the threads of answers, I have been experiencing the same problem for several weeks, CCR-1009 8G, so far I have tried everything you say.

Let's hope there's luck, right now version:
7.7rc1

And the RAW port 132 rule added.

Let's see if we're lucky, I'll be attentive to this thread, since my reboots were continuous, the maximum without rebooting that I've been able to do was 20h and then at the end every 20 minutes, it seems that the problem once factory reset, little by little little, it is increasing.
 
jfarre20
just joined
Posts: 4
Joined: Fri Sep 09, 2022 4:32 am

Re: CCR1009 Rebooting

Sun Dec 18, 2022 1:06 am

My CCR-1036 has spontaneously started crashing. Today is the 2nd time this happened. Previously it crashed over and over for about 20 mins on Dec 8.

Today, I disabled the watchdog and watched as it slowly started dropping pings, winbox started getting slow, and then it hardlocked. Right before it went, I saw that the cpu was pegged at 30%, some cores maxed. I think its the same attack. Have the raw filter with the SCTP thing, but it still reboots. This went on for about 20 minutes - where it crashed a few more times.

But seems the attack has stopped now, Its back online and seemingly stable.

Running ROS 7.7RC1
 
User avatar
inteq
Member
Member
Posts: 402
Joined: Wed Feb 25, 2015 8:15 pm
Location: Romania

Re: CCR1009 Rebooting

Sun Dec 18, 2022 9:50 am

Found several other 1009s doing this.
Mikrotik support sent me an alpha version that might fix this.
Try routeros-7.7alpha255-tile-2.npk from https://box.mikrotik.com/f/61b327d9fb854902bd58/?dl=1
Or ask support.
Installing now and will see how it goes.

later edit: 6 days now and no more reboots.
No clue if the "attacks" stopped or this version fixed the problem.

Who is online

Users browsing this forum: ivicask and 15 guests