any reboot till now on both routers ?FYI
I am still runing 7.1rc3 on this router and set tcp established timeout to 1hr
That's all i changed.. currently 33d and 15hr uptime
We have still no reboot after 39d 13h still running 7.1rc3 only thing we changed is the tcp established timeout.
We also updates a second CCR2004 it is running 7.1rc4 for 6d 9h now.
We dit not set the queue/pfifo setting.
7.1rc3 rebooted after 42 days.any reboot till now on both routers ?
FYI
We have still no reboot after 39d 13h still running 7.1rc3 only thing we changed is the tcp established timeout.
We also updates a second CCR2004 it is running 7.1rc4 for 6d 9h now.
We dit not set the queue/pfifo setting.
If youre ok with random reboots and can tolerate that. Go ahead. Otherwise best advice is to wait until r7 is a little further along. Imo.We recently bought two of these for use as BGP routers and I just discovered this topic. We haven't implemented them yet and now I'm a bit anxious about it. We might implement one of them just to see how it behaves.
What RouterOS version do you use? Have you tried to set proposed enhancement in previous posts?We are also in random reboots club. Device rebooted after 16 days 21 hours 34 minutes 24 seconds.
We have no software queue, only HW queues. Simple NAT router with one 10g trunk interface. Nothing fancy. Angry as hell.
Sergejs have you read this thread? None has solved reboots problem whatever ros version used with or without tricksWhat RouterOS version do you use? Have you tried to set proposed enhancement in previous posts?
Yes, I did it, as well as I worked with tickets that are related to the problem, currently there are very less 2004 that are rebooting (actually it helped for biggest portion of devices) and we are desperately looking for the best possible ways how to fix it.Sergejs have you read this thread? None has solved reboots problem whatever ros version used with or without tricksWhat RouterOS version do you use? Have you tried to set proposed enhancement in previous posts?![]()
Ticket : SUP-51953Yes, I did it, as well as I worked with tickets that are related to the problem, currently there are very less 2004 that are rebooting (actually it helped for biggest portion of devices) and we are desperately looking for the best possible ways how to fix it.
Sergejs have you read this thread? None has solved reboots problem whatever ros version used with or without tricks![]()
If you want to help us with it, it would be great you can apply settings from the previous posts, make sure you have 6.49 for RouterOS version and firmware, send us support output file in case of any issue.
Yes, I did above 2 queue changes, how to do same for ppoe ones ?Have you tried 6.49 and
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default
on all queues as well dynamic PPPoE ones.
As I do not see such settings in your support output file.
We have removed all CCR2004 less one from our network, actually I am on 6.48.4 and I opened several tickets without success. Reboot random some day ore some weeks, Tomorrow morning I 'll upgrade to 6.49 but I am quite sure it's uselessYes, I did it, as well as I worked with tickets that are related to the problem, currently there are very less 2004 that are rebooting (actually it helped for biggest portion of devices) and we are desperately looking for the best possible ways how to fix it.
If you want to help us with it, it would be great you can apply settings from the previous posts, make sure you have 6.49 for RouterOS version and firmware, send us support output file in case of any issue.
Provide us with support output, you can mention ticket number here. We will look into it.We have removed all CCR2004 less one from our network, actually I am on 6.48.4 and I opened several tickets without success. Reboot random some day ore some weeks, Tomorrow morning I 'll upgrade to 6.49 but I am quite sure it's uselessYes, I did it, as well as I worked with tickets that are related to the problem, currently there are very less 2004 that are rebooting (actually it helped for biggest portion of devices) and we are desperately looking for the best possible ways how to fix it.
If you want to help us with it, it would be great you can apply settings from the previous posts, make sure you have 6.49 for RouterOS version and firmware, send us support output file in case of any issue.
I have already did it a week ago [SUP-64102]Provide us with support output, you can mention ticket number here. We will look into it.
We have removed all CCR2004 less one from our network, actually I am on 6.48.4 and I opened several tickets without success. Reboot random some day ore some weeks, Tomorrow morning I 'll upgrade to 6.49 but I am quite sure it's useless
I never had reboot on 2004, but packet loss.
I use the latest lonterm+latest routerboot and it works flawlessy.
we have a policy that reboot each router every 14 days at night to avoid issues.
0 issues till now.
on the 7.1rc4 it was very unstable on some test units, rebooted on the bench, upgraded to rc6 and it doesnt reboot.
It definitely is designed for v7 and runs in compatibility mode in v6, which is why it cannot use all of the RAM on v6. And, mikeeg02 is finding that the CCR2004 seems to not crash on v7 at all, suggesting that the issue is something related to the kernel version or this compatibility mode.Do you think this device was build for V7 and that V6 is running in compatibility mode (32bit) on a 64bit CPU?
Changing the MTU and L2 MTU to 9216 that all interfaces you run as LDP interface.Tried 6.49 - no luck, reboot after 8 days.
Can't test 7.1rc4 - VPLS does not pass MTU > 1500, looks like it's a bug.
forgot itMikrotik, how about money back?? No resolution - no customer.
I dont need this trash anymore.
Does this mean that some fix for this has been applied in 6.49.1 and 7.1rc7 and that fix is not in 6.48.5?In case any of you are experiencing issue with random reboots on 2004 and 6.49.1/7.1rc7
Please, contact us (support@mikrotik.com) with the information and attached support output files.
It definitely is designed for v7 and runs in compatibility mode in v6, which is why it cannot use all of the RAM on v6. And, mikeeg02 is finding that the CCR2004 seems to not crash on v7 at all, suggesting that the issue is something related to the kernel version or this compatibility mode.
I suspect they probably won't come up with a fix for this on v6 and will ask people to upgrade to v7 to get rid of the issue. If the fix is some change that was introduced to the kernel in the past decade, it may be easier to just upgrade the kernel to fix it instead of trying to track down what change made in the past 10 years fixed it and apply it to v6.
IMHO I agree the best path is for Mikrotik to focus all resources on getting 7.1.x on par with 6.49.1 and not do anything more on 6.x or add any new features besides any blocking an upgrade from 6.xAgree 100%I suspect they probably won't come up with a fix for this on v6 and will ask people to upgrade to v7 to get rid of the issue. If the fix is some change that was introduced to the kernel in the past decade, it may be easier to just upgrade the kernel to fix it instead of trying to track down what change made in the past 10 years fixed it and apply it to v6.
don't get fooled, they are suggesting same changes from 6.48, still issue remains same.We applied the suggested Ethernet hardware queues changes to 6.49 and the router still rebooted.... We are now running 6.49.1 with the suggested fix and will monitor it over the coming days.
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default
I've sent multiple support output files now, in SUP-65050, but every time i just get the answer "The supout rif file does not contain any hardware or software errors, the device was rebooted by watchdog timer", which is exactly the issue I'm trying to explain. I have four of these devices, and every single one of them reboots randomly with no pattern what so ever.In case any of you are experiencing issue with random reboots on 2004 and 6.49.1/7.1rc7
Please, contact us (support@mikrotik.com) with the information and attached support output files.
That's what i see here too. No autogenetrated supout.rif on reboot."The supout rif file does not contain any hardware or software errors, the device was rebooted by watchdog timer",
[admin@CCR2004] > system watchdog export verbose
# dec/07/2021 09:22:08 by RouterOS 6.49.1
# software id = MFVV-BXU3
#
# model = CCR2004-1G-12S+2XS
# serial number = D4F00C0EB777
/system watchdog
set auto-send-supout=no automatic-supout=yes ping-start-after-boot=5m ping-timeout=1m watch-address=none watchdog-timer=yes
[admin@CCR2004] >
Are you new here? Mikrotik had a lot of fuckups before. Everyone has known for a long time - MT routers cannot be used in enterprise. Its only for SOHO with low-speed wan. Its too bugged and have very poor support.Something is obviously rotten on the CCR2004 platform, and i don't understand why this issue hasn't been fixed for more than a year.
As of now i have no idea whether i need to buy other CCR's or to completely switch to another brand, this is unacceptable, and of course noticeable for our customers.
Everyone has known for a long time - MT routers cannot be used in enterprise. Its only for SOHO with low-speed wan. Its too bugged and have very poor support.
It depends on use cases:What's the alternative? The equivalent Cisco would cost 100 times as much.CCR2004 trash hardware not usable in a professional network.
Even refurbished hardware costs 10x what these do, and the stuff you've suggested consumes a ludicrous amount of power (by comparison) - datacenter power ain't cheap!It depends on use cases:
What's the alternative? The equivalent Cisco would cost 100 times as much.
> BGP/OSPF with VRF: i will definetly go with ARISTA ( example: ARISTA DCS-7050QX-32S )
> 10G NAT: hard question ( example: Cisco 6504E + 2x Sup2T; Juniper MX240/MX480; HPE HSR6804 )
> 1k PPPOE BRAS - no idea
You can find companies who offer used/refurbished hardware with support/spare-parts services.
Shhh... Don't give them ideas...Hell, look at the CRS2116 - that box is basically everything I could've asked for in a $1000 router, MT could've charged almost 2x as much and I'd still have a preorder with our distributor.
Reboot on 6.48.4 after 76 days uptime (was latest available firmware at the time, supposedly had 'fixes' for this type of issue.) SUP-68916In case any of you are experiencing issue with random reboots on 2004 and 6.49.1/7.1rc7
Please, contact us (support@mikrotik.com) with the information and attached support output files.
I have 19 2004s running 7.1.x - no reboots.*I have been running 7.1.1 on a device that used to reboot every few days and it has been far more stable.... (no reboots since upgrade)
It depends on use cases:
What's the alternative? The equivalent Cisco would cost 100 times as much.
> BGP/OSPF with VRF: i will definetly go with ARISTA ( example: ARISTA DCS-7050QX-32S )
> 10G NAT: hard question ( example: Cisco 6504E + 2x Sup2T; Juniper MX240/MX480; HPE HSR6804 )
> 1k PPPOE BRAS - no idea
You can find companies who offer used/refurbished hardware with support/spare-parts services.
We couldent operate our network with Juniper MX204s and still make money. We have 18 main sites just for the main site and 300 subnodes running CRS317.Coming back to the main point - now that ROSv7 is starting to work towards stability with new feature sets, it's hard to ignore the ROI of something like a CCR2004 vs. ASR/MX or Arista gear
IPANetEngineer
Have to disagree, I've put MIkroTik CCRs into plenty of large enterprises for critical roles.
In one specific example, we put 4 MikroTik routers into the flagship data center of a 19 billion dollar publicly traded company
This does not mean anything other than that you like to take risks (and you're just lucky).
Yes, you can build a "server" on Core i3 and chipset raid array on WD Green, but it will not become a real server from this.
Anyone seen an issue with the CCR2004 on ROS V6.49.1 where there is packet loss each time BGP needs to converge its routes?
If the routes from the remote peer are stable, then there is no packet loss across any interfaces.
As soon as the CCR2004 needs to converge received or withdrawn routes, CPU goes up and packet loss appears on all interfaces traffic passes through. Even running a Timeout of 100ms over a point to point IP shows packet loss.
Are there any work arounds?
No, still weekly reboots on ccr2004 with stable v7.1.1Other users reports that problem is solved with ros 7.1.x?
thank you
Thank you I hope to have more positive reportsNo, still weekly reboots on ccr2004 with stable v7.1.1Other users reports that problem is solved with ros 7.1.x?
thank you
Thank you I hope to have more positive reports
No, still weekly reboots on ccr2004 with stable v7.1.1
Migrating to V7.1 resolved this for future FYIAnyone seen an issue with the CCR2004 on ROS V6.49.1 where there is packet loss each time BGP needs to converge its routes?
If the routes from the remote peer are stable, then there is no packet loss across any interfaces.
As soon as the CCR2004 needs to converge received or withdrawn routes, CPU goes up and packet loss appears on all interfaces traffic passes through. Even running a Timeout of 100ms over a point to point IP shows packet loss.
Are there any work arounds?
I experienced the same in Teraco data centre peering with NAP Africa, so my workaround was to make the 2004 the PPPoE AC and put a 1016 in place as edge/BGP/FW device
Are you using this router as ppp server ?
Thank you I hope to have more positive reports
Im very curious to understand what things are causing this.
We run pure ospf and bgp with multiple full feeds. We don't have any reboots but had loads before.
/M
I have no PPP users and no reboots.Are you using this router as ppp server ?
I am 200+ ppp clients.
.
For test, I only one ppp client on this ccr, no reboot for 30 days, then i gave full load of 200+ users, and it's started to reboot again. 7 days to 15 days gap.
Ros7?We have about 40 CCR2004s in production. Only two of them are rebooting regularily. Plus a third that has rebooted only once. Still, we can't determine what is making these two reboot. They aren't the most loaded device in the network or the most articulated configuration.
All they do is:
- an handful VRRPs
- BGP, announcing a couple handfuls of routes to reflectors, and receiving the default.
- basic firewall, with a dozen ACLs
load is consistently below 5%, average 1%. Aggregate traffic is in the 100Mb/s range.
Ironically enough these two are the most critical devices in the network. We went with the 2004s after having no issues with the rest of them.
Even refurbished hardware costs 10x what these do, and the stuff you've suggested consumes a ludicrous amount of power (by comparison) - datacenter power ain't cheap!
It depends on use cases:
> BGP/OSPF with VRF: i will definetly go with ARISTA ( example: ARISTA DCS-7050QX-32S )
> 10G NAT: hard question ( example: Cisco 6504E + 2x Sup2T; Juniper MX240/MX480; HPE HSR6804 )
> 1k PPPOE BRAS - no idea
You can find companies who offer used/refurbished hardware with support/spare-parts services.
The closest thing you could find in a somewhat comparable pricepoint would be running VyOS or TNSR on a Xeon D-15xx/21xx network appliance box from Supermicro or something along those lines; even then, if you want support, VyOS and TNSR licenses are half the cost of a 2004 annually all by themselves.
There is pretty much nothing else on the market that can handle the same level of throughput, at a comparable pricepoint, with a comparable feature set - even used. The closest you can get with used gear requires something that consumes 200-300W+ and still costs more to buy, to say nothing of the 3rd-party hardware maintenance contract.
I've spent the last year or so trying to find something that meets these requirements, and it just doesn't exist. The major vendors are all moving to running on top of x86 virtualization appliances for the lower-end devices, with prohibitively high licensing costs, so they've no interest in making hardware to fit the bill. You could maybe consider the EdgeRouter Infinity, but, well, that's got its own problems.
I may have missed something - if so, I'm all ears - but I suspect most of the reason there are so many annoyed/upset/angry people in this thread is because there are very few other options than these devices for a lot of their intended use-cases. That's probably a significant part of the reason why MikroTik made them in the first place - there is a massive market vacuum for "low-to-medium-cost router with multiple 10G ports and capacity to handle full routing tables / etc".
Hell, look at the CRS2116 - that box is basically everything I could've asked for in a $1000 router, MT could've charged almost 2x as much and I'd still have a preorder with our distributor.
Which model specifically ? Compared to ccr2004 or ccr216/ccr1036 ?
Even refurbished hardware costs 10x what these do, and the stuff you've suggested consumes a ludicrous amount of power (by comparison) - datacenter power ain't cheap!
The closest thing you could find in a somewhat comparable pricepoint would be running VyOS or TNSR on a Xeon D-15xx/21xx network appliance box from Supermicro or something along those lines; even then, if you want support, VyOS and TNSR licenses are half the cost of a 2004 annually all by themselves.
There is pretty much nothing else on the market that can handle the same level of throughput, at a comparable pricepoint, with a comparable feature set - even used. The closest you can get with used gear requires something that consumes 200-300W+ and still costs more to buy, to say nothing of the 3rd-party hardware maintenance contract.
I've spent the last year or so trying to find something that meets these requirements, and it just doesn't exist. The major vendors are all moving to running on top of x86 virtualization appliances for the lower-end devices, with prohibitively high licensing costs, so they've no interest in making hardware to fit the bill. You could maybe consider the EdgeRouter Infinity, but, well, that's got its own problems.
I may have missed something - if so, I'm all ears - but I suspect most of the reason there are so many annoyed/upset/angry people in this thread is because there are very few other options than these devices for a lot of their intended use-cases. That's probably a significant part of the reason why MikroTik made them in the first place - there is a massive market vacuum for "low-to-medium-cost router with multiple 10G ports and capacity to handle full routing tables / etc".
Hell, look at the CRS2116 - that box is basically everything I could've asked for in a $1000 router, MT could've charged almost 2x as much and I'd still have a preorder with our distributor.
You can go with "Raisecom" ( chinese vendor ) - cost-efficient solution, i've heard good feedbacks from other operators.
Everbody I know with this issue has had it gone entirely since 7.1.1, except for you. Probably there are two reasons for the reboots, one is fixed by 7.1.1 and the other is possibly configuration specific. If you share your config for the device, we might be able to identify the issue.Any update guys ?
Does anyone getting reboots with vi stable ?
For me, it's still rebooting with v7.1.1
If you are doing VLAN, routing, basic firewall, you can probably use 7.1.1 on them to fix the reboots.On two CCR2004 we have experimented two watchdog reboot, all have 6.49.2,
No, for your information, 7.1.1 also does not fix reboot issue for ccr2004 but also introduces reboots on other ccr like 1009.If you are doing VLAN, routing, basic firewall, you can probably use 7.1.1 on them to fix the reboots.On two CCR2004 we have experimented two watchdog reboot, all have 6.49.2,
Count me in too. We have seen reboots on 7.1.1 too, but we're not using 7.x in production.Everbody I know with this issue has had it gone entirely since 7.1.1, except for you. Probably there are two reasons for the reboots, one is fixed by 7.1.1 and the other is possibly configuration specific. If you share your config for the device, we might be able to identify the issue.Any update guys ?
Does anyone getting reboots with vi stable ?
For me, it's still rebooting with v7.1.1
Yes, issue is not solved at all.Count me in too. We have seen reboots on 7.1.1 too, but we're not using 7.x in production.
Everbody I know with this issue has had it gone entirely since 7.1.1, except for you. Probably there are two reasons for the reboots, one is fixed by 7.1.1 and the other is possibly configuration specific. If you share your config for the device, we might be able to identify the issue.
My best scenario is with V7, I only use it here doing ospf going from 3 to 4G, I use a reboot script that over time I increase a few days, so since I made the change I'm validating it, today I reboot every 15 days at 03:30 AM. I didn't have the courage to leave 2004 straight without this script.I have another 2004, we already put the V7 on it, the only problem presented was the IPv6 did not close adjacent, as behind it there is a BRAS for pppoe, I'm thinking of making a tunnel and taking IPv6 to the edges, the only solution I found to use to 2004 with the V7. Next week I'm going to change this 2004 and put the other one with the V7. Let us pray!!!
If its just doing ospf you may consider 7.1rc4. Been pretty stable for mine as a firewall. 30 days tops with any 6 release. There's a slight configuration change but it seems good.
Same ros version and similar config i have reboot around every 50/60 daysRunning 6.48.4 RouterOS and firmware on all three units, no reboots yet since each was installed... but definitely watching for a reboot to occur.
Uptime: 37d 12:50:20
Uptime: 27d 12:31:48
Uptime: 12d 16:13:14
These wound up rebooting on 6.48.4. Have since upgraded to 6.49.2 per Mikrotik support recommendation, at 55d uptime for all three devices. However am not confident that the reboot issue is resolved on this code, waiting for it to hit again. Once ROS 7.x is stable enough with OSPF etc will try that release, but probably will have to wait until mid 2022 at the earliest for the code base to be stable enough.Same ros version and similar config i have reboot around every 50/60 days
Can you try latest test build v7.2rc3 ?system,error,critical router was rebooted without proper shutdown by watchdog timer
MikroTik RouterOS 6.49.2
version: 6.49.2 (stable)
build-time: Dec/03/2021 14:53:53
factory-software: 6.47.8
free-memory: 1715.1MiB
total-memory: 1792.0MiB
cpu: ARMv7
cpu-count: 4
cpu-frequency: 1700MHz
cpu-load: 0%
free-hdd-space: 91.5MiB
total-hdd-space: 129.0MiB
architecture-name: arm64
board-name: CCR2004-1G-12S+2XS
platform: MikroTik
config - main router, ppp, firewall, nat, 1 vrrp, 5 vlans on bridge vlan-filtering=yes, dhcp-server, 1 simple queue 1 ip/32, 5 active sfp+ ports and 1 wan
347DAYS?We are still experiencing reboots. Just sent in a supout.
CCR2004_uptime.png
I have 40 max 50 days uptime before reboot it's early to call for a victory22 Days Uptime on private beta build which includes fixes for ccr2004.
This is maximum uptime for me.
I think they fixed issue.
These changes will be added to next rc release.
it's early to call for a victorybut I hope you are true
It's your maximum uptime, I never gone that far.I have 40 max 50 days uptime before reboot it's early to call for a victory22 Days Uptime on private beta build which includes fixes for ccr2004.
This is maximum uptime for me.
I think they fixed issue.
These changes will be added to next rc release.but I hope you are true
@vasa85 what version of ROS are you using?We are still experiencing reboots. Just sent in a supout.
CCR2004_uptime.png
for version 6.49.1 (stable)
sergejs b. indicates the following
There are two potential fixes for the particular problem.
1) Soft fix to modify queues,
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default
2) There are few important fixes in v7 regarding that matter, we can provide you with 7.3beta packages, if you have an option to use beta packages on your device.
Ah yes "long term support".
As v7.x is a little different than v6, currently fix backporting might take unreasonable amount of time, and it should not be expected at v6 at the particular moment.
Please let us know if you are aware of any issues with current 7.x version on your setups.Ah yes "long term support".
As v7.x is a little different than v6, currently fix backporting might take unreasonable amount of time, and it should not be expected at v6 at the particular moment.
So correct me if I am wrong. We're left to choose if we want to keep rebooting our production routers every week, or upgrade to a beta quality release with missing/incomplete features and no upgrade path from 6 that doesn't end into to a broken configuration?
Just to name an handful that comes to mind:
Please let us know if you are aware of any issues with current 7.x version on your setups.
@sergejs, as TheNetworkBerg indicates in the following post: viewtopic.php?t=184778#p924179, BGP/MPLS Layer 3 VPN is not working correctly in ROSV7.2. This situation prevents us from using this version in our CCR2004.Please let us know if you are aware of any issues with current 7.x version on your setups.
Ah yes "long term support".
So correct me if I am wrong. We're left to choose if we want to keep rebooting our production routers every week, or upgrade to a beta quality release with missing/incomplete features and no upgrade path from 6 that doesn't end into to a broken configuration?
Yes, I can confirm it.I know that it's early but someone else can report that the issue is solved with 7.2?
Thank
It's totally resolved after 7.3 update.Hi all,
so, this issue, CCR2004 rebooting, still not solved?
Oh very very nice!!!Hi all,
so, this issue, CCR2004 rebooting, still not solved?
Like that issue?But some users having issue in BGP.
This is for alle CCR2004?It's totally resolved after 7.3 update.Hi all,
so, this issue, CCR2004 rebooting, still not solved?
Is this 100% info that reboot is resolved?It's totally resolved after 7.3 update.Hi all,
so, this issue, CCR2004 rebooting, still not solved?
But some users having issue in BGP.
For us v7 is a no go as we cannot use ipipv6 tunnels with ospfv3 -- SUP-70226 open since Dec/21; no fix on the horizon so far.Please let us know if you are aware of any issues with current 7.x version on your setups.
Yes! Still has a problem on 7.5 7.6 and 7.7 version, but I have no BGP, I'm using this model as regular main office router (vlan, gateway, firewall, dhcp, capsman and vpn server). It has not much load, but reboot one time a week or five times a day. On 7.1.5 or 7.2 I don't have this issue.Anyone still seeing same reboots with BGP on v7.5 or 7.6
Thanks, it seems my issue was actually to do with the local winbox session folder on my PC. Deleting the sessions folder under Winbox>Tools and clearing my winbox cache fixed the issue.We have several CCR2004 in lab and in production, running 7.8, 7.9 and 7.10rc, no issues with winbox.
No, only the CCR2004-1G-12S+2XS
This is for alle CCR2004?
CCR2004-1G-12S+2XS
CCR2004-16G-2S+
2023 is almost over and the problem still persists, is any one test and get good results with v7.12.1 ?We just had a 2004 have all interfaces go down and up again. Also seems OSPF dident fully recover, so in the end we rebooted it.
/Mikael