We have mainly packet loss pinging from the remote end to ccr2004 gigabit interface, the problem probably occurs when the upstream traffic from the 10G->1G fullfill the ethernet..
No when the giga interface receive data from 10 giga interface: the slower port sometimes has to keep in pause the faster one before to receive new packets because the buffer is full or not have the buffer.We have mainly packet loss pinging from the remote end to ccr2004 gigabit interface, the problem probably occurs when the upstream traffic from the 10G->1G fullfill the ethernet..
So does the packet loss only occur when the 1G interface is full?
The first think that I have tried before to write here and to open a support ticket.Have you tried changing the interface queues for both ports from hardware-only to ethernet-default ?
You need buffering when mixing interface speeds on any router and i'm not sure what the default buffer capabilities are for the CCR2004.
I would at least try a few different queue types for the interfaces to see if performance improves.
But You did a downgrade! Take a look at the performance figures:Sincerely I am very disappointed for CCR2004's performance.
We replaced a CCR1036 + 10G switch (with trunks) to a CCR2004 with directly put in DACs cables 10G.
Well, I can't speak for the usage model on You network - but if it was taxing one CCR1036, I find remarkable that one CCR2004 can barely cope with the load. I mean, take a look at the results for 512 bytes and 25 ip rules: it's about 1/3 of the 1036 speed!Hello Paternot, I have read carefully the performance figures, but sincerely I expected a lot more.
I did not expect a 100% cpu load !!!
However I am planning to replace the 1036 witha 1072 because I need the pure 10G ports for each link.
I have some fear to get a 1072 that has issues as in the threads about "watchdog and 1072".
I use conntrack, I use Fasttrack, it is one of my border router.
it not workIf I was you I would give shaping just below 1Gbps a shot and see if you still get the packet loss.
I don't use 1G Ethernet modules in productions so I don't have too much skin in the game, goodluck.
Can I have an official answer about this problem by mickrotik support? I am pinging every day for ticket [SUP-30387] wihtout any answer :-(
You are loosing packets if you check on SFP28 you 'll find zero tx drops.Hello
I got 1649 tx drop on the interface with an RJ01 with AIRFIBER24 from UBNT.
it 4 days and 4 hours.
it is a 500mbit link.
no issues from the customers.
maybe the issue you are describing?
The interfaces are routed between each other, so no mixed speeds in the same bridge.
So you think the CCR2004 is causing tx drops on a completely separate device (the 309)? I don't think that's possible.I even put a CRS309 in between my ELAN and my CCR2004 and the tx drops moved to the interface going to the ELAN on the CRS309 and stoped on my cloud core. … It's odd that the problem moved down to the interface on my CRS309 instead of staying on my CCR2004.
I've also problem with tx drops on switch (CRS317). Never had this problem before with RB4011. Tx drop are only on interfaces connected to CCR2004. Tested "overloading" ports connected to server with CHR -> not drop, no issue.So you think the CCR2004 is causing tx drops on a completely separate device (the 309)? I don't think that's possible.I even put a CRS309 in between my ELAN and my CCR2004 and the tx drops moved to the interface going to the ELAN on the CRS309 and stoped on my cloud core. … It's odd that the problem moved down to the interface on my CRS309 instead of staying on my CCR2004.
In the case of connecting the 10G interface to the 1G interface and additionally overloading the 1G link with the test, the appearance of Tx-Drop is a rather natural situation.I've also problem with tx drops on switch (CRS317). Never had this problem before with RB4011. Tx drop are only on interfaces connected to CCR2004. Tested "overloading" ports connected to server with CHR -> not drop, no issue.
Give it up it's normal from mikrotik the lack of response when they can't resolve the problem.Hello,
I appreciate this post but do not appreciate the lack of response from MT on this. I need these CCR2004 routers but am NOT ordering them from MT due to this issue being unresolved.
Again, thanks for the posts!
Stop this nonsense. You are just polluting the topic, and this helps no one - far from it.And another day. 1 month 4 days.
You might be squeaking on your own - I unsubscribed because of the noise.If no one says anything then nothing will get done. We don't deserve to be left in the dark. Regardless of what's going on. The squeaky wheel gets the grease. I'm not going to stop squeaking until I get some grease.
Do a constructive squeaking, don't be this person. Post tests, ideas, suggestions. One post a day, just saying "I'm a victim, Mikrotik doesn't love me" don't help anyone - and will not solve the problem faster either.If no one says anything then nothing will get done. We don't deserve to be left in the dark. Regardless of what's going on. The squeaky wheel gets the grease. I'm not going to stop squeaking until I get some grease.
Hi,Could be interesting
6.49 released.
switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device;
viewtopic.php?f=21&t=172259&sid=b361e20 ... ddcaa12707
It's a beta, so test it out first.
6.49beta11 did not fix packetloss but interface instability. BUT beware - all ports negotiate in full 10G even if SFP is 1G. Works kind of until it becomes unstable causing ospf to flap.Could be interesting
6.49 released.
switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device;
viewtopic.php?f=21&t=172259&sid=b361e20 ... ddcaa12707
It's a beta, so test it out first.
I got this same message from tech support to try this exact same thing. It actually made the packet loss worse from 1 to 2 packets out of 1000 to 8 to 10 packets out of 10000 on any interface I put the rule on so yeah. Also did not fix tx drop errors.6.49beta11 did not fix packetloss but interface instability. BUT beware - all ports negotiate in full 10G even if SFP is 1G. Works kind of until it becomes unstable causing ospf to flap.Could be interesting
6.49 released.
switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device;
viewtopic.php?f=21&t=172259&sid=b361e20 ... ddcaa12707
It's a beta, so test it out first.
Also got this which Im testing on one box right now. Kind of defeats the purpose on 10G interfaces but very fast initial testing seems promising and mostly we are below 1 gig so its peferable to packetloss.
Regarding packetloss apply these settings on SFP+ ports which report tx-drop
/queue type
add kind=pcq name=queue1 pcq-burst-rate=990M pcq-burst-time=20s \
pcq-limit=2000KiB pcq-rate=980M
/queue interface
set sfp-sfpplus1 queue=queue1
/Mikael
Same here (though what we were sent didn't look like a full command). We're having packet loss when using only two 10G interfaces, but only when routing production-like traffic (i.e. lots of src and dst addresses). A single iperf runs fine.I got this same message from tech support to try this exact same thing. It actually made the packet loss worse from 1 to 2 packets out of 1000 to 8 to 10 packets out of 10000 on any interface I put the rule on so yeah. Also did not fix tx drop errors.6.49beta11 did not fix packetloss but interface instability. BUT beware - all ports negotiate in full 10G even if SFP is 1G. Works kind of until it becomes unstable causing ospf to flap.
…and now same here with the 6.49beta11 "SFP at 1G is not stable" bug.Same here (though what we were sent didn't look like a full command). We're having packet loss when using only two 10G interfaces, but only when routing production-like traffic (i.e. lots of src and dst addresses). A single iperf runs fine.
Support ticket to MikroTik updated with full details. We've been working with them for a month now — hopefully the test cases we've got are helping.
For us the 1Gs report 10G. Same for you?…and now same here with the 6.49beta11 "SFP at 1G is not stable" bug.
Hard down, will not establish link. Light is received, but port does not negotiate or come up when set to 1G.For us the 1Gs report 10G. Same for you?
Is anybody getting any feedback on support tickets from Mikrotik? I was told that they are working on it but not idea if any fix seems imminent or if this will first be resolved in ROS7...Hard down, will not establish link. Light is received, but port does not negotiate or come up when set to 1G.For us the 1Gs report 10G. Same for you?
Occasionally getting responses — we're being told to send supouts, console logs, change L2MTU, try setting ports to fixed speeds — but no visibility of what progress is being made.Is anybody getting any feedback on support tickets from Mikrotik? I was told that they are working on it but not idea if any fix seems imminent or if this will first be resolved in ROS7...Hard down, will not establish link. Light is received, but port does not negotiate or come up when set to 1G.For us the 1Gs report 10G. Same for you?
We have replaced the worst offenders with 1072s but are stuck with 20 2004s we had hoped to roll out and a need to upgrade 25 more sites with 2004s.
/M
What about: *) switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS deviceA week ago I was told by MT support they were hoping to have a fix out in the next beta. Seems one was launched today but nothing in release notes, but could be non-documented. Anybody have any more information?
/Mikael
That was only for port stability according to MT support and already present in beta 11.What about: *) switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS deviceA week ago I was told by MT support they were hoping to have a fix out in the next beta. Seems one was launched today but nothing in release notes, but could be non-documented. Anybody have any more information?
/Mikael
*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
Just got confirmationThere is a new related item in the latest beta:
*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
We've had the same, and will be deploying it on pre-production routers tonight.Just got confirmationThere is a new related item in the latest beta:
*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
The latest RouterOS v6.49beta22 has improved resource allocation which has been extensively tested before the release and addresses CCR2004 performance issues in various setups. Please upgrade CCR2004 to see if it now works fine in your planned application.
Maznu let know us if it works thx GiuseppeWe've had the same, and will be deploying it on pre-production routers tonight.
We'll know if it doesn't immediate crash and burn later tonight… but from experience it might be a few weeks before we know if it's stable or not. Will keep this thread updated as we go.Maznu let know us if it works thx GiuseppeWe've had the same, and will be deploying it on pre-production routers tonight.
10 minutes on 6.49beta22 so far, no issues to report. This device only has 10G links in the SFP+ ports, though, so your mileage may vary with SFP or SFP28.We'll know if it doesn't immediate crash and burn later tonight… but from experience it might be a few weeks before we know if it's stable or not. Will keep this thread updated as we go.Maznu let know us if it works thx GiuseppeWe've had the same, and will be deploying it on pre-production routers tonight.
So ROS 6.49.22 version solve the packet loss problem?10 minutes on 6.49beta22 so far, no issues to report. This device only has 10G links in the SFP+ ports, though, so your mileage may vary with SFP or SFP28.
I don't think it does, at least not if you have many thousands of routes with nexthops on different physical interfaces.So ROS 6.49.22 version solve the packet loss problem?10 minutes on 6.49beta22 so far, no issues to report. This device only has 10G links in the SFP+ ports, though, so your mileage may vary with SFP or SFP28.
Most has been fixed in last RC but still some left. MT says they are improving it still but the fix itself is well tested.Anyone has good news? :-)
thank youMost has been fixed in last RC but still some left. MT says they are improving it still but the fix itself is well tested.Anyone has good news? :-)
/M
Hows the uptime look so far?
Most has been fixed in last RC but still some left. MT says they are improving it still but the fix itself is well tested.
/M
No reboots yet on beta22.Hows the uptime look so far?
Hi,I have the same problem and opened a ticket with no response over 2 weeks ago. Very annoying. The odd thing is that im only getting the tx drops on the interface going to my elan. At first I thought it was an mti mismatch but have since ruled that out. I even put a CRS309 in between my ELAN and my CCR2004 and the tx drops moved to the interface going to the ELAN on the CRS309 and stoped on my cloud core. I've tried finisar, mikrotik, and fiberstore modules with the same results. Even tried a 1G and 10G DAC from multiple vendors. My ISP for my ELAN checked their equipment and show no drops on their side or FCS errors. It's odd that the problem moved down to the interface on my CRS309 instead of staying on my CCR2004. Even odder is when I unplug the 10G DIA and switch over to my backup 1G DIA the TX drops go away and as soon as I plug my 10G DIA back in they come back. It is weird that the TX drops only show up once you open the interface and not in the column view when I select them. It always shows 0 in the column view on both the crs2004 and crs309 depending on whether or not im using the crs309 inline but shows up after double clicking the interface and going to stats. The other weird thing is that the tx drops don't show up on any other 1G interface on the CCR2004. I would expect it to be consistent at the very least. I set a lab up with all the smae routers and have yet to be able to reproduce the result in the lab and I believe this to be due to the fact that I am not able to tax the CPU load high enough to perturbate the issue. Please Mikrotik. HELP!!!!! My CCR2004 like many others is a paperweight as it is causing out of order packets and voice quality issues even thought the packet loss through the ccr2004 is low at 0.08 to 0.12 percent.
Seems to still be the same in ROS7 too. Just did testing with a couple.I have exactly the same situation. Did you ever managed to make it work?
Sounds like something 7.1 will fix unfortunatly.Running 6.49.1 with the queue fix I am picking up packet loss only when BGP is syncing routes (100k). If I disable the BGP session then the router shows no signs of packet loss.
Packet loss is also experienced between the router and the internal network.
Anyone know of a way to stop BGP impacting the packet flow when syncing... ie it seems maybe when the CPU running the BGP process, packets in that CPU's queue timeout/drop/die.
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default
I see 7.1 stable has been released.... so might look at trying it, but how to upgrade remotely will be they key, since BGP route filters are not updated in to 7.1Sounds like something 7.1 will fix unfortunatly.Running 6.49.1 with the queue fix I am picking up packet loss only when BGP is syncing routes (100k). If I disable the BGP session then the router shows no signs of packet loss.
Packet loss is also experienced between the router and the internal network.
Anyone know of a way to stop BGP impacting the packet flow when syncing... ie it seems maybe when the CPU running the BGP process, packets in that CPU's queue timeout/drop/die.
What queuefix are you running?
Beware a little - we had issues with something like L2 in the routers after upgrade. We managed to get into it using 30 attempts of mac-telnet and a couple of reboots after just made it work.I see 7.1 stable has been released.... so might look at trying it, but how to upgrade remotely will be they key, since BGP route filters are not updated in to 7.1
We were instructed to use 500 and doing so on 7.x too.Code: Select all/queue type set ethernet-default pfifo-limit=300 /queue interface set [find where queue!=no-queue] queue=ethernet-default
viewtopic.php?p=895484#p895484
Just a clarification if someone reads this thread and sees my "rock solid". We dont have any reboots anymore, but we still have the packetloss on 2004s running 7.1.1.We were instructed to use 500 and doing so on 7.x too.Code: Select all/queue type set ethernet-default pfifo-limit=300 /queue interface set [find where queue!=no-queue] queue=ethernet-default
viewtopic.php?p=895484#p895484
/M
I have not. I am still waiting for route aggregation to get added to ROS7 and for ROS7 to get a little more seasoned with time before I try again. Hopefully in a few months I will be ready to bring the big brick out to see if it is still a paperweight for my desk or not.Hi,I have the same problem and opened a ticket with no response over 2 weeks ago. Very annoying. The odd thing is that im only getting the tx drops on the interface going to my elan. At first I thought it was an mti mismatch but have since ruled that out. I even put a CRS309 in between my ELAN and my CCR2004 and the tx drops moved to the interface going to the ELAN on the CRS309 and stoped on my cloud core. I've tried finisar, mikrotik, and fiberstore modules with the same results. Even tried a 1G and 10G DAC from multiple vendors. My ISP for my ELAN checked their equipment and show no drops on their side or FCS errors. It's odd that the problem moved down to the interface on my CRS309 instead of staying on my CCR2004. Even odder is when I unplug the 10G DIA and switch over to my backup 1G DIA the TX drops go away and as soon as I plug my 10G DIA back in they come back. It is weird that the TX drops only show up once you open the interface and not in the column view when I select them. It always shows 0 in the column view on both the crs2004 and crs309 depending on whether or not im using the crs309 inline but shows up after double clicking the interface and going to stats. The other weird thing is that the tx drops don't show up on any other 1G interface on the CCR2004. I would expect it to be consistent at the very least. I set a lab up with all the smae routers and have yet to be able to reproduce the result in the lab and I believe this to be due to the fact that I am not able to tax the CPU load high enough to perturbate the issue. Please Mikrotik. HELP!!!!! My CCR2004 like many others is a paperweight as it is causing out of order packets and voice quality issues even thought the packet loss through the ccr2004 is low at 0.08 to 0.12 percent.
I have exactly the same situation. Did you ever managed to make it work?
Thanks
Adrian
1%-2%, but is bad, with ccr1009 or ccr1036 we haven't any issue.What percentage of packets lost do you have?
The packetloss? Then the 2004 wont be usable is the conclusion?It's ha hardware problem due to cache missing on the switch chip, it can not be solved
Very interesting information...To avoid the packet loss you must use the same speed on all sfp ports
SPAMUnfortunatelly,