Page 1 of 1

CCR2004 packet loss

Posted: Mon Oct 12, 2020 11:00 pm
by server8
I have a seriuos problem with all my CCR2004 when traffic pass from giga interface to 10 giga interface and viceversa I have a packet loss aorund 1% or little less.

According with the block diagram https://i.mt.lv/cdn/product_files/CCR20 ... 200459.png I am thinking the problem is generated in the passive intelligent port extender

CCR2004 is not usable to mix different interface speeds :-(

how is it possible?

Re: CCR2004 packet loss

Posted: Mon Oct 12, 2020 11:24 pm
by mkx
Do you experience packet loss in both directions equllky or is one direction distinctively worse?

I'd expect problems in packet direction 10G->1G where quite some buffering occurs and enabled flow control should help a lot (another matter is whether flow control actually works).

Re: CCR2004 packet loss

Posted: Mon Oct 12, 2020 11:37 pm
by server8
We have mainly packet loss pinging from the remote end to ccr2004 gigabit interface, the problem probably occurs when the upstream traffic from the 10G->1G fullfill the ethernet..

Enabling flow control not solve the problem.

Re: CCR2004 packet loss

Posted: Thu Oct 15, 2020 10:28 am
by paulct
buffer memory issue? I.e not enough buffer space when going from 10Gbps to 1Gbps and vice versa. Not sure how to get the stats from a Mikrotik on this.

Re: CCR2004 packet loss

Posted: Thu Oct 15, 2020 11:38 am
by jebz
There was a similar issue on the CCR1072 where the ethernet port was claimed only to be suitable for management traffic.
viewtopic.php?t=125361

Re: CCR2004 packet loss

Posted: Thu Oct 15, 2020 4:30 pm
by IPANetEngineer
We have mainly packet loss pinging from the remote end to ccr2004 gigabit interface, the problem probably occurs when the upstream traffic from the 10G->1G fullfill the ethernet..

So does the packet loss only occur when the 1G interface is full?

Re: CCR2004 packet loss

Posted: Thu Oct 15, 2020 11:49 pm
by server8
We have mainly packet loss pinging from the remote end to ccr2004 gigabit interface, the problem probably occurs when the upstream traffic from the 10G->1G fullfill the ethernet..

So does the packet loss only occur when the 1G interface is full?
No when the giga interface receive data from 10 giga interface: the slower port sometimes has to keep in pause the faster one before to receive new packets because the buffer is full or not have the buffer.

Re: CCR2004 packet loss

Posted: Thu Oct 15, 2020 11:52 pm
by IPANetEngineer
What's the lowest throughput (on the 1G) you've seen the packet loss?

Re: CCR2004 packet loss

Posted: Fri Oct 16, 2020 8:46 am
by server8
The problem is how fast the data arrive on the giga port from the 10 giga port, you can loose packet also whit 100kb or less. If the number of packets arriving exeeds tha capibility of gigaport to send it you will lose packet.
Generally is around 1% or littble bit less. This is a big hardware poblem for ccr2004 and it is not usable to mix different speed ports.

Re: CCR2004 packet loss

Posted: Fri Oct 16, 2020 10:32 am
by normis
The 1G port is for management (winbox, ssh to the router), as it's also labeled on the front panel. This port is somewhat limited, and is meant for what it says. The diagram also gives a clue to this.

Re: CCR2004 packet loss

Posted: Fri Oct 16, 2020 11:58 pm
by server8
Normis we are speaking about SFP ports

Re: CCR2004 packet loss

Posted: Sat Oct 17, 2020 8:09 am
by Maggiore81
Sincerely I am very disappointed for CCR2004's performance.
We replaced a CCR1036 + 10G switch (with trunks) to a CCR2004 with directly put in DACs cables 10G.

We do about 4.5gb traffic through it.
20 firewall rules, most in RAW for "simple" packet filter at the border.
fasttrack enabled.


4x99% cpu, one 100%.
I tried disabling everything, just fastpath, 80% cpu.
even with 2g traffic the voip was chopping.
Replaced back the 1036, and we are with it now. I could consider the 1072 but I fear the watchdog issues that are on the forum. Till now the 1036 is 2+ years working solid.
we use the long-term ROS.

Re: CCR2004 packet loss

Posted: Sat Oct 17, 2020 11:32 am
by server8
A router where you can't mix different sfp ports speeds it's not usable in production, if I deploy fiber I need to have 0% packets loss and no 1% like now.

I opened a ticket [SUP-30387] but for now no solution to the issue :-(

Re: CCR2004 packet loss

Posted: Sat Oct 17, 2020 4:55 pm
by IPANetEngineer
Have you tried changing the interface queues for both ports from hardware-only to ethernet-default ?

You need buffering when mixing interface speeds on any router and i'm not sure what the default buffer capabilities are for the CCR2004.

I would at least try a few different queue types for the interfaces to see if performance improves.

Re: CCR2004 packet loss

Posted: Sat Oct 17, 2020 6:05 pm
by Maggiore81
Where can I find precise documentation about queue types?
Well I read the wiki... but I have not found any REAL example of WHERE to use wich queue type or interface type

Re: CCR2004 packet loss

Posted: Sat Oct 17, 2020 9:00 pm
by server8
Have you tried changing the interface queues for both ports from hardware-only to ethernet-default ?

You need buffering when mixing interface speeds on any router and i'm not sure what the default buffer capabilities are for the CCR2004.

I would at least try a few different queue types for the interfaces to see if performance improves.
The first think that I have tried before to write here and to open a support ticket.

Re: CCR2004 packet loss

Posted: Sun Oct 18, 2020 12:59 am
by Paternot
Sincerely I am very disappointed for CCR2004's performance.
We replaced a CCR1036 + 10G switch (with trunks) to a CCR2004 with directly put in DACs cables 10G.
But You did a downgrade! Take a look at the performance figures:

https://mikrotik.com/product/CCR1036-12 ... estresults
https://mikrotik.com/product/ccr2004_1g ... estresults

Yes, the 2004 is faster with pure fastpath and big packets, but only in this situation. Everything else, and the 1036 wipes the floor with it. It isn't meant as a direct substitute to the 1036 - it is more something above 1009 and just below 1016.

Yes, each one of its 4 cores is much faster than the 10XX ones - but not enough to offset 4 cores versus 36 cores. It will outperform the CCR10XX series in single threaded activities, but not in firewalling.

Re: CCR2004 packet loss

Posted: Sun Oct 18, 2020 1:05 am
by Maggiore81
Hello Paternot, I have read carefully the performance figures, but sincerely I expected a lot more.
I did not expect a 100% cpu load !!!
However I am planning to replace the 1036 witha 1072 because I need the pure 10G ports for each link.
I have some fear to get a 1072 that has issues as in the threads about "watchdog and 1072".

I use conntrack, I use Fasttrack, it is one of my border router.

Re: CCR2004 packet loss

Posted: Sun Oct 18, 2020 6:05 am
by Paternot
Hello Paternot, I have read carefully the performance figures, but sincerely I expected a lot more.
I did not expect a 100% cpu load !!!
However I am planning to replace the 1036 witha 1072 because I need the pure 10G ports for each link.
I have some fear to get a 1072 that has issues as in the threads about "watchdog and 1072".

I use conntrack, I use Fasttrack, it is one of my border router.
Well, I can't speak for the usage model on You network - but if it was taxing one CCR1036, I find remarkable that one CCR2004 can barely cope with the load. I mean, take a look at the results for 512 bytes and 25 ip rules: it's about 1/3 of the 1036 speed!

Re: CCR2004 packet loss

Posted: Sun Oct 18, 2020 11:39 am
by Maggiore81
You may be right, I may have chosen the wrong product.
The 1036 was at about 15% average cpu...

Re: CCR2004 packet loss

Posted: Sun Oct 25, 2020 9:38 pm
by server8
Any update on this issue?

Re: CCR2004 packet loss

Posted: Tue Oct 27, 2020 11:28 am
by server8
Any update on this issue?

Re: CCR2004 packet loss

Posted: Thu Oct 29, 2020 10:34 pm
by server8
Any update on this issue?

Re: CCR2004 packet loss

Posted: Fri Oct 30, 2020 11:45 am
by ozar
I would just like to say that I experienced the same issues.

I have x4 CCR2004's and when they were in my lab before being shipped to production I had them connected to each other using Mikrotik 1G S-RJ01 Ethernet modules. NSW-RTR-01 (172.16.4.1 - CCR2004) was connected to NSW-RTR-02 (172.16.4.2 - CCR2004) via a Mikrotik 1G S-RJ01 Ethernet module. I ran "ping 172.16.4.2 interface=sfp-sfpplus2 do-not-fragment count=100 size=1500" and received high packet loss. Since I did this a while ago now I would have to dig up this packet loss data if you want proof.

Alternatively Mikrotik you could grab x2 CCR2004's and connect them to each other via Mikrotik 1G S-RJ01 Ethernet modules and run the same ping test and see the packet loss yourself. I don't see any packet loss if the Router to Router connections use 10G interfaces instead. I haven't tested this but I assume 1G Fibre SFPs will be affected too. Hopefully shaping or setting the Tx/Rx Flow Control to on will resolve this issue.

Worth investigating.

Re: CCR2004 packet loss

Posted: Fri Oct 30, 2020 2:27 pm
by ozar
I still have x2 CCR2004's that aren't in production yet so I did some testing on them. The CCR2004's are directly connected to each other via sfp-sfpplus2.

I tested with:
- 1G Ethernet module - Mikrotik S-RJ01 - 17% packet loss (First attempt)
- DIFFERNT 1G Ethernet module & Ethernet cable - Mikrotik S-RJ01 - 3% packet loss (Second attempt)
- 1G SM Fibre SFP - Ubqiuiti UF-SM-1G-S - 0% packet loss
- 1G MM Fibre SFP - Ubiquiti UF-MM-1G - 0% packet loss

Obviously something is wrong with the CCR2004 and 1G ethernet modules. In production I have the Router to Router connections as 10Gbps DAC cables and they get 0% packet loss.

Mikrotik S-RJ01:
[x@NSW-RTR-01] > ping 172.16.4.2 count=420 do-not-fragment size=1500 interface=sfp-sfpplus2
419 172.16.4.2 1500 64 0ms
sent=420 received=347 packet-loss=17% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

Mikrotik S-RJ01:
[x@NSW-RTR-01] > ping 172.16.4.2 count=420 do-not-fragment size=1500 interface=sfp-sfpplus2
419 172.16.4.2 1500 64 0ms
sent=420 received=406 packet-loss=3% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

Ubqiuiti UF-SM-1G-S:
[x@NSW-RTR-01] > ping 172.16.4.2 count=420 do-not-fragment size=1500 interface=sfp-sfpplus2
419 172.16.4.2 1500 64 0ms
sent=420 received=420 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

Ubiquiti UF-MM-1G:
[x@NSW-RTR-01] > ping 172.16.4.2 count=420 do-not-fragment size=1500 interface=sfp-sfpplus2
419 172.16.4.2 1500 64 0ms
sent=420 received=420 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

Re: CCR2004 packet loss

Posted: Fri Oct 30, 2020 2:49 pm
by ozar
If I was you I would give shaping just below 1Gbps a shot and see if you still get the packet loss.

I don't use 1G Ethernet modules in productions so I don't have too much skin in the game, goodluck.

Re: CCR2004 packet loss

Posted: Fri Oct 30, 2020 6:20 pm
by server8
If I was you I would give shaping just below 1Gbps a shot and see if you still get the packet loss.

I don't use 1G Ethernet modules in productions so I don't have too much skin in the game, goodluck.
it not work

Re: CCR2004 packet loss

Posted: Fri Oct 30, 2020 6:22 pm
by server8
Can I have an official answer about this problem by mickrotik support? I am pinging every day for ticket [SUP-30387] wihtout any answer :-(

Re: CCR2004 packet loss

Posted: Mon Nov 02, 2020 9:23 am
by server8
Any update on this issue?

Re: CCR2004 packet loss

Posted: Mon Nov 02, 2020 3:27 pm
by mkx
Can I have an official answer about this problem by mickrotik support? I am pinging every day for ticket [SUP-30387] wihtout any answer :-(

If you're not getting response through official channels, then you surely won't get any response here. This is user forum (where MT employees might check from time to time and answer some questions) not official support forum.

Re: CCR2004 packet loss

Posted: Tue Nov 03, 2020 1:00 am
by server8
I have a support tikcet opened without any answer too

Re: CCR2004 packet loss

Posted: Wed Nov 04, 2020 2:03 pm
by oouups
Exactly the same problem here !
Tired of playing "Betatester" for Mikrotik. Paid for something that really not Works.. and no one help of them.... THX Mikrotik.
Lost time, lost patience, lost money... and you loose a costumer.

Re: CCR2004 packet loss

Posted: Thu Nov 05, 2020 9:03 am
by server8
I am using back old CCR016 to connect gigabit site. CCR2004 is not usable in real because if we have problem to use different port speeds we can't use 25 Gb/s uplink ports with 10 gbis/s ports.

Re: CCR2004 packet loss

Posted: Fri Nov 06, 2020 8:31 am
by server8
Any update on this issue?

Re: CCR2004 packet loss

Posted: Mon Nov 09, 2020 8:29 am
by server8
Any update on this issue?

Re: CCR2004 packet loss

Posted: Mon Nov 09, 2020 10:21 am
by mkx
Not much will happen in 3 days ... specially so not over weekend.

Re: CCR2004 packet loss

Posted: Sat Nov 21, 2020 9:39 am
by server8
There are a lot of users with same problem around and after 2 months no answer from mikrotik :-(

At this point I have to suppose that is รจ hw problem without solutions. CCR2004 is an expensive paperweight

Re: CCR2004 packet loss

Posted: Sat Nov 21, 2020 10:21 am
by Maggiore81
Hello. I want to tell you my personal experience.
My experience with 2004 was bad because I replaced a 1036, but maybe the 2004 was too "light" for the purpose.

Now I have installed two 2004 with:

1x10g uplink
1x rj01 to AF24 ubiquiti
2x SFP+ 10G to other sites
everything in routing via BGP
0 issues and cpu low.
I tried also to disable conntrack for the routing (previously I used fasttrack everywhere) and 0 issues, just a higher cpu.

Re: CCR2004 packet loss

Posted: Sat Nov 21, 2020 5:33 pm
by server8
Please check the tx drop under the tab Tx Stats

Re: CCR2004 packet loss

Posted: Sat Nov 21, 2020 5:39 pm
by Maggiore81
Hello
I got 1649 tx drop on the interface with an RJ01 with AIRFIBER24 from UBNT.
it 4 days and 4 hours.

it is a 500mbit link.
no issues from the customers.

maybe the issue you are describing?
The interfaces are routed between each other, so no mixed speeds in the same bridge.

Re: CCR2004 packet loss

Posted: Sat Nov 21, 2020 8:38 pm
by server8
Hello
I got 1649 tx drop on the interface with an RJ01 with AIRFIBER24 from UBNT.
it 4 days and 4 hours.

it is a 500mbit link.
no issues from the customers.

maybe the issue you are describing?
The interfaces are routed between each other, so no mixed speeds in the same bridge.
You are loosing packets if you check on SFP28 you 'll find zero tx drops.

Now I have all 10 gigabit pots but I loose packets onyl on sfplus ports.

Re: CCR2004 packet loss

Posted: Sun Nov 22, 2020 12:32 am
by Maggiore81
Hello.
I use:
Sfp1 1g sfp
Sfp2 10g link
Sfp3 1g on rj01 with airfiber
Sfp11 and sfp12 on a bridge. Two 1g links

I have drops only on the sfp3 1g.
I don't use 25g ports

Re: CCR2004 packet loss

Posted: Tue Nov 24, 2020 3:25 pm
by server8
It depens how fast the trafffic coming on the 10 gibi/s port to be routed or swithed to the other ports.

We are speaking about 0,1% of packet loss that is not to much for a wireless link but is too much for a fiber link.

Re: CCR2004 packet loss

Posted: Tue Nov 24, 2020 3:33 pm
by Maggiore81
Everything is routed!

Re: CCR2004 packet loss

Posted: Wed Nov 25, 2020 8:56 am
by wojrep
I have a similar problem, CCR2004 post sfp-sfplus2 - SFP 1G insert, no auto, flow controll On ... and I have a lot of drop packages.
They start to appear when I exceed the speed of 100Mbit / s, at 200-500Mbit / s I have several thousand of them per minute, multicast transmission is interrupted (frame stop and IPTV artifacts).
Changing queue improves things a bit, but fails to get stability.

Re: CCR2004 packet loss

Posted: Wed Nov 25, 2020 10:01 am
by Maggiore81
Hello
I never found final documentation in how changing queue helps.
I leave them at only-hardware-queue

Re: CCR2004 packet loss

Posted: Wed Nov 25, 2020 10:30 am
by wojrep
Today I have tested all combinations with different queue sizes.
There is a difference, if you set ethernet-default and experiment with the size of the queue it may be enough for you.
I checked between 50-1000 packets and there was always a TX DROP problem, with queue values of 150-250 the least problem was. However, I am using multicast for IPTV, so dropping packets too often is unacceptable.
This problem occurs only with optical inserts with 1Gbps link speed, no matter if the insert itself is 1Gbps or 10Gbps.

I went back to CCR1036, wait for the new original MikroTik inserts and check it again.

Re: CCR2004 packet loss

Posted: Wed Nov 25, 2020 2:27 pm
by Maggiore81
We use only hardware queue
on the wiki I didnt found any interesting data about changing queue types on interfaces.