Community discussions

MikroTik App
 
guipoletto
Member Candidate
Member Candidate
Topic Author
Posts: 195
Joined: Mon Sep 19, 2011 5:31 am

CRS317 and TX-drops (maybe a workaround?)

Mon Apr 18, 2022 1:29 pm

I've been dealing with TX-Drops and some level of packet loss since the dawn of the 317's on my network

This happens when there is a transition from 10G->1G, and affects the whole switch (as i'm led to believe, due to the nature/organization of the switch-chip packetbuffer)
So far, i have tried to not mix 1G and 10G ports on the switches (not always possible), or concentrating the 1G stuff in a single switch
The second option wields weird results, as the Uplink ports of the "mixed" switch still present TX-Drops (when viewed from the "Uplink interface" in a Not-Mixed switch)

There may be other stuff affecting this scenario, but this is what i observed so far. (this is a production network, not a controlled lab scenario)

Recently i implemented a topology change in ~ half the network, building a "ring" of bonded interfaces, "two-by-two" (a 20gb ring using CRS317's, V6.49.3)
Well, i eventually noticed that there were no TX-Drops in that half of the network

So that got me a little hopefull, and i moved the Uplink port for the whole network(~8gb), from an unbonded port at a switch outside the "ring", to a bonded port in the "just adjacent" ring switch. (the interconnect between those two was not changed)

My overall packet loss from WAN --> LAN dropped from ~~6% to ~~3% instantly (as measured in one of the webscale portal/tools, cannot say more due to NDA)

I then changed all the connected 1G ports from a "bare interface", to a "single slave", bonded, 802.3ad port
That caused there to be no more TX-Drops


So, either this is a cosmetic thing (drops still occur, but ROS does not populate the field in the statistics)
Or there is really some quirk of the CRS317 Switch-Chip that makes a bonded port behave differently than a "bare" port, mitigating the lack-of-buffer situation.

Does anyone see similar behavior?
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: CRS317 and TX-drops (maybe a workaround?)

Mon Apr 18, 2022 4:23 pm

Thanks for writing up these observations...don't have a specific answer but you have me thinking.

I wonder if putting a port into bonding changes the allocation of buffer resources in any way?

Also what version of ROS are you using? Have you tried this on ROSv7?
 
guipoletto
Member Candidate
Member Candidate
Topic Author
Posts: 195
Joined: Mon Sep 19, 2011 5:31 am

Re: CRS317 and TX-drops (maybe a workaround?)

Mon Apr 18, 2022 4:29 pm

ROS is v6.49.3

As i have no remote access to this devices, i have refrained from updating any of those (for now)
 
netmk
just joined
Posts: 6
Joined: Mon Aug 31, 2020 9:31 pm

Re: CRS317 and TX-drops (maybe a workaround?)

Tue Apr 26, 2022 7:23 pm

I have a similar problem with the CRS317 switch:

- I have an LACP with 4 ports (sfp5, 6, 7 and 8) operating at 1Gbps each. I have another port (SFP12) without LACP operating at 1Gbps.
- LACP and SFP12 port are on a bridge, all have Hardware Offloading active.
- I have a VLAN 2000 between LACP and SFP12.

Problem: TX DROP is occurring on the SFP12 port.
When I disable LACP and configure only one port without LACP between 1036 and CRS317 (keeping the VLAN), the TX DROP of SFP12 ends.

I cannot understand why.

Note: I changed the SFP12 port to the SFP16 port, still the problem remained on the SFP16 port

Characteristics:
CRS317 running ROS 3.48.6
CRS317 with CPU at 0% usage
CRS317 with HW Offloading only
Only 400Mbps traffic
CRS317 only layer2
You do not have the required permissions to view the files attached to this post.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS317 and TX-drops (maybe a workaround?)

Wed Apr 27, 2022 12:02 am

please post your 1036 and 317 config to properly understand your scenario (please edit any confidential/private info)
 
netmk
just joined
Posts: 6
Joined: Mon Aug 31, 2020 9:31 pm

Re: CRS317 and TX-drops (maybe a workaround?)

Thu Apr 28, 2022 6:02 pm

please post your 1036 and 317 config to properly understand your scenario (please edit any confidential/private info)
Hi
I simplified/updated the settings to see if the problem ended, however, even with only 2 ports on LACP the TX DROP continues

1036 -----------------------------------------------------------------------------------------------------------------------------
# apr/28/2022 00:12:39 by RouterOS 6.48.6
# model = CCR1036-12G-4S

/interface bridge
add name=IpLocal

/interface ethernet
set [ find default-name=sfp1 ] advertise=10M-full,100M-full,1000M-full comment="LACP3"
set [ find default-name=sfp2 ] auto-negotiation=no comment="LACP3"

add comment="LACP3" mode=802.3ad name=LACP3-R65 slaves=sfp2,sfp3 transmit-hash-policy=layer-2-and-3

/interface vlan
add comment="" interface=LACP3-R65 name=pppoeservice1 vlan-id=101
add comment="X" interface=LACP3-R65 name=pppoeservice2 vlan-id=103
add comment="X" interface=LACP3-R65 name=pppoeservice3 vlan-id=104
add comment="X" interface=LACP3-R65 name=pppoeservice4 vlan-id=105
add comment="X" interface=LACP3-R65 name=pppoeservice5 vlan-id=106
add comment="X" interface=LACP3-R65 name=pppoeservice6 vlan-id=107
add comment="X" interface=LACP3-R65 name=pppoeservice7 vlan-id=401
add comment="X" interface=LACP3-R65 name=pppoeservice8 vlan-id=402
add comment="X" interface=LACP3-R65 name=pppoeservice9 vlan-id=403
add comment="X" interface=LACP3-R65 name=pppoeservice10 vlan-id=404
add comment="X" interface=LACP3-R65 name=pppoeservice11 vlan-id=405
add comment="X" interface=LACP3-R65 name=pppoeservice12 vlan-id=406
add comment="X" interface=LACP3-R65 name=pppoeservice13 vlan-id=407
add comment="X" interface=LACP3-R65 name=pppoeservice14 vlan-id=408
add comment="X" interface=LACP3-R65 name=pppoeservice15 vlan-id=601
add comment="X" interface=LACP3-R65 name=pppoeservice16 vlan-id=602
add comment="X" interface=LACP3-R65 name=pppoeservice17 vlan-id=603
add comment="X" interface=LACP3-R65 name=pppoeservice18 vlan-id=604
add comment="X" interface=LACP3-R65 name=pppoeservice19 vlan-id=605
add comment="X" interface=LACP3-R65 name=pppoeservice20 vlan-id=606
add comment="X" interface=LACP3-R65 name=pppoeservice21 vlan-id=607
add comment="X" interface=LACP3-R65 name=pppoeservice22 vlan-id=608
add comment="X" interface=LACP3-R65 name=pppoeservice23 vlan-id=801
add comment="X" interface=LACP3-R65 name=pppoeservice24 vlan-id=802
add comment="X" interface=LACP3-R65 name=pppoeservice25 vlan-id=803
add comment="X" interface=LACP3-R65 name=pppoeservice26 vlan-id=804
add comment="X" interface=LACP3-R65 name=pppoeservice27 vlan-id=805
add comment="X" interface=LACP3-R65 name=pppoeservice28 vlan-id=806
add comment="X" interface=LACP3-R65 name=pppoeservice29 vlan-id=807
add comment="X" interface=LACP3-R65 name=pppoeservice30 vlan-id=808

/ip firewall connection tracking set enabled=no

/interface pppoe-server server
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice1 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice1
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice2 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice2
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice3 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice3
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice4 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice4
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice5 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice5
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice6 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice6
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice7 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice7
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice8 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice8
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice9 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice9
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice10 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice10
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice11 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice11
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice12 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice12
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice13 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice13
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice14 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice14
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice15 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice15
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice16 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice16
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice17 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice17
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice18 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice18
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice19 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice19
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice20 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice20
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice21 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice21
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice22 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice22
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice23 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice23
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice24 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice24
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice25 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice25
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice26 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice26
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice27 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice27
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice28 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice28
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice29 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice29
add authentication=chap,mschap1,mschap2 disabled=no interface=pppoeservice30 max-mru=1492 max-mtu=1492 mrru=1600 one-session-per-host=yes service-name=pppoeservice30

/system resource irq rps
set sfp1 disabled=no
set sfp2 disabled=no
set sfp3 disabled=no
set sfp4 disabled=no
set ether1 disabled=no
set ether2 disabled=no

Switch --------------------------------------------------------------------------------------------------------------------------------------------------
# mar/10/1970 10:52:43 by RouterOS 6.48.6
# model = CRS317-1G-16S+

/interface ethernet
set [ find default-name=ether1 ] comment=gerencia
set [ find default-name=sfp-sfpplus6 ] advertise=10M-full,100M-full,1000M-full auto-negotiation=no comment="LACP3"
set [ find default-name=sfp-sfpplus7 ] advertise=1000M-full auto-negotiation=no comment="LACP3"
set [ find default-name=sfp-sfpplus12 ] auto-negotiation=no
set [ find default-name=sfp-sfpplus15 ] advertise=10000M-full comment="Dir SP-JQ"
set [ find default-name=sfp-sfpplus16 ] advertise=1000M-full

/interface bridge
add frame-types=admit-only-vlan-tagged ingress-filtering=yes name=bridge2 pvid=3 vlan-filtering=yes

/interface vlan add interface=bridge2 name=XXXXX vlan-id=2001

/interface bonding
add mode=802.3ad name="LACP3" slaves=sfp-sfpplus6,sfp-sfpplus7 transmit-hash-policy=layer-2-and-3

/interface bridge port
add bridge=bridge2 frame-types=admit-only-vlan-tagged ingress-filtering=yes interface="LACP3"
add bridge=bridge2 frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=sfp-sfpplus15
add bridge=bridge2 frame-types=admit-only-vlan-tagged ingress-filtering=yes interface=sfp-sfpplus12

/interface bridge vlan
add bridge=bridge2 tagged="sfp-sfpplus15,LACP3" vlan-ids=101
add bridge=bridge2 tagged="sfp-sfpplus12,LACP3" vlan-ids="103,104,105,106,107,401,402,403,404,405,406,407,408,601,602,603,604,605,606,607,608,801,802,803,804,805,806,807,808"

/ip address
add address=XXXXX interface=gerencia network=XXXXXXXXX

/ip route
add distance=1 gateway=XXXXXXX

/system identity
set name=Switch

/system routerboard settings
set boot-os=router-os

/system routerboard reset-button
set enabled=yes

/system swos
set address-acquisition-mode=static allow-from-ports=\
p1,p2,p3,p4,p5,p6,p7,p8,p9,p10,p11,p12,p13,p14,p15,p16,p17 identity=CRS1
You do not have the required permissions to view the files attached to this post.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS317 and TX-drops (maybe a workaround?)

Thu Apr 28, 2022 8:26 pm

maybe try using sfp12 as a member of another bonding interface to see if tx-drops are reduced
 
sirkike
just joined
Posts: 20
Joined: Mon Mar 22, 2010 10:34 pm
Location: Peru

Re: CRS317 and TX-drops (maybe a workaround?)

Fri Apr 29, 2022 1:42 pm

I also have TX Drops in CRS317-1G-16S+.
Hardware offload on.
CPU load 0%.
Only 5 VLAN's
3 SFP+ 10G no tx drops
2 Mikrotik S-RJ01 1GB a lot of tx drop (also try ubiquiti UF-RJ45-1G, same result)
ROS 6.49.6 (also try ROS 7.x, and routerboard firmware.)
Apparently it's the mix of 10G and 1GB. I read in another post, with swos this problem go away. So the problem is software not hardware?
 
biomesh
Long time Member
Long time Member
Posts: 561
Joined: Fri Feb 10, 2012 8:25 pm

Re: CRS317 and TX-drops (maybe a workaround?)

Fri Apr 29, 2022 3:01 pm

For the slower ports/interfaces try and set the ingress / egress port speed (switch interface) to close to what the interface can actually support. In the past when I used a 2.5 gb sfp+ adapter I had to set the rate to 2400 to get basically no retries and full speed (basically when sending traffic to my device)
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS317 and TX-drops (maybe a workaround?)

Fri Apr 29, 2022 5:25 pm

For the slower ports/interfaces try and set the ingress / egress port speed (switch interface) to close to what the interface can actually support. In the past when I used a 2.5 gb sfp+ adapter I had to set the rate to 2400 to get basically no retries and full speed (basically when sending traffic to my device)
i will try this
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS317 and TX-drops (maybe a workaround?)

Fri Apr 29, 2022 5:28 pm

i think we have to measure how much is "a lot"

i was checking on some crs3xx switches with tx-trops counters but i have not find any with even close to 0.01% of total frames on interface

will be nice to follow this trend based on clear numbers to measure it properly
 
guipoletto
Member Candidate
Member Candidate
Topic Author
Posts: 195
Joined: Mon Sep 19, 2011 5:31 am

Re: CRS317 and TX-drops (maybe a workaround?)

Sat Apr 30, 2022 9:21 pm

i think we have to measure how much is "a lot"

I personally have seen "surges" of TX-Drops when instantaneous loads get close to line-capacity (such as running a speedtest)

This causes test results to be unpredictable (even though low-troughput applications, such as web browsing remain mostly unaffected)

Mostly what i've done is bond the network to 20Gb wherever possible, and added single interfaces to a 802.3ad "one-legged-bond", with good results.

TX-Drops seem to kick in when port load is >70%, though i have no way to know the actual instantaneous load (in winbox, what we see is the average of the last 1-2 seconds)

Since the switch chip has (extrapolating from its big brother) maybe a 2MB packet buffer, the low performance when dealing with congestion is to be expected...
The interesting stuff is the fact that adding to a "bond" alleviates the problem. That was unexpected.

(source: https://isp-tech.ru/en/switch-asic/ )
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: CRS317 and TX-drops (maybe a workaround?)

Sat Apr 30, 2022 11:06 pm

i hope queue management on MikroTik switches is under development, for example there is no support for multiple output queues per port and that is a standard feature in the industry

i think some tx-drop on congestion is normal, and internet applications are prepared to deal with in moderate ammounts

at least mikrotik now is doing this kind of tx-dropping on outgoing interface i think to avoid HOL blocking, in CRS3xx early days rx-overflow happened on inbound interface with worse consequences

Who is online

Users browsing this forum: aoravent, menyarito, phascogale, Soleous75 and 77 guests