Community discussions

MikroTik App
 
iamGavinJ
just joined
Topic Author
Posts: 1
Joined: Wed Feb 01, 2023 12:34 am

Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Wed Feb 01, 2023 1:11 am

Hi,
I'm fairly new to ROS so I'm guessing there is a gap in my understanding, and would appreciate someone explaining to me where I'm going wrong. I have a CCR2004-16G-2S+.

I have a BRIDGE with a BOND (802.3ad) as a member port, that in turn has two ETH interfaces as slaves, feeding upstream to a Unifi USW-8. All settings mostly as default (inc. ARP & HW Offload). When the LAG came up I was seeing insane amount of packet loss. Monitoring the BOND I would see flags (C) and (D) flapping. Disabling one of the slave ETH's alleviated the packet loss.
BRIDGE <--> BOND <--> ETH01 <--> USW-8
                 <--> ETH02 <-->
I then found this article [https://help.mikrotik.com/docs/display/ ... dbalancing] talking about ARP and HW Offload and decided I'd rebuild the config on the other side of my device which was completely vacant. i.e. No other contending configs using the switch chip/HW offload.

I first built a basic BOND with two ETHs, with all default settings (ARP Enabled), and everything worked fine. No packet loss. Then I created the BRIDGE and added the BOND to it, swapped the IP address from the BOND to the BRIDGE, and the packet loss returned.

When I disabled HW Offload for the bridge port (BOND), everything returned to normal.

Would someone please explain to me what's happening in the switch chip with the config I had, and why disabling HW Offload fixes the problem? Also is there a way I can correctly re-configure the stack to take advantage of HW Offload?

Thanks so much.
 
ourplan
newbie
Posts: 28
Joined: Mon Oct 23, 2017 10:42 am

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Wed May 10, 2023 7:51 pm

I guess I have the same problem. After bonding is established, when I add it to the bridge, rx is fine, but tx is causing problems. traffic is saturated. for example;
eth0=100mbps
eth1=30mbps
working as Since eth0 works at 100mbps on two ethers, it affects the total traffic when it reaches the maximum traffic.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Wed May 10, 2023 8:27 pm

until now CCR2004-16G-2S+ is the only product using 88E6191X switch chip

maybe you have found a bug
 
unlikely
Frequent Visitor
Frequent Visitor
Posts: 53
Joined: Fri Feb 21, 2020 1:16 pm

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Wed Jun 21, 2023 12:23 am

I think I have the same problem.

CCR2004-16G-2S+ with one bridge and two bond as two member ports (sfp+1&2, eth1&2).
When hw offload in enabled on the bond made by eth1+eth2 I notice massive packet loss from both bond.
I started to notice today without any conscoius change in the configuration regarding bonds, vlan, and so on.

Any news?
 
wiseroute
Member
Member
Posts: 352
Joined: Sun Feb 05, 2023 11:06 am

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Wed Jun 21, 2023 4:17 am

interesting 🤔
first built a basic BOND with two ETHs, with all default settings (ARP Enabled), and everything worked fine. No packet loss. Then I created the BRIDGE and added the BOND to it, swapped the IP address from the BOND to the BRIDGE, and the packet loss returned.
could the problem be in the underlying bridge stp operation? since stp operation is the opposite from bond/lacp operation? 🤔
guess I have the same problem. After bonding is established, when I add it to the bridge, rx is fine, but tx is causing problems. traffic is saturated. for example;
eth0=100mbps
eth1=30mbps
hmm... what kind of bond mode did you create? 🤔
CCR2004-16G-2S+ with one bridge and two bond as two member ports (sfp+1&2, eth1&2).
When hw offload in enabled on the bond made by eth1+eth2 I notice massive packet loss from both bond.
i don't have ccr. from the datasheet - what is the maximum throughput?

maybe the cpu got overrun by the bond logic and the amount of traffic passing through?

--- edit

have you tried to make 2 bonds with the same port speed? ie 2x 2 100mbps ports? or 1x 4 100mbps ports 🤔
 
EdPa
MikroTik Support
MikroTik Support
Posts: 274
Joined: Fri Sep 15, 2017 10:05 am
Location: Riga
Contact:

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Wed Jun 21, 2023 11:30 am

Have not seen your config, but do you have disabled RSTP on the CCR2004-16G-2S+ bridge? Can you test if packet loss disappears when RSTP is enabled?

Edit:
LACPDUs use MAC destination address 01:80:C2:00:00:02, which are non-forwardable by STP-compatible bridges. My guess is that "protocol-mode=none" together with HW offloading on 88E6393X, 88E6191X, 88E6190 switch chips does not correctly receive them and LACP negotiation fails. With default "protocol-mode=rstp" these non-forwardable packets are always redirected to the CPU.
 
wiseroute
Member
Member
Posts: 352
Joined: Sun Feb 05, 2023 11:06 am

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Wed Jun 21, 2023 1:08 pm

@edpa

i think these OP's @unlikely and @iamgavinj maybe correct - it seems adding bond interface to a bridge causing some packet lost. but, i am not really sure about their config/performance lost

while @ourplan - the bond mode still in question?

output - balance-rr, without bridge on v7.9:

[admin@sw2] > /tool/bandwidth-test 192.168.2.1 direction=both user=admin password=admin duration=5m

3 minutes test no-bridge-bond1

[admin@sw1] > /interface/monitor-traffic bond1
                         name:      bond1
        rx-packets-per-second:         83
           rx-bits-per-second:  1005.2...
     fp-rx-packets-per-second:          0
        fp-rx-bits-per-second:       0bps
         rx-drops-per-second:          0
         rx-errors-per-second:          0
        tx-packets-per-second:         85
           tx-bits-per-second:  997.1kbps
     fp-tx-packets-per-second:          0
        fp-tx-bits-per-second:       0bps
          tx-drops-per-second:          0
    tx-queue-drops-per-second:          0
         tx-errors-per-second:          0

with bond inside the bridge :

[admin@sw1] > /interface/bonding/print detail
Flags: X - disabled; R - running
 0  R name="bond1" mtu=1500 mac-address=50:00:00:01:00:00 arp=enabled arp-timeout=auto slaves=ether1,ether2
      mode=balance-rr primary=none link-monitoring=mii arp-interval=100ms arp-ip-targets="" mii-interval=5s
      down-delay=0ms up-delay=0ms lacp-rate=30secs transmit-hash-policy=layer-2 min-links=0

[admin@sw1] > /interface/bridge/print detail
Flags: X - disabled, R - running
 0 R name="bridge-bond1" mtu=auto actual-mtu=1500 l2mtu=1500 arp=enabled arp-timeout=auto mac-address=50:00:00:01:00:00
     protocol-mode=rstp fast-forward=yes igmp-snooping=no auto-mac=yes ageing-time=5m priority=0x8000
     max-message-age=20s forward-delay=15s transmit-hold-count=6 vlan-filtering=no dhcp-snooping=no

[admin@sw1] > /interface/bridge/port print
Columns: INTERFACE, BRIDGE, HW, PVID, PRIORITY, PATH-COST, INTERNAL-PATH-COST, HORIZON
# INTERFACE  BRIDGE        HW   PVID  PRIORITY  PATH-COST  INTERNAL-PATH-COST  HORIZON
0 bond1      bridge-bond1  yes     1  0x80             10                  10  none

[admin@sw1] > /interface/monitor-traffic bridge-bond1
                         name:  bridge-bond1
        rx-packets-per-second:            83
           rx-bits-per-second:     996.0kbps
     fp-rx-packets-per-second:             0
        fp-rx-bits-per-second:          0bps
          rx-drops-per-second:             0
         rx-errors-per-second:             0
        tx-packets-per-second:            73
           tx-bits-per-second:     884.1kbps
     fp-tx-packets-per-second:             0
        fp-tx-bits-per-second:          0bps
          tx-drops-per-second:             0
    tx-queue-drops-per-second:        88 596
         tx-errors-per-second:             0
-- [Q quit|D dump|C-z pause]

rstp activated.

this was on vm, so doesnt represent hardware performance.

--- edit

yes. @gavin was correct - disabling hw-offload *seemed* to temporary solve the error/drop problem?
[admin@sw1] > /interface/bridge/port print detail
Flags: X - disabled, I - inactive; D - dynamic; H - hw-offload
 0     interface=bond1 bridge=bridge-bond1 priority=0x80 path-cost=10 internal-path-cost=10 edge=auto
       point-to-point=auto learn=auto horizon=none hw=no auto-isolate=no restricted-role=no restricted-tcn=no pvid=1
       frame-types=admit-all ingress-filtering=yes unknown-unicast-flood=yes unknown-multicast-flood=yes
       broadcast-flood=yes tag-stacking=no bpdu-guard=no trusted=no multicast-router=temporary-query fast-leave=no

[admin@sw1] > /interface/monitor-traffic bridge-bond1 duration=1m
                         name:  bridge-bond1
        rx-packets-per-second:            80
           rx-bits-per-second:     960.0kbps
     fp-rx-packets-per-second:             0
        fp-rx-bits-per-second:          0bps
          rx-drops-per-second:             0
         rx-errors-per-second:             0
        tx-packets-per-second:            82
           tx-bits-per-second:     993.1kbps
     fp-tx-packets-per-second:             0
        fp-tx-bits-per-second:          0bps
          tx-drops-per-second:             0
    tx-queue-drops-per-second:             0
         tx-errors-per-second:             0
         
 
EdPa
MikroTik Support
MikroTik Support
Posts: 274
Joined: Fri Sep 15, 2017 10:05 am
Location: Riga
Contact:

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Wed Jun 21, 2023 2:50 pm

@wiseroute thanks, but it appears we might be discussing a separate topic here. Changing the "hw=yes/no" setting on CHR bridges won't have any impact. The slight variation in kilobits per second and the occurrence of tx queue drops are likely a result of reaching the maximum limit of the 1Mbps interface.
 
wiseroute
Member
Member
Posts: 352
Joined: Sun Feb 05, 2023 11:06 am

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Wed Jun 21, 2023 3:07 pm

@ edpa,

agreed. vm lab doesnt represent true hardware performance.

but - if you take a closer on those tx-rx error/drops between the bridged bonded interface vs plain bonded one, i think the error/drop would be quite significant in heavy traffic load. again, i just doing a balance-rr, i dont know about other variation results.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Tue Jul 18, 2023 7:39 pm

It seems that I am also affected by this problem, with an 802.3ad bonding interface on the CCR2004-16G-2S+ router.
I would not say there is a "massive" packet loss, but there appears to be packet loss for sure.
It looks like sporadically there is something that delays the traffic by some 500-700ms, e.g. visible when I ping a fixed address.
Probably when this happens and there is enough load, the result is packet loss due to buffer overflow?

I observe when working through the day on a VNC desktop connected via the router, that sometimes the session stalls for a brief moment then resumes.
I also got some complaints about "dropped calls" by users of WiFi calling (indoor the 4G coverage is poor so users of mobile phones use that).
It looks like the traffic sometimes quenches for a short period.

I have enabled RSTP on the bridge, no difference. But before that, the 802.11ad status alread was showing OK on both sides (router and Aruba switch).
Any progress on this matter, @EdPa?
 
wiseroute
Member
Member
Posts: 352
Joined: Sun Feb 05, 2023 11:06 am

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Tue Jul 18, 2023 8:34 pm

@ pe1chl
Probably when this happens and there is enough load, the result is packet loss due to buffer overflow?
can you try that @ gavinj suggested - hw=no?
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Tue Jul 18, 2023 8:59 pm

I have for now removed the bridge, so the bonding interface is used directly (with VLAN interfaces on top). That also removes the hw accelleration.
As the office is now closed it is more difficult to validate that it is resolved.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: Packet Loss!!! BOND (802.3ad) on BRIDGE w/ HW Offload

Wed Jul 19, 2023 12:36 pm

As the office is now closed it is more difficult to validate that it is resolved.
It looks like the issue has been resolved by that change. I intended to make the change anyway as EdPa earlier already explained that HW accel of a bonding interface would probably not do much: viewtopic.php?p=1013026#p1013026

However, it still is a bug that should be fixed, as it will lead to wasted debugging time. When it does not work, remove the feature (hw accel).

Who is online

Users browsing this forum: No registered users and 49 guests