Community discussions

MikroTik App
 
jkroon
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 59
Joined: Thu Apr 03, 2008 2:18 am
Contact:

traffic counters combined with HWL3 offloading

Mon Mar 04, 2024 3:50 pm

Hi,

What we're trying to do:

- track the per-VLAN traffic ingress and egress volumes.

How:

We utilize SNMP from another host that once a minute queries the traffic counters for all interfaces and logs this to rrd files.

The problem:

Whist we are in agreement (from the Switch side) to the traffic on the LACP port back to the switching infrastructure, the traffic volumes for the individual VLAN interfaces does not. We're seeing kbps values (even when live monitoring, perhaps mbps) when we're expecting several hundred mbps and even gbps values.

Our configuration:

Our setup uses HWL3 offloading for IPv4, which is the bulk of our traffic (without this we don't stand a chance of dealing with the relevant traffic volumes).
/interface ethernet switch
set 0 l3-hw-offloading=yes
sfp-sfpplus1 through 4 is combined into a LACP bond, on which is in turn the only port in a bridge:
/interface bonding
add mode=802.3ad name=bond-cisco slaves=\
    sfp-sfpplus1,sfp-sfpplus2,sfp-sfpplus3,sfp-sfpplus4

/interface bridge
add ingress-filtering=no name=bridge-cisco port-cost-mode=short \
    protocol-mode=none vlan-filtering=yes
add name=bridge-lo port-cost-mode=short

/interface bridge port
add bridge=bridge-cisco interface=bond-cisco internal-path-cost=10 path-cost=\
    10
/interface ethernet switch l3hw-settings
set fasttrack-hw=no

/interface bridge vlan
add bridge=bridge-cisco tagged=bond-cisco,bridge-cisco vlan-ids=\
    2-102,104-4094
Note 103 is excluded for reasons of external peering causing problems if it sees multiple MACs on the switch port on their end, so this just eliminates the accidental risk, but we don't think this is a major concern.

We would then add VLANs similar to this:
/interface vlan
add interface=bridge-cisco name=bond-cisco.2-internal-routing vlan-id=2
Various routing protocols (some static routes, OSPF and BGP) then manages the routes that's available and gets offloaded to hardware.

Once routes are offloaded however what we're seeing is that whilst bond-cisco will show ~2.8Gbps, the sum of the VLANs will only show ~10Mbps.

We take this as the massive BULK of our traffic is getting offloaded (a good thing), but it also means it's extremely difficult to monitor which peering links are congested - especially considering that the switches don't provide /port/vlan SNMP counters either, and in some cases we've got a handful of related VLANs aggregated on the same LACP bundles (typically less than 5, but enough to skew the results of our traffic graphing).

Is there a way to fix this that I don't seem to find? Bug or intended behaviour?
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 270
Joined: Mon Apr 27, 2020 10:14 am

Re: traffic counters combined with HWL3 offloading

Mon Mar 04, 2024 5:09 pm

Hi,

Per-VLAN L3HW counting is neither a bug nor an intended behavior. It is an unimplemented feature. Since traffic is routed via hardware, VLAN software counters are not updated (because the software does not see the traffic). We need to find a way to gather per-VLAN stats from the hardware to update the software counters. The task is on the roadmap.
 
jkroon
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 59
Joined: Thu Apr 03, 2008 2:18 am
Contact:

Re: traffic counters combined with HWL3 offloading

Mon Mar 04, 2024 10:42 pm

Hi Raymond,

Thank you, that makes perfect sense. I've also tried to look if I can get this info on the CISCO side, and it looks like I can get /port in/out stats, as well as /vlan in/out stats, but not /port/vlan.

What's funny is that the CISC /vlan the in/out ~ matches up because well, all traffic ingress on a vlan also generally egresses on one or more ports. The problem being that I still can't get up/down stats on a per peering link basis.

Anyway, I know and understand that the following question probably has no concrete answer, nor can you provide any information to which I can hold Mikrotik because, well, things change, and there might not be an actual solution no matter how hard you look.

Do you by any chance have any idea of what kind of time frames we might be looking at? This is turning into a bit of a commercial issue, so I've got the commercial people on my case and with no answers to give them ...

Kind regards,
Jaco
 
User avatar
ahmdzaki18
just joined
Posts: 12
Joined: Fri Oct 06, 2023 7:52 pm
Location: Jakarta, Indonesia
Contact:

Re: traffic counters combined with HWL3 offloading

Sun Apr 07, 2024 10:28 am

Hi,

Per-VLAN L3HW counting is neither a bug nor an intended behavior. It is an unimplemented feature. Since traffic is routed via hardware, VLAN software counters are not updated (because the software does not see the traffic). We need to find a way to gather per-VLAN stats from the hardware to update the software counters. The task is on the roadmap.
Hope Mikrotik concern about this. Other vendor like Cisco or Huawei also can monitor the counter using "statistic enable".
 
jkroon
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 59
Joined: Thu Apr 03, 2008 2:18 am
Contact:

Re: traffic counters combined with HWL3 offloading

Tue Apr 16, 2024 12:08 am

I've not tested but I'm also assuming netflow data will suffer the same problem.

Who is online

Users browsing this forum: baragoon, boocko, jaclaz, melody, phascogale and 40 guests