Community discussions

MikroTik App
 
PackElend
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Tue Sep 29, 2020 6:05 pm

how does L3HW actually works?

Fri Feb 11, 2022 9:24 am

Hallo MikroTik,
I would like to discuss layer3-offloading (again), I hope some of you guys is reading this.

I am trying to understand the actual functionality. The only reference I can find in L3HW Feature Support is:
This works only for directly connected networks. Since HW does not know how to send ARP requests,
CPU sends an ARP request and waits for a reply to find out a DST MAC address on the first received packet of the connection that matches a DST IP address.
After DST MAC is determined, HW entry is added and all further packets will be processed by the switch chip.


I understand it like this. For all CRS300 and CCR2000:
  1. a new packet arrives and goes through the CPU and passes through everything as described in Manual:Packet Flow. If a direct route to a neighbouring network (VLAN) is open, without any restrictions/special handling such as NATQueuing, etc. are configured, a route is entered into the memory of the switch chip.
  2. further packets of the same connection now go directly via the switch chip
  3. inactive connections are removed from the memory of the switch chip.
-->
This means that if several VLANs are tagged via one port, then they will talk to each other via the switch chip, as far as this is permitted, and no longer via the CPU.
So even tagged traffic is offloaded to the Switch-Chip but if I read CRS317 l3hw + firewall question - MikroTik this is only possible in the latest v7.2RC-Version. The stable version can only handle untagged traffic. 


For the DX8000 and DX4000 series, as well as the CCR2000, there are two additional points:
  1. NAT, e.g. Masquerade
    1. the learning of the NAT rule(s) is as described in point 1 above
    2. the translation of the IPs for NAT'd connections is entered into the switch chip
    3. further packets of the same connection now go directly through the switch chip and I can hide the individual IPs from your VLAN when communicating with another VLAN
  2. FASTTRACK, for connections that no longer flow through the same bridge but after they are ESTABLISHED, do not being handled by e.g. Queuing. The best example here is the LAN2WAN connection.
    1. The learning of FASTTRACK connections is analogous to general rule 1 above.
    2. Source and destination interfaces of the directly connected networks are entered into the switch chip.
    3. Further packets of the same connection now go directly via the switch chip.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Mon Feb 14, 2022 4:32 pm

There are two types of Hardware Routing (L3HW): Full Hardware Routing and Firewall-Compatible Hardware Routing. Full L3HW, in turn, differs between routing via an explicit nexthop gateway(-s) and routing to a connected L2 network (a.k.a. Connected Routes). As a result, there are three different cases, so let's describe each of them.


1. Full Hardware Routing via Nexthop Gateway(-s)
Full Hardware Routing is set by enabling l3-hw-offloading both on the switch and switch ports:
/interface/ethernet/switch set 0 l3-hw-offloading=yes
/interface/ethernet/switch port set [find] l3-hw-offloading=yes

Now, let's define a routing via a direct gateway. For instance, let's route the 10.0/8 network via 192.168.1.1:
/ip/route add add dst-address=10.0.0.0/8 gateway=192.168.1.1

And then check the "H" flag to make sure that the route got offloaded:
/ip/route print 
Flags: D - DYNAMIC; A - ACTIVE; c, s, y - COPY; H - HW-OFFLOADED; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#       DST-ADDRESS      GATEWAY        DISTANCE
0  AsH  10.0.0.0/8       192.168.1.1           1

Offloading the above rule tells the switch chip to route all packets with destination IP in subnet 10.0/8 to 192.168.1.1. None of the packets to the 10.0/8 will ever enter the CPU. Hence, none of the IP Firewall rules will ever trigger. That's why Full L3HW is incompatible with IP Firewall. On the other hand, Full L3HW is the fastest one.

However, you may offload some traffic control via switch ACL rules:
/in/eth/sw/rule
So if a limited stateless firewall is enough for the given task, you may continue with Full L3HW + ACL Rules, reaching near wire-speed performance.


2. HW-Offloading Connected Routes
That is also part of Full L3HW, but this time we specify an interface as a gateway instead of a nexthop IP. RouterOS creates Connected Routes automatically (dynamically) when assigning an IP address to an interface. In the next example, we set IP addresses to ether1, vlan32, and vlan40 interfaces, which resulted in connected routes creation in the respective networks.
/ip address
add address=192.168.1.17/24 interface=ether1 network=192.168.1.0
add address=192.168.32.17/24 interface=vlan32 network=192.168.32.0
add address=192.168.40.17/24 interface=vlan40 network=192.168.40.0

/ip/route print 
Flags: D - DYNAMIC; A - ACTIVE; c, s, y - COPY; H - HW-OFFLOADED; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#       DST-ADDRESS      GATEWAY        DISTANCE
  DAcH  192.168.1.0/24   ether1                0
  DAcH  192.168.32.0/24  vlan32                0
  DAcH  192.168.40.0/24  vlan40                0
To understand how Connected Routes can be offloaded, we need to know how Connected Routes are processed. Let's imagine an Inter-VLAN routing case where a host 192.168.32.5 (vlan32) wants to connect to 192.168.40.10 (vlan40). The former sends a packet to our router (192.168.32.17). The router checks the routing table and identified the destination is somewhere in vlan40. So it broadcasts an ARP request "Who has 192.168.40.10?" to all bridge ports that belong to vlan40 ("/in/br/vlan print where vlan-ids=40"). Let's say the router gets the reply from sfp-sfpplus5 (which belongs to vlan40) and learns its L2 (MAC) address. Now, the router knows the physical interface and MAC address of the destination 192.168.40.10, caching the data in its Forwarding Database (FDB) for reuse in subsequent packet routing to the same IP address.

The answer "How Connected Routes can be offloaded?" is "They cannot". The switch chip cannot send ARP requests and resolve new hosts inside the subnet. RouterOS (CPU) does that instead. Connected Routes are redirected to CPU by default, but the resolved hosts are offloaded to the hardware as /32 routes. For instance, "dst-address=192.168.40.10/32 gateway=192.168.40.10%sfp-sfpplus5 VLAN-ID=40". That's why there is a remark in the documentation telling:
H-flag does not indicate that route is actually HW offloaded, it indicates only that route can be selected to be HW offloaded.
A connected route having the "H" flag means that the hosts within the subnet can be hw-offloaded. The subnet itself stays on the CPU. However, once both source and destination hosts are offloaded (as /32 routes), Full Hardware Routing is established between them, offering near wire-speed performance.

Since L3HW depends on L2HW, it utilizes the full capability of the hardware L2 processing, including VLAN tagging/untagging. In other words, Full Hardware Routing provides Inter-VLAN routing between tagged and/or untagged ports out of the box. And it works in RouterOS v7.1 too. The changes in v7.2 are related to FastTrack only, which we'll discuss in the next chapter.


3. FastTrack Connection HW Offloading
Full HW Routing is incompatible with the stateful IP Firewall. So the tradeoff must be made between speed and security. Even NAT, which requires connection tracking -- a feature of stateful L4 Firewall -- is incompatible with Full HW Routing. Fortunately, some device models support FastTrack Offloading, allowing to achieve near wire-speed routing performance for a limited amount of connections while keeping the Firewall running. In other words, FastTrack Offloading provides Hardware-accelerated L4 Stateful Firewall.

Firewall-Compatible HW Routing is set by disabling l3-hw-offloading on the switch ports, where Firewall rules must be applied:
# Enable full hardware routing on LAN ports
:foreach i in=[/interface/list/member/find where list=LAN] do={
    /interface/ethernet/switch/port set [/interface/list/member/get $i interface] l3-hw-offloading=yes
}
 
# Disable full hardware routing on WAN or IoT ports
:foreach i in=[/interface/list/member/find where list=WAN or list=IoT] do={
    /interface/ethernet/switch/port set [/interface/list/member/get $i interface] l3-hw-offloading=no
}
 
# Activate Layer 3 Hardware Offloading on the switch chip
/interface/ethernet/switch/set 0 l3-hw-offloading=yes

1. Packets are processed by the CPU/Firewall by default.
2. Established connections can be fast-tracked - a faster (shorter) processing path, resulting in higher speeds.
3. FastTrack connections can be offloaded to the switch chip (if the latter supports that), offering near wire-speed performance.
4. If a FastTrack connection requires network address translation, the NAT rule gets offloaded too.

Before v7.2, RouterOS didn't support FastPath on vlan-filtered bridges, meaning no FastTrack for Inter-VLAN routing and no HW offloading. The feature has been introduced in RouterOS v7.2rc2, allowing Firewall-Compatible Inter-VLAN Routing.


Difference Between Connected Routes and FastTrack Connection Offloading
While it may seem that those two follow a similar pattern (redirect to CPU first, then offload to HW), there are fundamental differences.
  • When a connected host gets offloaded (/32 route), all traffic to it gets routed by the hardware, bypassing the firewall. For instance, once your computer got connected to a server, the server's IP gets offloaded to the hardware, and any device on the network can access it, including a random IoT device from an insecure VLAN.
  • FastTrack Offloading applies to the offloaded connections only, and only until the respective connections are closed. If you're connected to the server, nobody else can connect to it without going through the Firewall first.
  • Amount of connected hosts that can be offloaded greatly exceeds the number of hardware FastTrack connections. At the moment of writing, the highest amount of HW FastTrack connection is 4.5k while some devices can offload up to 128k hosts (e.g. CRS317).
  • When possible, try establishing Full HW Routing between trusted networks (e.g., between admin and server VLANs), leaving Firewall for crossing the unsecured zone (or NAT).
  • Fine-tune FastTrack connections that can be HW-offloaded via firewall filter rules by setting hw-offload=yes|no. For instance, there is no need to HW-offload low-bandwidth connections, such as a smart power socket or a fridge controller.
 
PackElend
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Tue Sep 29, 2020 6:05 pm

Re: how does L3HW actually works?

Mon Feb 14, 2022 5:34 pm

thx a lot for the comprehensive answer, I have to find a silent moment tonight, read any details and make sure that I grasp it :)
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: how does L3HW actually works?

Mon Feb 14, 2022 6:53 pm

Thank you, @raimondsp. I wish all the documentation was written this way.
 
User avatar
Hominidae
Member
Member
Posts: 309
Joined: Thu Oct 19, 2017 12:50 am

Re: how does L3HW actually works?

Mon Feb 14, 2022 6:59 pm

+1
thank you!
 
PackElend
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Tue Sep 29, 2020 6:05 pm

Re: how does L3HW actually works?

Mon Feb 14, 2022 10:09 pm

@raimondsp thx for the great effort. Before trying to describe your answers in my own words could you tell me where the L3 information is stored in the switch chip?
As that causes me difficulties to imaging how the processes on the Switch Chip are happening.

According to the https://help.mikrotik.com/docs/display/ ... s-Features there is only the
  • Host table
  • VLAN table
  • Rule table
if I exclude inter-VLAN routing there is only the Host and Rule table left.
Host table stores only MAC and corresponding port, so only the Rule table remains.

If I compare the size of the Rule table with the possible IP4 routes there is quite a gap e.g. CRS326-24S+2Q+, ACL rules:170, IP4 Routes 16K - 30K.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Tue Feb 15, 2022 8:47 am

@raimondsp thx for the great effort. Before trying to describe your answers in my own words could you tell me where the L3 information is stored in the switch chip?
As that causes me difficulties to imaging how the processes on the Switch Chip are happening.

According to the https://help.mikrotik.com/docs/display/ ... s-Features there is only the
  • Host table
  • VLAN table
  • Rule table
if I exclude inter-VLAN routing there is only the Host and Rule table left.
Host table stores only MAC and corresponding port, so only the Rule table remains.

If I compare the size of the Rule table with the possible IP4 routes there is quite a gap e.g. CRS326-24S+2Q+, ACL rules:170, IP4 Routes 16K - 30K.

I'm afraid that I cannot reveal technical details of switch chips due to NDA. Switch chips have their internal memory: RAM and TCAM. In some cases, memory is shared between multiple processing units (e.g., FastTrack connections share TCAM with ACL rules), while most of the units have their exclusive memory regions (e.g., Routing Table).

L3HW Device Support
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: how does L3HW actually works?

Tue Feb 15, 2022 10:16 am

Before trying to describe your answers in my own words could you tell me where the L3 information is stored in the switch chip?

[snip]

According to the https://help.mikrotik.com/docs/display/ ... s-Features there is only the

My feeling is that the linked document (briefly) describes the basic switch chips used in low-end devices and those don't support L3HW offload. But doesn't describe the upper-end switch chips, which can offload L3. Or am I wrong @raimondsp?
 
PackElend
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Tue Sep 29, 2020 6:05 pm

Re: how does L3HW actually works?

Tue Feb 15, 2022 6:29 pm

I'm afraid that I cannot reveal technical details of switch chips due to NDA. Switch chips have their internal memory: RAM and TCAM.

L3HW Device Support
so we just say there are other tables in the TCAM besides the in Switch Chip Features - RouterOS - MikroTik Documentation describes one. These additional tables store information as given in the tables in L3 Hardware Offloading - RouterOS - MikroTik Documentation.
spacer
Could some be that kind and explained the difference between
IPv4 Route Prefixes
and
IPv4 Routes
?
I think and read my head sore but cannot come up with a proper way to explain the difference and the impact on how to configure the rules.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Wed Feb 16, 2022 11:31 am

The Switch Chip Features document had been written before L3HW implementation, so it does not contain L3-related tables. The latter are specified on L2HW Device Support page.

Marvell switch chips use classified proprietary algorithms for routing, which we cannot reveal without violating NDA. What is important for MikroTik users is that we have implemented an abstraction layer to keep all chip-specific functions under the hood while presenting the common routing UI: no matter if it is a software routing, DX3000, or DX8000 L3HW - the end-users have zero configuration overhead in terms of routing.

In the case of DX3000/DX2000 switch chip serries, it is quite simple: one RouterOS route entry (/ip/route/) reflects into one HW IPv4 route prefix entry. Connected hosts (/32 routes) also occupy the same table. As long as the total number of routes ("ip/route print count-only") + connected host count ("/interface/bridge/host print count-only") , 13312 (13k), everything gets offloaded. Exceeding the number, routes with shorter prefixes stay on the CPU.

The DX8000/DX4000 have entirely different routing tables. The new routing model is more hardware-friendly but causes headaches for human beings who want to understand it. Instead of prefixes, we have to offload route indexes (once again, in hardware-friendly but human-pain format). The entire IPv4 address range must be indexed, i.e., 0.0.0.0 - 255.255.255.255. Adding new route entries causes index rebuild, increasing hardware memory by 0-5 entries depending on the complexity of the routing table. That's why no exact numbers are given. For example, some routing tables containing 240K entries can be fully offloaded to CRS317 HW, while others with 160K entries barely fit. And you will never know until you try. Set up a BGP feed and look at RouterOS LOG: if the "Route HW table FULL" warning message does appear, everything got offloaded. If you see the warning, either configure routing filters to suppress hw-offload or buy another MikroTik device and offload half of the table to it. Yes, you can stack multiple CRS3xx/CCR2x16 devices to split the HW routing table - that way, you can do L3HW processing on the full BGP table.
 
almdandi
Frequent Visitor
Frequent Visitor
Posts: 64
Joined: Sun May 03, 2015 5:22 pm

Re: how does L3HW actually works?

Wed Feb 16, 2022 2:15 pm

So i also unterstund the limitation table wrong. For example the CRS317 can hold up to 240k routes and can route packets in hardware for all routes that a stored in the routing table of the switch chip. There is no connection limit because there is no connection tracking? The limitation for fastrack connection is independent from the routing table limitation rigth? The 4,5k Fasttrack connection limit is, if i offload connection with a firewall rule with hw-offload=yes ticked. Or are all fasttrack connection automatically offloaded?

For inter vlan routing (aka HW-Offloading Connected Routes). For example if i have two access switch connected to a CRS317. Each access switch is a separate L2 with a /24 subnet. Now if the CRS317 attempts to route between those two networks, two /32 routes a creaded in the routing table of the switch chip. If a second client in network A start to communicate to the same host in network B, only one new /32 route in added to the routing table for the return path. So worst case, i end up with 508 /32 routes for two /24 networks.
 
User avatar
jbl42
Member Candidate
Member Candidate
Posts: 214
Joined: Sun Jun 21, 2020 12:58 pm

Re: how does L3HW actually works?

Wed Feb 16, 2022 2:54 pm

For inter vlan routing (aka HW-Offloading Connected Routes). For example if i have two access switch connected to a CRS317. Each access switch is a separate L2 with a /24 subnet. Now if the CRS317 attempts to route between those two networks, two /32 routes a creaded in the routing table of the switch chip. If a second client in network A start to communicate to the same host in network B, only one new /32 route in added to the routing table for the return path. So worst case, i end up with 508 /32 routes for two /24 networks.
This is how L3 switching works and what makes is so efficient: Packets are forwarded on directly on L2 but based on L3 IP dest adresses. Instead of looking up the egress port in the host table based on L2 mac, it is looked up based on L3 dest IP. Same as there is one table entry per L2 MAC, there is also one table entry per L3 IP. In reality, it is a bit more ecomlicated, but the principle applies.
 
mada3k
Long time Member
Long time Member
Posts: 682
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: how does L3HW actually works?

Wed Feb 16, 2022 6:04 pm

Connection tracking is a firewall feature, not a router feature.

Personally I would rather see better MPLS-HW support. Just plain flat non-VRF L3 routing is a bit limiting and the hardware can never hold full BGP feeds anyways.
 
PackElend
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Tue Sep 29, 2020 6:05 pm

Re: how does L3HW actually works?

Wed Feb 16, 2022 6:42 pm

The new routing model is more hardware-friendly but causes headaches for human beings who want to understand it. Instead of prefixes, we have to offload route indexes (once again, in hardware-friendly but human-pain format).
I love your answer, you are so 100% right, you made my day :lol:


back to my question :) :
  1. DX3000/DX2000 stores as:
    1. manual added route
      /ip/route add add dst-address=10.1.1.0/24 gateway=192.168.1.1
      adds 254 routes, first route is 10.1.1.0 --> 192.168.1.1 and the last route is 10.1.1.254 --> 192.168.1.1
    2. learned host IP due to ARP request
      "dst-address=192.168.40.10/32 gateway=192.168.40.10%sfp-sfpplus5 VLAN-ID=40"
      This adds exactly one single route (a known host).
    both only happen, if source and destination port configured to allow L3HW
    .
  2. DX8000/DX4000 stores as:
    Instead of prefixes, we have to offload route indexes (once again, in hardware-friendly but human-pain format). The entire IPv4 address range must be indexed, i.e., 0.0.0.0 - 255.255.255.255.
    Could you explain what you mean by route indexes?
    The only thing I can conclude from your description is that you may refer to IPv4 classes?
    That would mean in the case
    /ip/route add add dst-address=10.1.1.0/24 gateway=192.168.1.1
    it is a call A address range, so the IP range is 1.0.0.0 to 127.0.0.0 which yields 16'777'214 IPs, each of them has an index, e.g. 1 to 16'777'214 and only the used indexes are getting offloaded but the Switch-Chip knows how to translate from index to an actual route?
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Thu Feb 17, 2022 10:05 am

So i also unterstund the limitation table wrong. For example the CRS317 can hold up to 240k routes and can route packets in hardware for all routes that a stored in the routing table of the switch chip. There is no connection limit because there is no connection tracking? The limitation for fastrack connection is independent from the routing table limitation right?
Everything is right.

The 4,5k Fasttrack connection limit is, if i offload connection with a firewall rule with hw-offload=yes ticked. Or are all fasttrack connection automatically offloaded?
FastTrack connection offloading require a filter rule with hw-offload=yes. The latter is enabled by default unless you explicitly disable it. FastTrack connections happen only if the packets traverse the CPU/Firewall, i.e., l3-hw-offloading=no on ingress or egress switch port.
/ip/firewall/filter add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
You can fine-tune which connections get offloaded by adding multiple rules. The next example offloads only TCP connections, leaving UDP packets to the CPU.
/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=no protocol=udp
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
add action=accept chain=forward connection-state=established,related
^ do not repeat that at home, or your kid will complain about missing shots in Counter-Strike due to latency /s

For inter vlan routing (aka HW-Offloading Connected Routes). For example if i have two access switch connected to a CRS317. Each access switch is a separate L2 with a /24 subnet. Now if the CRS317 attempts to route between those two networks, two /32 routes a creaded in the routing table of the switch chip. If a second client in network A start to communicate to the same host in network B, only one new /32 route in added to the routing table for the return path. So worst case, i end up with 508 /32 routes for two /24 networks.
In the original post, I forgot to mention that DX4000/DX8000 switch chips store /32 routes (host L3 entries) in Forwarding Database (FDB) rather than in the routing table. For instance, CRS317 can store 160K-240K routes + up to 64K L3 host entries.

Personally I would rather see better MPLS-HW support. Just plain flat non-VRF L3 routing is a bit limiting and the hardware can never hold full BGP feeds anyways.
We have plans for implementing MPLS-HW support, at least for the DX8000 series. Not in near future, though. Currently, we are working on v7.2 stabilization and IPv6 L3HW.

DX3000/DX2000 stores as: manual added route
/ip/route add add dst-address=10.1.1.0/24 gateway=192.168.1.1
adds 254 routes, first route is 10.1.1.0 --> 192.168.1.1 and the last route is 10.1.1.254 --> 192.168.1.1
[/code]
NO. It adds just one route: 10.1.1.0/24 ---> 192.168.1.1. I think you're confusing it with a connected route, e.g.:
/ip/route add add dst-address=10.1.1.0/24 gateway=ether1
.

Could you explain what you mean by route indexes?
Imagine that our routing table has only one entry - default gateway:
/ip/route add gateway=10.0.0.1
. The entire IPv4 range is covered, so the index is:
0.0.0.0 - 255.255.255.255 => 10.0.0.1
Then we add "10.4.0.0/16 => 10.0.0.2" entry. The index changes to:
1. 0.0.0.0 - 10.3.255.255 => 10.0.0.1
2. 10.4.0.0 - 10.4.255.255 => 10.0.0.2 
3. 10.5.0.0 - 255.255.255.255 => 10.0.0.1
In the above case, adding one route entry increased the number of index entries by two. Now, let's add "10.6.0.0/15 => 10.0.0.2":
1. 0.0.0.0 - 10.3.255.255 => 10.0.0.1
2. 10.4.0.0 - 10.4.255.255 => 10.0.0.2 
3. 10.5.0.0 - 10.5.255.255 => 10.0.0.1
4. 10.6.0.0 - 10.7.255.255 => 10.0.0.2
5. 10.8.0.0 - 255.255.255.255 => 10.0.0.1
Again, +1 route entry caused +2 index entries. However, adding "10.8.0.0/16 => 10.0.0.3" create only one additional index entry:
1. 0.0.0.0 - 10.3.255.255 => 10.0.0.1
2. 10.4.0.0 - 10.4.255.255 => 10.0.0.2 
3. 10.5.0.0 - 10.5.255.255 => 10.0.0.1
4. 10.6.0.0 - 10.7.255.255 => 10.0.0.2 
5. 10.8.0.0 - 10.8.255.255 => 10.0.0.3
6. 10.9.0.0 - 255.255.255.255 => 10.0.0.1
Adding "10.5.0.0/16 => 10.0.0.2" leads to an interesting result - instead of adding index entries, it reduces them by merging adjacent ranges:
1. 0.0.0.0 - 10.3.255.255 => 10.0.0.1
2. 10.4.0.0 - 10.7.255.255 => 10.0.0.2 
3. 10.8.0.0 - 10.8.255.255 => 10.0.0.3
4. 10.9.0.0 - 255.255.255.255 => 10.0.0.1

I showed you only the top of the iceberg. The hardware memory layout is way more complex, leading to even more variations. The specified "IPv4 Ranges" in the table is the worst and best common case. I bet the one can come up with a crazy setup and overflow the table with half of the specified number. On the other hand, with only two nexthops, no ECMP, and no recursive routes it might be possible to squeeze in the entire BGP table in CRS317 HW memory.
 
PackElend
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Tue Sep 29, 2020 6:05 pm

Re: how does L3HW actually works?

Thu Feb 17, 2022 4:17 pm

@raimondsp thx a lot of the patient and sharing all the details :D
I think you're confusing it with a connected route, e.g.:
yes, I did, day was long...
I have to be more detailed:
  1. DX3000/DX2000 stores as:
    1. manual added route, host to gateway
      /ip/route add dst-address=10.1.1.17/32 gateway=192.168.1.1
      adds a single route for the host 10.1.1.17/32 --> 192.168.1.
      That would happen to per host, who is asking for route, so in /24 you can end up with 254 routes pointing to e.g. 192.168.1
    2. manual added route, subnet to gateway
      /ip/route add dst-address=10.1.1.0/24 gateway=192.168.1.1
      adds a single route for the entire subnet 10.1.1.0/24 --> 192.168.1.1
    3. Connected Routes,
      CPU assumes there are more hosts in the same subnet behind the interface with the configured IP
      /ip address add address=192.168.1.1/24 interface=ether1 network=192.168.1.0
      /ip/route add dst-address=10.1.1.17/24 gateway=ether1
      
      adds a single route for the subnet 10.1.1.17/24 (or host 10.1.1.17/32) --> ether1(192.168.1.1/24)
    4. learned host IP due to ARP request
      "dst-address=192.168.40.10/32 gateway=192.168.40.10%sfp-sfpplus5 VLAN-ID=40"
      This adds exactly one single route (a known host).
    both only happen, if source and destination port configured to allow L3HW
    .
  2. DX8000/DX4000 stores as:
    There is ost likely a better wording but in my words, you cannot have overlapping routes like
    0.0.0.0 --> 192.168.1.1
    10.10.1.1 --> 192.168.1.1
    which would be applied in order from most specific to least specific (well explained in https://unix.stackexchange.com/question ... nalyzed-in)
    There can be only a single entry per pair, either subnet to gateway/interface or host to gateway/interface, so a table is created as excellent illustrated by you.
 
PackElend
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Tue Sep 29, 2020 6:05 pm

Re: how does L3HW actually works?

Fri Feb 18, 2022 12:39 am

Another thing as we talk about VLAN to VLAN routing.
The CPU becomes a tagged port allowing several VLAN (routed ones and Management) to enter the CPU.

What happens to VLAN id, what functionality replaces the source ID by the destination ID, so that it won't be dropped by the ingres filter + VLAN filter = yes?
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 2095
Joined: Mon Jan 14, 2008 1:53 pm
Location: Over the Rainbow
Contact:

Re: how does L3HW actually works?

Fri Feb 18, 2022 3:10 am

@raimondsp

Thank you for that extremely detailed explanation !
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 776
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: how does L3HW actually works?

Fri Feb 18, 2022 7:30 pm

If you see the warning, either configure routing filters to suppress hw-offload or buy another MikroTik device and offload half of the table to it. Yes, you can stack multiple CRS3xx/CCR2x16 devices to split the HW routing table - that way, you can do L3HW processing on the full BGP table.

Oh? How so? How would I hypothetically configure enough CRS3xx/CCR2x16 devices to have a full table among them?
 
User avatar
netzwerghh
Frequent Visitor
Frequent Visitor
Posts: 74
Joined: Sun Aug 07, 2011 4:23 pm
Location: Hamburg, DE
Contact:

Re: how does L3HW actually works?

Sun Feb 20, 2022 9:31 pm

If you see the warning, either configure routing filters to suppress hw-offload or buy another MikroTik device and offload half of the table to it. Yes, you can stack multiple CRS3xx/CCR2x16 devices to split the HW routing table - that way, you can do L3HW processing on the full BGP table.

Oh? How so? How would I hypothetically configure enough CRS3xx/CCR2x16 devices to have a full table among them?
I'm also interested to know how this might work. The new CCR2216 can handle up to 120k routes in hardware. How can I do full table (850k v4 and 150k v6) with 2x 120k?
 
PackElend
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Tue Sep 29, 2020 6:05 pm

Re: how does L3HW actually works?

Sun Feb 20, 2022 11:09 pm

Another thing as we talk about VLAN to VLAN routing.
The CPU becomes a tagged port allowing several VLAN (routed ones and Management) to enter the CPU.

What happens to VLAN id, what functionality replaces the source ID by the destination ID, so that it won't be dropped by the ingres filter + VLAN filter = yes?
that can be ignored as I forgot the actual L3 action, which means a Router does its router-thing and works through the headers. Decapsulates the packet, looking at the ethernet header first which contains the VLAN ID. Using that VLAN ID it knows which of its own VLAN-Interfaces the traffic belongs to. It replaces the MACs according to the new destination and source.

In the case of DHCP current destination and source, the MACs would be just swapped, in the case of routing, the new source, is the MAC of the VLAN interface and the destination MAC, the MAC of the routing destination.
Can I expect that ROS does the same thing as well as when L3HW is happening?


The given examples are basically static and dynamic rules, where traffic can only be controlled by ACL Rules
If you have a CRS3xx with DX8000 or DX4000 Series, you can use the IP Firewall, as this allows you to define which routes are allowed to be FastTrack giving you the option to do L3HW them, all other routes will still go through the CPU.

For those, who don't have CRS3xx with these chips, as for me (I have a CRS328), what would be configuration look like, if there are two Gateways, one on the Switch for Inter-VLAN-Routing and the second on the Route for IP-Firewall, WAN-Access etc.?
I want to use the CRS328 for Inter-VLAN-Routing only, anything else shall still go to my CCR1009-7G-1C-1S+, which is the exit to the WAN.

Would be there a default gateway configured per VLAN configured, what would redirect non-Inter-VLAN-traffic to the second gateway or shall that be solved by ACL rules?
 
OlofL
Member Candidate
Member Candidate
Posts: 113
Joined: Mon Oct 12, 2015 2:37 pm

Re: how does L3HW actually works?

Tue Feb 22, 2022 3:01 pm

Good read, is there a way to tweak the max amount of fasttrack connections? For instance less memory for l3hw offload but more memory for fasttrack?
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: how does L3HW actually works?

Tue Feb 22, 2022 3:39 pm

Good read, is there a way to tweak the max amount of fasttrack connections? For instance less memory for l3hw offload but more memory for fasttrack?

not to date but, there are some things you have to keep in mind

When the HW limit of Fasttrack or NAT entries is reached, other connections will fall back to the CPU. MikroTik's smart connection offload algorithm ensures that the connections with the most traffic are offloaded to the hardware.

3 Fasttrack connections share the same HW memory with ACL rules. Depending on the complexity, one ACL rule may occupy the memory of 3-6 Fasttrack connections.

4 MPLS shares the HW memory with Fasttrack connections. Moreover, enabling MPLS requires the allocation of the entire memory region, which could store up to 768 (0.75K) Fasttrack connections otherwise. The same applies to Bridge Port Extender. However, MPLS and BPE may use the same memory region, so enabling them both doesn't double the limitation of Fasttrack connections.

quoted from

https://help.mikrotik.com/docs/display/ ... iceSupport
 
PackElend
Member Candidate
Member Candidate
Topic Author
Posts: 268
Joined: Tue Sep 29, 2020 6:05 pm

Re: how does L3HW actually works?

Thu Feb 24, 2022 5:18 pm

Another thing as we talk about VLAN to VLAN routing.
The CPU becomes a tagged port allowing several VLAN (routed ones and Management) to enter the CPU.

What happens to VLAN id, what functionality replaces the source ID by the destination ID, so that it won't be dropped by the ingres filter + VLAN filter = yes?
that can be ignored as I forgot the actual L3 action, which means a Router does its router-thing and works through the headers. Decapsulates the packet, looking at the ethernet header first which contains the VLAN ID. Using that VLAN ID it knows which of its own VLAN-Interfaces the traffic belongs to. It replaces the MACs according to the new destination and source.

In the case of DHCP current destination and source, the MACs would be just swapped, in the case of routing, the new source, is the MAC of the VLAN interface and the destination MAC, the MAC of the routing destination.
Can I expect that ROS does the same thing as well as when L3HW is happening?


The given examples are basically static and dynamic rules, where traffic can only be controlled by ACL Rules
If you have a CRS3xx with DX8000 or DX4000 Series, you can use the IP Firewall, as this allows you to define which routes are allowed to be FastTrack giving you the option to do L3HW them, all other routes will still go through the CPU.

For those, who don't have CRS3xx with these chips, as for me (I have a CRS328), what would be configuration look like, if there are two Gateways, one on the Switch for Inter-VLAN-Routing and the second on the Route for IP-Firewall, WAN-Access etc.?
I want to use the CRS328 for Inter-VLAN-Routing only, anything else shall still go to my CCR1009-7G-1C-1S+, which is the exit to the WAN.

Would be there a default gateway configured per VLAN configured, what would redirect non-Inter-VLAN-traffic to the second gateway or shall that be solved by ACL rules?
I got answers from support:

VLAN ID is
  1. stripped off when routing from VLAN tagged Subnet to non VLAN tagged Subnet / Gateway
  2. is replaced, when routing from one VLAN to another VLAN?

would be great if some could help me on
the best strategy for CRS328 is when the routing to www happens on the router and only inter-VLAN-routing shall happen on the CRS328?
 
mada3k
Long time Member
Long time Member
Posts: 682
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: how does L3HW actually works?

Tue Mar 01, 2022 8:55 pm

But isn't quite inefficent to try to store millions of routes in a HW FIB ?

Cisco once solved it many years ago with caching "active traffic" in the hardware, but letting "unused" routes just reside in regular memory until needed.

It's not likley that you will communicate with the whole world simultaneously.
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 776
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: how does L3HW actually works?

Mon Mar 07, 2022 12:45 am

But isn't quite inefficent to try to store millions of routes in a HW FIB ?

Cisco once solved it many years ago with caching "active traffic" in the hardware, but letting "unused" routes just reside in regular memory until needed.

It's not likley that you will communicate with the whole world simultaneously.
It may be inefficient, but it's the measuring stick by which "real" DFZ routers are measured.
 
User avatar
Hammy
Forum Veteran
Forum Veteran
Posts: 776
Joined: Fri May 28, 2004 5:53 pm
Location: DeKalb, IL
Contact:

Re: how does L3HW actually works?

Tue Mar 22, 2022 6:06 pm

If you see the warning, either configure routing filters to suppress hw-offload or buy another MikroTik device and offload half of the table to it. Yes, you can stack multiple CRS3xx/CCR2x16 devices to split the HW routing table - that way, you can do L3HW processing on the full BGP table.

Oh? How so? How would I hypothetically configure enough CRS3xx/CCR2x16 devices to have a full table among them?
*bump*
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Tue Mar 22, 2022 7:52 pm

Stacking multiple CCR2x16 for full BGP table offloading wouldn't be cost-efficient. On the other hand, CRS317 is reasonably cheap for its switch chip capabilities.

For example, get four CRS317 and bridge them together. Create routing filters to accept dynamic routes only within the 0.0/2 range on the first router. Add a static route to route 64.0/2 traffic to router #2, 128.0/2 - to R3, and 192.0/2 - to R4. The second router accepts dynamic routes in 64.0/2, route 0.0/2 to R1, etc. As a result, you split the full BGP table into four hardware routers (switch chips). Well, route count in different subnets is uneven, and you need to fine-tune the ranges or maybe introduce the fifth CRS317, but I hope you got the idea.
 
User avatar
asaleh75
Trainer
Trainer
Posts: 193
Joined: Thu Nov 17, 2011 2:51 pm
Location: Dhaka, Bangladesh
Contact:

Re: how does L3HW actually works?

Wed Apr 27, 2022 2:46 pm

L3 Hardware Offloading or Offloading Fasttrack Connections works for vlans ? Testing CCR2216-1G-12XS-2XQ but not working for vlans.
Last edited by asaleh75 on Wed Apr 27, 2022 3:11 pm, edited 1 time in total.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Wed Apr 27, 2022 3:10 pm

Offloading Fasttrack Connections works for vlans ? Testing CCR2216-1G-12XS-2XQ but Offloading Fasttrack Connections for vlans not working.
Which RouterOS version are you using? VLAN FastPath/FastTrack support has been introduced in RouterOS v7.2.
 
User avatar
asaleh75
Trainer
Trainer
Posts: 193
Joined: Thu Nov 17, 2011 2:51 pm
Location: Dhaka, Bangladesh
Contact:

Re: how does L3HW actually works?

Wed Apr 27, 2022 3:13 pm

I am using v7.2.1. If works please help to configure it.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Wed Apr 27, 2022 3:15 pm

I am using v7.2.1. If works please help to configure it.

Show me your setup:
/interface export
/ip export
 
User avatar
asaleh75
Trainer
Trainer
Posts: 193
Joined: Thu Nov 17, 2011 2:51 pm
Location: Dhaka, Bangladesh
Contact:

Re: how does L3HW actually works?

Wed Apr 27, 2022 3:22 pm

[admin@2216-1] >
/interface/export 
# apr/27/2022 18:18:35 by RouterOS 7.2.1
# software id = 13BN-J2X6
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HCA07HJM7WW
/interface vlan
add interface=qsfp28-1-1 name=vlan101 vlan-id=101
add interface=qsfp28-1-1 name=vlan102 vlan-id=102
add interface=qsfp28-1-1 name=vlan103 vlan-id=103
add interface=qsfp28-2-1 name=vlan201 vlan-id=201
add interface=qsfp28-2-1 name=vlan202 vlan-id=202
add interface=qsfp28-2-1 name=vlan203 vlan-id=203
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 4 l3-hw-offloading=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik


[admin@2216-1] > /ip/export 
# apr/27/2022 18:21:45 by RouterOS 7.2.1
# software id = 13BN-J2X6
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HCA07HJM7WW
/ip address
add address=192.168.11.1/24 interface=qsfp28-1-1 network=192.168.11.0
add address=192.168.21.1/24 interface=qsfp28-2-1 network=192.168.21.0
add address=192.168.22.1/24 interface=vlan202 network=192.168.22.0
add address=192.168.12.1/24 interface=vlan102 network=192.168.12.0
/ip dhcp-client
add interface=ether1
/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes protocol=tcp
add action=accept chain=forward connection-state=established,related
/ip smb shares
add comment="default share" directory=/pub name=pub
/ip smb users
add name=guest
[admin@2216-1] >
Last edited by asaleh75 on Wed Apr 27, 2022 3:29 pm, edited 1 time in total.
 
User avatar
asaleh75
Trainer
Trainer
Posts: 193
Joined: Thu Nov 17, 2011 2:51 pm
Location: Dhaka, Bangladesh
Contact:

Re: how does L3HW actually works?

Wed Apr 27, 2022 3:24 pm

3 CCR2216 connected with QSFP28
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Wed Apr 27, 2022 3:37 pm

Hardware Inter-VLAN Routing requires a Hardware Bridge for VLAN tagging. More info here.
 
User avatar
asaleh75
Trainer
Trainer
Posts: 193
Joined: Thu Nov 17, 2011 2:51 pm
Location: Dhaka, Bangladesh
Contact:

Re: how does L3HW actually works?

Wed Apr 27, 2022 3:44 pm

Configuration will be same as switch?
 
User avatar
asaleh75
Trainer
Trainer
Posts: 193
Joined: Thu Nov 17, 2011 2:51 pm
Location: Dhaka, Bangladesh
Contact:

Re: how does L3HW actually works?

Wed Apr 27, 2022 4:16 pm

Hardware Inter-VLAN Routing requires a Hardware Bridge for VLAN tagging. More info here.
L3 Hardware Offloading working with below configuration. VLAN interface doesn't show any bandwidth. Now what to do for Offloading Fasttrack Connections?
[admin@2216-1] > /interface/export 
# apr/27/2022 19:05:55 by RouterOS 7.2.1
# software id = 13BN-J2X6
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HCA07HJM7WW
/interface bridge
add ingress-filtering=no name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=vlan101 vlan-id=101
add interface=bridge1 name=vlan102 vlan-id=102
add interface=bridge1 name=vlan103 vlan-id=103
add interface=bridge1 name=vlan201 vlan-id=201
add interface=bridge1 name=vlan202 vlan-id=202
add interface=bridge1 name=vlan203 vlan-id=203
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge1 interface=qsfp28-1-1
add bridge=bridge1 interface=qsfp28-2-1
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,qsfp28-1-1 vlan-ids=100-110
add bridge=bridge1 tagged=bridge1,qsfp28-2-1 vlan-ids=200-210
[admin@2216-1] > 
[admin@2216-1] > /ip/export 
# apr/27/2022 19:06:14 by RouterOS 7.2.1
# software id = 13BN-J2X6
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HCA07HJM7WW
/ip address
add address=192.168.11.1/24 interface=vlan101 network=192.168.11.0
add address=192.168.21.1/24 interface=vlan201 network=192.168.21.0
add address=192.168.22.1/24 interface=vlan202 network=192.168.22.0
add address=192.168.12.1/24 interface=vlan102 network=192.168.12.0
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Wed Apr 27, 2022 4:37 pm

L3 Hardware Offloading working with below configuration. VLAN interface doesn't show any bandwidth. Now what to do for Offloading Fasttrack Connections?
[admin@2216-1] > /interface/export 
# apr/27/2022 19:05:55 by RouterOS 7.2.1
# software id = 13BN-J2X6
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HCA07HJM7WW
/interface bridge
add ingress-filtering=no name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=vlan101 vlan-id=101
add interface=bridge1 name=vlan102 vlan-id=102
add interface=bridge1 name=vlan103 vlan-id=103
add interface=bridge1 name=vlan201 vlan-id=201
add interface=bridge1 name=vlan202 vlan-id=202
add interface=bridge1 name=vlan203 vlan-id=203
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=bridge1 interface=qsfp28-1-1
add bridge=bridge1 interface=qsfp28-2-1
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,qsfp28-1-1 vlan-ids=100-110
add bridge=bridge1 tagged=bridge1,qsfp28-2-1 vlan-ids=200-210
[admin@2216-1] > 
[admin@2216-1] > /ip/export 
# apr/27/2022 19:06:14 by RouterOS 7.2.1
# software id = 13BN-J2X6
#
# model = CCR2216-1G-12XS-2XQ
# serial number = HCA07HJM7WW
/ip address
add address=192.168.11.1/24 interface=vlan101 network=192.168.11.0
add address=192.168.21.1/24 interface=vlan201 network=192.168.21.0
add address=192.168.22.1/24 interface=vlan202 network=192.168.22.0
add address=192.168.12.1/24 interface=vlan102 network=192.168.12.0

VLAN interfaces do not show bandwidth because Inter-VLAN routing is fully offloaded to the hardware, and the traffic never enters the CPU. To redirect initial traffic to CPU (e.g., to use IP Firewall), disable l3-hw-offloading of the respective switch ports:
/interface/ethernet/switch/port
set qsfp28-1-1 l3-hw-offloading=no
set qsfp28-2-1 l3-hw-offloading=no
 
User avatar
asaleh75
Trainer
Trainer
Posts: 193
Joined: Thu Nov 17, 2011 2:51 pm
Location: Dhaka, Bangladesh
Contact:

Re: how does L3HW actually works?

Wed Apr 27, 2022 5:25 pm

When WAN has vlans & LAN has vlans. How to configure Fasttrack Connections Offloading + NAT ?
Below configurations are not working properly
/interface bridge
add ingress-filtering=no name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=vlan101 vlan-id=101
add interface=bridge1 name=vlan102 vlan-id=102
add interface=bridge1 name=vlan103 vlan-id=103
add interface=bridge1 name=vlan201 vlan-id=201
add interface=bridge1 name=vlan202 vlan-id=202
add interface=bridge1 name=vlan203 vlan-id=203
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 1 l3-hw-offloading=no
set 2 l3-hw-offloading=no
set 3 l3-hw-offloading=no
set 4 l3-hw-offloading=no
set 5 l3-hw-offloading=no
set 6 l3-hw-offloading=no
set 7 l3-hw-offloading=no
set 8 l3-hw-offloading=no
set 9 l3-hw-offloading=no
set 10 l3-hw-offloading=no
set 11 l3-hw-offloading=no
set 12 l3-hw-offloading=no
set 13 l3-hw-offloading=no
set 14 l3-hw-offloading=no
set 15 l3-hw-offloading=no
set 16 l3-hw-offloading=no
set 17 l3-hw-offloading=no
set 18 l3-hw-offloading=no
set 19 l3-hw-offloading=no
/interface bridge port
add bridge=bridge1 interface=qsfp28-1-1
add bridge=bridge1 interface=qsfp28-2-1
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,qsfp28-1-1 vlan-ids=100-110
add bridge=bridge1 tagged=bridge1,qsfp28-2-1 vlan-ids=200-210
/ip address
add address=192.168.11.1/24 interface=vlan101 network=192.168.11.0
add address=192.168.21.1/24 interface=vlan201 network=192.168.21.0
add address=192.168.22.1/24 interface=vlan202 network=192.168.22.0
add address=192.168.12.1/24 interface=vlan102 network=192.168.12.0
add address=192.168.13.1/24 interface=vlan103 network=192.168.13.0
add address=192.168.23.1/24 interface=vlan203 network=192.168.23.0
/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes protocol=tcp
add action=accept chain=forward connection-state=established,related
/ip firewall nat
add action=masquerade chain=srcnat out-interface=vlan101
add action=masquerade chain=srcnat out-interface=vlan102
add action=masquerade chain=srcnat out-interface=vlan103
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: how does L3HW actually works?

Wed Apr 27, 2022 6:17 pm

When WAN has vlans & LAN has vlans. How to configure Fasttrack Connections Offloading + NAT ?
Below configurations are not working properly
/interface bridge
add ingress-filtering=no name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=vlan101 vlan-id=101
add interface=bridge1 name=vlan102 vlan-id=102
add interface=bridge1 name=vlan103 vlan-id=103
add interface=bridge1 name=vlan201 vlan-id=201
add interface=bridge1 name=vlan202 vlan-id=202
add interface=bridge1 name=vlan203 vlan-id=203
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 1 l3-hw-offloading=no
set 2 l3-hw-offloading=no
set 3 l3-hw-offloading=no
set 4 l3-hw-offloading=no
set 5 l3-hw-offloading=no
set 6 l3-hw-offloading=no
set 7 l3-hw-offloading=no
set 8 l3-hw-offloading=no
set 9 l3-hw-offloading=no
set 10 l3-hw-offloading=no
set 11 l3-hw-offloading=no
set 12 l3-hw-offloading=no
set 13 l3-hw-offloading=no
set 14 l3-hw-offloading=no
set 15 l3-hw-offloading=no
set 16 l3-hw-offloading=no
set 17 l3-hw-offloading=no
set 18 l3-hw-offloading=no
set 19 l3-hw-offloading=no
/interface bridge port
add bridge=bridge1 interface=qsfp28-1-1
add bridge=bridge1 interface=qsfp28-2-1
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,qsfp28-1-1 vlan-ids=100-110
add bridge=bridge1 tagged=bridge1,qsfp28-2-1 vlan-ids=200-210
/ip address
add address=192.168.11.1/24 interface=vlan101 network=192.168.11.0
add address=192.168.21.1/24 interface=vlan201 network=192.168.21.0
add address=192.168.22.1/24 interface=vlan202 network=192.168.22.0
add address=192.168.12.1/24 interface=vlan102 network=192.168.12.0
add address=192.168.13.1/24 interface=vlan103 network=192.168.13.0
add address=192.168.23.1/24 interface=vlan203 network=192.168.23.0
/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes protocol=tcp
add action=accept chain=forward connection-state=established,related
/ip firewall nat
add action=masquerade chain=srcnat out-interface=vlan101
add action=masquerade chain=srcnat out-interface=vlan102
add action=masquerade chain=srcnat out-interface=vlan103

try this

viewtopic.php?p=925441&hilit=ether1#p925441
 
User avatar
slackR
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Sat May 23, 2009 1:46 pm
Location: Buffalo, New York, USA

Re: how does L3HW actually works?

Fri May 06, 2022 9:23 pm

Here is a L3HW question. Is it possible to see the switch chip memory utilization? I understand that each model has different restrictions when it comes to ACLs, Routes/prefixes, Nexthops, Fasttrack connections and NAT entries.

It would be nice to monitor the memory utilization for these items. Other vendors provide switch "resource" counters to see if you are close to maxing out the memory. Is this possible to implement in RouterOS?

Thanks,

Robert
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: how does L3HW actually works?

Fri May 06, 2022 9:26 pm

good idea now that we will be using switch chip resources will be useful to see TCAM usage or something like that
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Mon May 09, 2022 9:06 am

Here is a L3HW question. Is it possible to see the switch chip memory utilization? I understand that each model has different restrictions when it comes to ACLs, Routes/prefixes, Nexthops, Fasttrack connections and NAT entries.

It would be nice to monitor the memory utilization for these items. Other vendors provide switch "resource" counters to see if you are close to maxing out the memory. Is this possible to implement in RouterOS?

Thanks,

Robert

I like this idea. We'll investigate what hardware counters can be provided via RouterOS and how hard it is to implement.

Thank you for the suggestion!
 
paraplu
just joined
Posts: 14
Joined: Fri Sep 18, 2015 9:35 pm

Re: how does L3HW actually works?

Thu Oct 20, 2022 9:03 pm

Here is a L3HW question. Is it possible to see the switch chip memory utilization? I understand that each model has different restrictions when it comes to ACLs, Routes/prefixes, Nexthops, Fasttrack connections and NAT entries.

It would be nice to monitor the memory utilization for these items. Other vendors provide switch "resource" counters to see if you are close to maxing out the memory. Is this possible to implement in RouterOS?
+1 for this monitor feature. My CCR309 just slowed down with time-outs, eventually no connections possible anymore. Fully L3HW offloaded (NAT/FT); basically configured as a default router. With only 5 active devices within my home network, this happened only after a couple of hours uptime. No idea what happened. When do these offloaded FT/NAT-connections flush out, if ever? An option to flush the offloaded entries would be great too. Great to see a 1Gbps speedtest with almost 0% cpu usage!
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: how does L3HW actually works?

Thu Oct 20, 2022 9:14 pm

try this
/ip firewall connection tracking set tcp-established-timeout=15m 
 
paraplu
just joined
Posts: 14
Joined: Fri Sep 18, 2015 9:35 pm

Re: how does L3HW actually works?

Thu Oct 20, 2022 11:09 pm

Could we please have some more insight on how this "smart connection offload algorithm" works? And how it handles the timing of (idle) established/related offloaded connections? The only thing I found from confluence was this:

2 When the HW limit of Fasttrack or NAT entries is reached, other connections will fall back to the CPU. MikroTik's smart connection offload algorithm ensures that the connections with the most traffic are offloaded to the hardware.
 
paraplu
just joined
Posts: 14
Joined: Fri Sep 18, 2015 9:35 pm

Re: how does L3HW actually works?

Thu Oct 20, 2022 11:16 pm

try this
/ip firewall connection tracking set tcp-established-timeout=15m 
thank you I will try, but seriously doubt that this affects the connections within the ASIC.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Fri Oct 21, 2022 9:32 am

Marvell Prestera DX switch chips provide hardware traffic counters that RouterOS utilizes for connection tracking. That's how RouterOS detects idle/slow hardware connections and unloads them to free HW space for faster connections. The algorithm is quite complex; I don't want to go deep into the details.

Also, RouterOS prevents HW connections from timing out on the software side, so there is NO need to increase timeouts in /ip/firewall/connections/tracking.

A quote from RouterOS v7.6 release notes:
*) l3hw - fixed "H" flag presence for accelerated connection tracking entries;
*) l3hw - fixed possible packet loss when using HW offloaded NAT;
*) l3hw - improved connected host offloading on startup;
Try upgrading to v7.6 - maybe you encountered some of the issues above.
 
guipoletto
Member Candidate
Member Candidate
Posts: 195
Joined: Mon Sep 19, 2011 5:31 am

Re: how does L3HW actually works?

Fri Oct 21, 2022 11:12 am

"RouterOS prevents HW connections from timing out"

Wait, does RouterOS inject Keepalives in the HW-Offloaded connections?

what exactly does "prevents HW connections from timing out on the software side" mean?
 
llag
Frequent Visitor
Frequent Visitor
Posts: 72
Joined: Sat Aug 04, 2018 12:12 am

Re: how does L3HW actually works?

Fri Oct 21, 2022 12:52 pm

I have one question about L3 support: is L3 offloading for the firewall supposed to work between 2 VLANs?.

I have my WAN connection coming in on a VLAN.

I need to firewall (NAT) it to my internal router/firewall. All VLAN interface are defined on the bridge.
The fasttrack rule has hw-offload=yes.

When I tried with L3-HW-OFFLOADING=no on the port where the WAN vlan comes in and global forwarding on, I did no see any change in speed and connections don't have the H flag.

Is there something I missed?
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Fri Oct 21, 2022 12:56 pm

"RouterOS prevents HW connections from timing out"

Wait, does RouterOS inject Keepalives in the HW-Offloaded connections?

what exactly does "prevents HW connections from timing out on the software side" mean?

I meant that the software timer gets updated to prevent connection timeout and deletion in RouterOS (the one you see in /ip/firewall/connection/print) while the hardware routes the traffic. RouterOS does not inject any keepalive packets.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Fri Oct 21, 2022 1:00 pm

I have one question about L3 support: is L3 offloading for the firewall supposed to work between 2 VLANs?.

I have my WAN connection coming in on a VLAN.

I need to firewall (NAT) it to my internal router/firewall. All VLAN interface are defined on the bridge.
The fasttrack rule has hw-offload=yes.

When I tried with L3-HW-OFFLOADING=no on the port where the WAN vlan comes in and global forwarding on, I did no see any change in speed and connections don't have the H flag.

Is there something I missed?

Yes, FastTrack connection hw-offloading supports Inter-VLAN routing too. Does the connection have the F (Fasttrack) flag? Also, I highly recommend updating to RouterOS v7.6, as there have been bugfixes and improvements specifically for FastTrack/NAT connection hw-offloading.
 
doush
Long time Member
Long time Member
Posts: 665
Joined: Thu Jun 04, 2009 3:11 pm

Re: how does L3HW actually works?

Fri Oct 21, 2022 1:47 pm

Does Traffic Flow (Netflow) works with L3 HW Offloaded Interfaces ?
 
llag
Frequent Visitor
Frequent Visitor
Posts: 72
Joined: Sat Aug 04, 2018 12:12 am

Re: how does L3HW actually works?

Fri Oct 21, 2022 2:33 pm

Yes, FastTrack connection hw-offloading supports Inter-VLAN routing too. Does the connection have the F (Fasttrack) flag? Also, I highly recommend updating to RouterOS v7.6, as there have been bugfixes and improvements specifically for FastTrack/NAT connection hw-offloading.
Thanks, this was a really fast reponse!
It is good to know that l3-offloading for NAT should work between VLANs. I am quite sure that there is something stupid I did.

I am on 7.6 (upgraded especially for the hw-offloading improvements). fasttrack-hw=yes is set on the switch too..
All connections have the F flag, but the H flag is not set
[admin@Switch0-MikrotikCRS317] /interface/ethernet/switch> /ip/firewall/connection/print
Flags: S - SEEN-REPLY; A - ASSURED; C - CONFIRMED; F - FASTTRACK; s - SRCNAT
Columns: PROTOCOL, SRC-ADDRESS, DST-ADDRESS, TCP-STATE, TIMEOUT, ORIG-RATE, REPL-RATE, ORIG-PACKETS, REPL-PACKETS, ORIG-BYTES, REPL-BYTES
 #       PROTOCOL    SRC-ADDRESS            DST-ADDRESS           TCP-STATE    TIMEOUT    ORIG-RATE  REPL-RATE  ORIG-P  REPL-P  ORIG-BYTES  REPL-BYTES
 0 S C   icmp        192.168.180.252        192.168.180.1                      7s         0bps       0bps            1       1          56          56
 1 SACFs tcp         192.168.159.6:33440    157.240.247.60:443    established  23h59m52s  0bps       0bps          811     446      76 154      54 672
 2   C   igmp        172.16.1.31            224.0.0.1                          8m14s      0bps       0bps        1 092       0      39 312           0
 3 SACFs tcp         192.168.148.242:38694  142.250.102.188:5228  established  23h59m10s  0bps       0bps           42      44       3 753      26 014
 4 SACFs tcp         192.168.150.45:41850   185.20.209.25:993     established  23h59m31s  0bps       0bps        3 395   3 590     412 770     433 259
 5 SACFs tcp         192.168.159.6:35870    34.210.107.213:443    established  23h59m36s  0bps       0bps           51      72       5 260       8 666
 6 S CFs icmp        192.168.180.3          8.8.8.8                            9s         464bps     464bps      3 752   3 752     108 808     108 808
 7 SAC   tcp         192.168.159.6:39078    172.16.1.14:22        established  23h59m59s  4.6kbps    7.9kbps     5 684   5 257     336 201   1 035 804
 8 SACFs tcp         192.168.148.242:37442  18.184.210.55:5222    established  23h54m29s  0bps       0bps           34      18       2 598       1 284
 9 SACFs udp         192.168.180.16:5060    185.29.203.27:5060                 59m20s     0bps       0bps           74      37      19 897      17 191
10 SACFs tcp         192.168.150.46:46882   139.15.227.109:5222   established  23h59m35s  0bps       0bps        1 904   2 213     318 047     713 350
11 S CFs icmp        192.168.180.2          8.8.8.8                            9s         464bps     464bps      3 872   3 872     112 288     112 288
12 SAC   tcp         192.168.159.6:39134    172.16.1.14:443       established  23h56m46s  0bps       0bps        2 026   2 111     537 815   4 777 893
13 S C   ipv6-encap  87.212.48.242          216.66.84.46                       9m59s      2.2kbps    2.2kbps    39 045  38 001   3 314 750   6 468 664
14 SACFs tcp         192.168.159.6:58836    172.217.168.206:443   established  23h59m57s  0bps       0bps          386     554     138 628      77 355
15 SACFs tcp         192.168.148.241:59349  139.15.227.109:5222   established  23h59m35s  0bps       0bps        2 070   1 671     575 728     254 942
16   C s udp         192.168.180.16:5060    185.29.203.29:5060                 59m20s     0bps       0bps           53       0       1 696           0
17 SACFs tcp         192.168.148.246:60756  142.250.102.188:5228  established  23h50m32s  0bps       0bps          283     282      17 952     175 987
18 SAC   tcp         192.168.159.6:46378    172.16.1.14:443       established  23h59m59s  9.7kbps    237.3kbps   2 337   2 005     307 492   7 127 665
19 SACFs tcp         192.168.159.6:54306    185.20.209.25:993     established  23h57m52s  0bps       0bps          105     108      15 880      46 292
20 SACFs tcp         192.168.159.6:54324    185.20.209.25:993     established  23h57m50s  0bps       0bps           14      13       1 730       5 087
21 SACFs tcp         192.168.159.6:54340    185.20.209.25:993     established  23h57m50s  0bps       0bps           13      13       1 666       5 087
22 SACFs tcp         192.168.159.6:54362    185.20.209.25:993     established  23h57m50s  0bps       0bps           14      13       1 718       5 087
23 SACFs tcp         192.168.159.6:54314    185.20.209.25:993     established  23h57m50s  0bps       0bps           33      31       4 738      11 908
24 SACFs tcp         192.168.159.6:54300    185.20.209.25:993     established  23h57m50s  0bps       0bps           28      26       3 817       9 159
25 SACFs tcp         192.168.159.6:54322    185.20.209.25:993     established  23h57m50s  0bps       0bps           34      32       4 975      12 573
26 SACFs tcp         192.168.159.6:54346    185.20.209.25:993     established  23h57m50s  0bps       0bps           14      13       1 718       5 087
27 SACFs tcp         192.168.148.242:48962  142.251.39.106:443    established  23h58m40s  0bps       0bps           14      14       2 743       6 454
28 SACFs tcp         192.168.148.242:44278  142.250.179.202:443   established  23h58m40s  0bps       0bps           13      14       3 657       4 168
29 SACFs tcp         192.168.148.242:46372  142.251.36.3:443      established  23h58m40s  0bps       0bps           12      12       1 332       5 626
30 SACFs tcp         192.168.148.242:39228  142.251.36.42:443     established  23h58m41s  0bps       0bps           18      19       4 692       6 862
31 SACFs tcp         192.168.148.242:46376  142.251.36.3:443      established  23h58m40s  0bps       0bps           11      11       1 268       5 506
32 SACFs tcp         192.168.148.242:43860  157.240.201.60:443    established  23h58m40s  0bps       0bps           10       9       1 376       6 856
33 SACFs tcp         192.168.148.242:41876  142.250.179.206:443   established  23h58m41s  0bps       0bps           10       9       1 181       5 724
34 SACFs tcp         192.168.148.242:39884  216.239.32.36:443     established  23h58m42s  0bps       0bps           10       8       1 816       5 764
35 SACFs tcp         192.168.148.242:39886  216.58.214.10:443     established  23h58m45s  0bps       0bps            9       7       1 091       6 043
36 SACFs tcp         192.168.148.242:46314  142.250.179.138:443   established  23h58m44s  0bps       0bps           12      10       2 947       6 498
37 SACFs tcp         192.168.148.242:37740  142.251.36.22:443     established  23h58m45s  0bps       0bps           12       9       1 376       6 210
38 SACFs tcp         192.168.159.6:43538    142.250.179.194:443   established  23h59m6s   0bps       0bps           12      13       1 935       2 699
39 S CFs icmp        192.168.150.8          67.219.144.68                      6s         0bps       0bps           20      20       1 680       1 680
40 S CFs icmp        192.168.150.8          8.8.8.8                            6s         0bps       0bps           20      20       1 680       1 680
41 S CFs icmp        192.168.150.8          91.205.215.54                      6s         0bps ]

And this are the switch settings:
admin@Switch0-MikrotikCRS317] /interface/ethernet/switch> export verbose
# oct/21/2022 13:02:18 by RouterOS 7.6
# software id = DL3W-9T1J
#
# model = CRS317-1G-16S+
# serial number = 955C098CEE62
/interface ethernet switch
set 0 !cpu-flow-control l3-hw-offloading=yes mirror-source=none mirror-target=none name=switch1
/interface ethernet switch port
set 0 !egress-rate !ingress-rate l3-hw-offloading=no limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 1 !egress-rate !ingress-rate l3-hw-offloading=no limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 2 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 3 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 4 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 5 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 6 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 7 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 8 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 9 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 10 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 11 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 12 !egress-rate !ingress-rate l3-hw-offloading=no limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 13 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 14 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 15 !egress-rate !ingress-rate l3-hw-offloading=no limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 16 !egress-rate !ingress-rate l3-hw-offloading=yes limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
set 17 !egress-rate !ingress-rate limit-broadcasts=yes limit-unknown-multicasts=no limit-unknown-unicasts=no storm-rate=100
/interface ethernet switch l3hw-settings
set fasttrack-hw=yes icmp-reply-on-error=yes ipv6-hw=no
I have for now disabled l3-hw-offloading on all interfaces involved in the VLAN routing. But enabling it on the downstream ports does not make any difference.
B.T.W sfp01 (port 1) is the port that brings me WAN on VLAN300
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Fri Oct 21, 2022 3:14 pm

The config looks ok fine, but I'd like to see the full picture. Can you provide the output of the following commands please?
/interface export
/ip export
 
User avatar
osc86
Member Candidate
Member Candidate
Posts: 197
Joined: Wed Aug 09, 2017 1:15 pm

Re: how does L3HW actually works?

Fri Oct 21, 2022 5:32 pm

When WAN has vlans & LAN has vlans. How to configure Fasttrack Connections Offloading + NAT ?
...
Since it's introduction I never got it working on my CCR2116, and I'm pretty sure it's not a configuration issue, following the requirements of L3+Hardware+Offloading.
I attended a webinar held by iparchitechs a few weeks ago (thanks again) where they talked about layer 3 offloading capabilities of newer mikrotik switch/router hardware.
In their tests, l3 fw offloading was not successful if one or more of the following conditions apply:
- vlans are set up
- bonding interface is added as bridge port
- ingress/egress traffic is passing the same physical port
These limitations, if true, are nowhere to be found in the wiki.
@raimondsp can you please give a statement whether these limitations exist and lead to non-working l3 fw offloading?
 
llag
Frequent Visitor
Frequent Visitor
Posts: 72
Joined: Sat Aug 04, 2018 12:12 am

Re: how does L3HW actually works?

Fri Oct 21, 2022 5:33 pm

The config looks ok fine, but I'd like to see the full picture. Can you provide the output of the following commands please?
No problem, here we go:
Interface export:


[admin@Switch0-MikrotikCRS317] > /interface export 

# oct/21/2022 16:27:32 by RouterOS 7.6

# software id = DL3W-9T1J

#

# model = CRS317-1G-16S+

# serial number = 955C098CEE62

/interface bridge

add admin-mac=B8:69:F4:25:EB:5B auto-mac=no comment=defconf ingress-filtering=no mtu=1508 name=bridge vlan-filtering=yes

/interface ethernet

set [ find default-name=ether1 ] l2mtu=1592 speed=100Mbps

set [ find default-name=sfp-sfpplus1 ] advertise=10000M-full l2mtu=1592 name=sfp01-lair speed=10Gbps

set [ find default-name=sfp-sfpplus2 ] l2mtu=1592 name=sfp02 speed=10Gbps

set [ find default-name=sfp-sfpplus3 ] l2mtu=1592 name=sfp03 speed=10Gbps

set [ find default-name=sfp-sfpplus4 ] advertise="" l2mtu=1592 name=sfp04 speed=10Gbps

set [ find default-name=sfp-sfpplus5 ] disabled=yes l2mtu=1592 name=sfp05 speed=10Gbps

set [ find default-name=sfp-sfpplus6 ] disabled=yes l2mtu=1592 name=sfp06 speed=10Gbps

set [ find default-name=sfp-sfpplus7 ] disabled=yes l2mtu=1592 name=sfp07 speed=10Gbps

set [ find default-name=sfp-sfpplus8 ] l2mtu=1592 name=sfp08 speed=10Gbps

set [ find default-name=sfp-sfpplus9 ] disabled=yes l2mtu=1592 name=sfp09 speed=10Gbps

set [ find default-name=sfp-sfpplus10 ] disabled=yes l2mtu=1592 name=sfp10 speed=10Gbps

set [ find default-name=sfp-sfpplus11 ] disabled=yes l2mtu=1592 name=sfp11 speed=10Gbps

set [ find default-name=sfp-sfpplus12 ] disabled=yes l2mtu=1592 name=sfp12 speed=10Gbps

set [ find default-name=sfp-sfpplus13 ] l2mtu=1592 name=sfp13-den speed=10Gbps

set [ find default-name=sfp-sfpplus14 ] l2mtu=1592 name=sfp14-travel speed=10Gbps

set [ find default-name=sfp-sfpplus15 ] l2mtu=1592 name=sfp15-tankard speed=10Gbps

set [ find default-name=sfp-sfpplus16 ] l2mtu=1592 mtu=1508 name=sfp16-switch1 speed=10Gbps

/interface 6to4

add clamp-tcp-mss=no comment="HE IPv6 tunnelbroker" !keepalive mtu=1480 name=sit1 remote-address=216.66.84.46

/interface vlan

add interface=bridge name="vlan6 Cheapconnect DSL" vlan-id=6

add interface=bridge name="vlan180 -Transit" vlan-id=180

add interface=bridge name="vlan300 Tmo-WAN" vlan-id=300

add interface=bridge name="vlan2001 - NW man" vlan-id=2001

/interface pppoe-client

add add-default-route=yes default-route-distance=5 interface="vlan6 Cheapconnect DSL" max-mru=1480 max-mtu=1480 name=pppoe-DSL profile=default-ipv6 user=2986CN140

/interface ethernet switch

set 0 l3-hw-offloading=yes

/interface ethernet switch port

set 1 l3-hw-offloading=no

set 15 l3-hw-offloading=no

/interface list

add comment="all LAN facing interfaces" name=lan

add name=wan

/interface lte apn

set [ find default=yes ] ip-type=ipv4 use-network-apn=no

/interface wireless security-profiles

set [ find default=yes ] supplicant-identity=MikroTik

/interface bridge port

add bridge=bridge comment=defconf ingress-filtering=no interface=ether1

add bridge=bridge comment=defconf edge=yes frame-types=admit-only-vlan-tagged interface=sfp01-lair

add bridge=bridge comment=defconf edge=yes frame-types=admit-only-untagged-and-priority-tagged interface=sfp02 pvid=300

add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfp03

add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfp04

add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfp05

add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfp06

add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfp07

add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfp08

add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfp09

add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfp10

add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfp11

add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfp12

add bridge=bridge comment=defconf edge=yes frame-types=admit-only-vlan-tagged interface=sfp13-den

add bridge=bridge comment=defconf edge=yes frame-types=admit-only-vlan-tagged interface=sfp14-travel

add bridge=bridge comment=defconf edge=yes frame-types=admit-only-vlan-tagged interface=sfp15-tankard

add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfp16-switch1

/interface bridge settings

set use-ip-firewall=yes use-ip-firewall-for-vlan=yes

/interface bridge vlan

add bridge=bridge tagged=bridge,sfp01-lair,sfp03,sfp04,sfp05,sfp06,sfp07,sfp08,sfp09,sfp10,sfp11,sfp12,sfp13-den,sfp14-travel,sfp15-tankard,sfp16-switch1 untagged=\

    "vlan2001 - NW man,vlan180 -Transit,vlan6 Cheapconnect DSL" vlan-ids=6,100,148,149,150,151,155,158,159,160,165,170,178,180,181,199,640,1000,2000,2001,2002,2003,2004

add bridge=bridge tagged=bridge,sfp16-switch1 vlan-ids=300

/interface list member

add comment="Main LAN interface" interface="vlan180 -Transit" list=lan

add comment="VLAN Management interface " interface="vlan2001 - NW man" list=lan

add comment="Management port" interface=ether1 list=lan

add comment="DSL cheapconnect" interface=pppoe-DSL list=wan

add comment="T-mobile wan" interface="vlan300 Tmo-WAN" list=wan

add interface=sfp01-lair list=wan

/interface ovpn-server server

set auth=sha1,md5

and ip export:
[admin@Switch0-MikrotikCRS317] > /ip export 

# oct/21/2022 16:29:20 by RouterOS 7.6

# software id = DL3W-9T1J

#

# model = CRS317-1G-16S+

# serial number = 955C098CEE62

/ip pool

add name=dhcp_pool0 ranges=192.168.180.32-192.168.180.64

/ip dhcp-server

add address-pool=dhcp_pool0 interface="vlan180 -Transit" name=dhcp1

/ip address

add address=192.168.88.1/24 comment=defconf interface=bridge network=192.168.88.0

add address=172.16.1.14/24 interface="vlan2001 - NW man" network=172.16.1.0

add address=192.168.180.252/24 interface="vlan180 -Transit" network=192.168.180.0

/ip dhcp-client

add interface="vlan300 Tmo-WAN"

/ip dhcp-server lease

add address=192.168.180.16 client-id=ff:11:5a:de:fc:0:3:0:1:24:65:11:5a:de:fc mac-address=24:65:11:5A:DE:FC server=dhcp1

/ip dhcp-server network

add address=192.168.180.0/24 dns-server=1.1.1.1 gateway=192.168.180.252

/ip dns

set servers=172.16.1.1

/ip firewall address-list

add address=192.168.0.0/16 list=LAN-RFC1918

add address=172.16.0.0/16 list=LAN-RFC1918

add address=172.16.0.0/16 list=All-Lan-Addresses

add address=192.168.0.0/16 list=All-Lan-Addresses

add address=147.78.236.135 comment="Cheapconnect public IP" list="Public IP"

/ip neighbor discovery-settings

set discover-interface-list=!dynamic

/ip settings

set max-neighbor-entries=8192

/ip firewall filter

add action=fasttrack-connection chain=forward comment="fasttrack established, related, untracked" connection-state=established,related,untracked hw-offload=yes

add action=accept chain=input comment="Allow established, related, untracked" connection-state=established,related,untracked

add action=drop chain=input comment="Drop invalid" connection-state=invalid

add action=accept chain=input comment="Allow ICMP" protocol=icmp

add action=accept chain=input comment="Allow input from LAN IPs from LAN interfaces" in-interface-list=lan src-address-list=LAN-RFC1918

add action=drop chain=input comment="Explicitely drop all input traffic from PPPOE" in-interface-list=wan

add action=reject chain=input comment="Reject everything on input that is not specifically allowed" reject-with=icmp-network-unreachable

add action=accept chain=forward comment="Allow established, related, untracked" connection-state=established,related,untracked

add action=drop chain=forward comment="Drop invalid on forward" connection-state=invalid

add action=accept chain=forward comment="Allow LAN interface access to all local IPv4" disabled=yes dst-address-list=LAN-RFC1918 in-interface-list=lan

add action=accept chain=forward comment="Allow all traffic from LAN to WAN" in-interface-list=lan out-interface-list=wan

add action=accept chain=forward comment="Allow input traffic on port 5060 to Fritz" dst-address=192.168.180.16 dst-port=5060 in-interface-list=wan out-interface="vlan180 -Transit" protocol=udp

add action=drop chain=forward comment="Drop all not explicitly allowed traffic" log=yes log-prefix="Drop all other traffic"

/ip firewall mangle

add action=accept chain=prerouting routing-mark=ManagementIF src-address=172.16.1.14

/ip firewall nat

add action=dst-nat chain=dstnat dst-port=5060-5061 in-interface-list=wan log-prefix=SIP-5060 protocol=udp to-addresses=192.168.180.16 to-ports=5060

add action=dst-nat chain=dstnat dst-port=7078-7110 in-interface-list=wan log-prefix=RTP-Fritzbox protocol=udp to-addresses=192.168.180.16 to-ports=7078-7110

add action=masquerade chain=srcnat out-interface-list=wan src-address-list=LAN-RFC1918

/ip firewall service-port

set ftp disabled=yes

set tftp disabled=yes

set h323 disabled=yes

set pptp disabled=yes

/ip route

add check-gateway=ping disabled=no dst-address=172.16.0.0/16 gateway=192.168.180.1

add disabled=no dst-address=192.168.0.0/16 gateway=192.168.180.1

/ip service

set telnet disabled=yes

set ftp disabled=yes

set www address=192.168.0.0/16,172.16.0.0/16 disabled=yes

set ssh address=192.168.0.0/16,172.16.0.0/16

set www-ssl address=192.168.0.0/16,172.16.0.0/16 certificate=switch0.home.fazant.net.crt_0 disabled=no

set api disabled=yes

set winbox address=192.168.0.0/16,172.16.0.0/16 disabled=yes

set api-ssl disabled=yes

[admin@Switch0-MikrotikCRS317] >  
As I said before: this is probably something stupid I did. So far I managed to solve all my configuration issues myself, but this one really has me stumped...
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Fri Oct 21, 2022 6:02 pm

Try disabling Bridge Firewall and see if that helps:
/interface bridge settings
set use-ip-firewall=no use-ip-firewall-for-vlan=no 
 
paraplu
just joined
Posts: 14
Joined: Fri Sep 18, 2015 9:35 pm

Re: how does L3HW actually works?

Fri Oct 21, 2022 7:56 pm

Try upgrading to v7.6 - maybe you encountered some of the issues above.
Running on 7.6 already during the issues, on CRS309. I just tried again with non-tagged ports instead of tagged. And gave it some stress with 8 clients. Same issue: all worked perfectly for an hour or so. And then suddenly got a strange issue: connecting with winbox works, but all blank screens. Using another VLAN address to the CRS309 with winbox works, but a disconnect/reconnect again: no life anymore. Could this have something to do with ACL (mac) rules which I am using to allow my laptop through? Same ACL setting on a RB5009 works perfectly fine.
In addition I noticed that adding a ACL rule to block IGMP on the CRS309 blocks all! Same setting on a RB5009 works fine; does its job. Weird.

Anyway I will raise a ticket for this. No need to discuss this issue on this thread of "how does it work". My apologies.
 
teleweb
just joined
Posts: 7
Joined: Fri Jul 15, 2016 5:11 am

Re: how does L3HW actually works?

Fri Oct 21, 2022 10:26 pm

@Mikrotik
What's the progress on hw offloading vxlan?
Particularly on CRS317, we are looking to do a project with 100+ CRS317 switches, L3 HW, and VXLAN.
So the vxlan performance would need to be near wire speed.

Thanks!
 
llag
Frequent Visitor
Frequent Visitor
Posts: 72
Joined: Sat Aug 04, 2018 12:12 am

Re: how does L3HW actually works?

Fri Oct 21, 2022 11:18 pm

Try disabling Bridge Firewall and see if that helps:
/interface bridge settings
set use-ip-firewall=no use-ip-firewall-for-vlan=no 

I now have some funny results: with
/interface ethernet switch l3hw-settings
set fasttrack-hw=yes icmp-reply-on-error=yes ipv6-hw=no

pretty much no traffic passes from LAN <-> WAN vlans.
If I however switch global l3-hw-offloading OFF
interface ethernet switch
set 0 !cpu-flow-control l3-hw-offloading=no mirror-source=none mirror-target=none name=switch1
all of a sudden I get near line speed NAT (I have a 1Gbs WAN and I reach that).

I do not see the H flag in the firewall connection table however:
Flags: S - SEEN-REPLY; A - ASSURED; C - CONFIRMED; F - FASTTRACK; s - SRCNAT; d - DSTNAT
Columns: PROTOCOL, SRC-ADDRESS, DST-ADDRESS, TCP-STATE, TIMEOUT, ORIG-RATE, REPL-RATE, ORIG-PACKETS, REPL-PACKETS, ORIG-BYTES, REPL-BYTES
  #        PROTOCOL  SRC-ADDRESS            DST-ADDRESS           TCP-STATE    TIMEOUT    ORIG-RATE  REPL-RATE  ORIG-PACKETS  REPL-P  ORIG-BYTES  REPL-BYTES
  0 S C    icmp      192.168.180.252        192.168.180.1                      4s         0bps       0bps                  1       1          56          56
  1 SACFs  tcp       192.168.148.242:38158  23.216.254.231:443    established  23h58m48s  0bps       0bps                 40      70       3 448      96 342
  2 SACFs  tcp       192.168.148.239:48450  95.101.78.170:443     established  23h54m     0bps       0bps                 11      12       2 291       8 675
  3 SACFs  tcp       192.168.148.239:37266  104.73.152.79:443     established  23h53m57s  0bps       0bps                 10      10       2 015       6 034
  4 SACFs  tcp       192.168.159.6:42612    104.26.15.92:443      established  23h59m37s  0bps       0bps                 28      29       5 190       6 731
  5 SACFs  tcp       192.168.148.239:44780  104.17.73.14:443      established  23h54m2s   0bps       0bps                 14      18       1 859      16 100
  6 SACFs  tcp       192.168.148.239:48278  151.101.193.253:443   established  23h54m1s   0bps       0bps                 12      13       2 009       6 786
  7 SACFs  tcp       192.168.148.239:49404  151.101.66.49:443     established  23h54m2s   0bps       0bps                 13      14       2 955       6 909
  8 S CF d udp       92.204.144.22:7025     87.212.48.242:5060                 5m28s      0bps       0bps                  1       1         440         364
  9 SACFs  tcp       192.168.159.6:36812    34.120.208.123:443    established  23h59m3s   0bps       0bps                 16      14       3 872       2 307
 10 SACFs  tcp       192.168.148.242:37098  157.240.201.61:443    established  23h58m32s  0bps       0bps                272     275      24 308      22 125
 11 SACFs  tcp       192.168.148.246:50602  142.250.27.188:5228   established  23h48m16s  0bps       0bps                 16      18       2 065       9 872
 12 SACFs  tcp       192.168.148.242:48244  145.221.37.237:443    established  23h58m43s  0bps       0bps                 10      10       4 934       2 576
 13 SAC    tcp       192.168.159.6:60530    172.16.1.14:22        established  23h59m59s  15.9kbps   153.3kbps           791     645      55 441     143 000
Anyhow, I get full speed now, but not in the prescribed manner?
I wonder if this is a relict of the introduction of the /interface ethernet switch l3hw-settings options?
 
mfedotov
just joined
Posts: 18
Joined: Mon Oct 25, 2021 3:32 am

Re: how does L3HW actually works?

Tue Oct 25, 2022 7:03 pm

If I however switch global l3-hw-offloading OFF
interface ethernet switch
set 0 !cpu-flow-control l3-hw-offloading=no mirror-source=none mirror-target=none name=switch1
all of a sudden I get near line speed NAT (I have a 1Gbs WAN and I reach that).

I do not see the H flag in the firewall connection table however:

I think CRS3xx can do almost 1Gb/s on CPU. I suggest that you monitor the CPU utilization with '/system/resource/monitor' during your tests to confirm...
 
llag
Frequent Visitor
Frequent Visitor
Posts: 72
Joined: Sat Aug 04, 2018 12:12 am

Re: how does L3HW actually works?

Wed Oct 26, 2022 11:10 pm

I will need to test this tomorrow or when I have time. You are probably quite right. When I did a quick check I indeed noticed a load on one core of upto 80%. I did not think about it more as I did not realize that the CRS317 might do 1Gbs as I only looked at the 512 bytes result for 25 rules (423.5 Mbs). So that indeed suggest that L3 still does not work for me. @raimondsp is there more information I need to provide to get L3 forwarding to work for me?
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Mon Oct 31, 2022 9:56 am

I will need to test this tomorrow or when I have time. You are probably quite right. When I did a quick check I indeed noticed a load on one core of upto 80%. I did not think about it more as I did not realize that the CRS317 might do 1Gbs as I only looked at the 512 bytes result for 25 rules (423.5 Mbs). So that indeed suggest that L3 still does not work for me. @raimondsp is there more information I need to provide to get L3 forwarding to work for me?

Please create a support ticket, and we will try to reproduce your issue on our side.
 
S8T8
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Thu Sep 15, 2022 7:15 pm

Re: how does L3HW actually works?

Tue Nov 15, 2022 4:29 am

Hello, a little bit dumb question...
In a basic SOHO enviroment, is L3HW useful only when multiple subnets, firewall rules or routes are involved?
As example in a CRS switch L3HW is supported but not usefu (in most of cases), correct?
 
biomesh
Long time Member
Long time Member
Posts: 561
Joined: Fri Feb 10, 2012 8:25 pm

Re: how does L3HW actually works?

Tue Nov 15, 2022 4:04 pm

For soho, if you only have one subnet and you are not using the switch as your main router, then there really is no benefit.

If you have multiple subnets/vlans on your network but use a separate router, then yes there is a definite benefit.

If you use your crs device as a router, it might be better than your existing/external router - but you will probably run into cpu issues due to the lower end cpus used in the switches.

I have my crs317 handle all of the ipv4 and ipv6 on my network to get full 10gig speed between devices and have a ccr to handle the routing out to the Internet, dhcp, vpn, etc.
 
S8T8
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Thu Sep 15, 2022 7:15 pm

Re: how does L3HW actually works?

Tue Nov 15, 2022 8:42 pm

I'm using a MT router for handling PPPoE, couple of VLANs, Firewall, DHCP, etc... a CRS device wtih the trunk port to the router, connected devices are separated from VLAN A and VLAN B.
From what I understand using L3HW is ignoring firewall roules, but those roules are "managed" by the router, in fact activating L3HW to the switch, VLAN A is still separated from VLAN B.

Interesting approach with your CRS317, could you please explain with a quick example?
 
biomesh
Long time Member
Long time Member
Posts: 561
Joined: Fri Feb 10, 2012 8:25 pm

Re: how does L3HW actually works?

Tue Nov 15, 2022 9:44 pm

Here is a snippet - not going into the actual enabling of l3hw as that is in the manual/wiki. This should be most of it to understand how I use it.

CRS317
/ip address
add address=192.168.6.1/23 interface=vlan600 network=192.168.6.0
add address=192.168.5.1/24 interface=vlan500 network=192.168.5.0
add address=192.168.0.2/24 interface=vlan10 network=192.168.0.0
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.0.1 pref-src=0.0.0.0 routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
/ip dhcp-relay
add dhcp-server=192.168.0.1 disabled=no interface=vlan600 local-address=192.168.6.1 name=vlan600
add dhcp-server=192.168.0.1 disabled=no interface=vlan500 local-address=192.168.5.1 name=vlan500
/ipv6 address
add address=fd00::2 advertise=no interface=vlan10
add address=fd00:5::1 interface=vlan500
add address=fd00:6::1 interface=vlan600
add from-pool=from_comcast_vlan500 interface=vlan500
add from-pool=from_comcast_vlan600 interface=vlan600
/ipv6 dhcp-client
add interface=vlan600 pool-name=from_comcast_vlan600 request=prefix use-peer-dns=no
add interface=vlan500 pool-name=from_comcast_vlan500 request=prefix use-peer-dns=no
/ipv6 route
add disabled=no dst-address=::/0 gateway=fd00::1 routing-table=main
router
/ip dhcp-server
add address-pool=vlan500pool  interface=vlan10 lease-time=5d name="vlan500 dhcp" relay=192.168.5.1 use-framed-as-classless=no
add address-pool=vlan600pool interface=vlan10 lease-time=2w1d name="vlan600 dhcp" relay=192.168.6.1 use-framed-as-classless=no
/ip address
add address=192.168.0.1/24 interface=vlan10 network=192.168.0.0
/ip route
add disabled=no distance=1 dst-address=192.168.6.0/23 gateway=192.168.0.2 pref-src=0.0.0.0 routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=192.168.5.0/24 gateway=192.168.0.2 pref-src=0.0.0.0 routing-table=main scope=30 \
    suppress-hw-offload=no target-scope=10
/ipv6 dhcp-client
add add-default-route=yes interface=wan pool-name=comcast_ipv6 prefix-hint=::/60 request=address,prefix use-peer-dns=no    
/ipv6 dhcp-server
add address-pool=comcast_ipv6 interface=vlan600 lease-time=12h name=server600
add address-pool=comcast_ipv6 interface=vlan500 lease-time=12h name=server500
/ipv6 address
add address=fd00::1 interface=vlan10
 
S8T8
Frequent Visitor
Frequent Visitor
Posts: 81
Joined: Thu Sep 15, 2022 7:15 pm

Re: how does L3HW actually works?

Tue Nov 15, 2022 9:49 pm

Thanks @biomesh, really appreciated!
 
ajgnet
newbie
Posts: 35
Joined: Wed Apr 27, 2022 1:57 am

Re: how does L3HW actually works?

Sun Dec 11, 2022 6:44 pm

/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=no protocol=udp
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
add action=accept chain=forward connection-state=established,related
^ do not repeat that at home, or your kid will complain about missing shots in Counter-Strike due to latency /s
Question: why would the above would negatively impact latency?
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 298
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: how does L3HW actually works?

Mon Dec 12, 2022 12:34 am

/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=no protocol=udp
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
add action=accept chain=forward connection-state=established,related
^ do not repeat that at home, or your kid will complain about missing shots in Counter-Strike due to latency /s
Question: why would the above would negatively impact latency?
Note the "/s" at the end of the line. It's sarcasm. He posted an example of how you could choose to accelerate NAT for one protocol and not for another, based on your network's needs.

In the vast majority of use cases, you'd want to offload as much to the hardware as possible, and only CPU-route packets going directly to the router, or traffic that particular chipsets can't handle properly.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: how does L3HW actually works?

Mon Dec 12, 2022 1:24 am

In the vast majority of use cases, you'd want to offload as much to the hardware as possible, and only CPU-route packets going directly to the router, or traffic that particular chipsets can't handle properly.

i dont think so

L3HW Device Support dictates a maximum Fasttrack Connections around 4.5k

4.5k is easily achievable in many scenarios

in my home i easily reach 2.0k Connections with a few devices, beacuse of that can be better for example to offload only "big" connections with the most ammount of badwidth/traffic, does not make too much sense to offload short lived connections with only a few packets, CPU can cope whit them without problem leaving valuable fasttrack connection offload "slots" for the "heavy" traffic

in each scenario the strategy can be slightly different, but in the most cases you will have to do a good selection of the traffic to be offloaded

of course, offload of Fasttrack Connections in some cases with too many connections will not make any sense
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Mon Dec 26, 2022 4:12 am

I read the thread, but still have some doubts.

Let's say I have a single CCR2216-1G-12XS-2XQ unit, whereby I religiously follow the proper bridge configuration to ensure hardware offloading etc. And there is no connection_tracking/NAT. I'm assuming BGP affinity for input/output is set to “alone” per peer.

Let's say I have two 25G transit from two separate transit providers and both provide me full tables for IPv4 + IPv6. This means two full IPv4 tables and IPv6 tables which exceeds the “memory” limits of the switch chip/ASIC. Now let's say, I use a QSFP28 port for my internal network that where both egress/ingress traffic is averaging 40 Gigs from thousands of hosts/customers.

So Questions:
1. How does RouterOS v7.6-7.7 select what routes to offload to ASIC, and what routes to go via CPU?
2. How does complex routing filters affect the algorithm for route offloading selection? Can we influence the algorithm?
3. Assuming the network is fully 1500 MTU capable (because both transit providers are limited to that), what CPU usage would be the expected average for 40-50Gigs traffic?
4. What happens if I add another 25G transit provider with full tables for both IPv4 and IPv6? Will CPU start choking?
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Tue Dec 27, 2022 7:51 am

I read the thread, but still have some doubts.

Let's say I have a single CCR2216-1G-12XS-2XQ unit, whereby I religiously follow the proper bridge configuration to ensure hardware offloading etc. And there is no connection_tracking/NAT. I'm assuming BGP affinity for input/output is set to “alone” per peer.

Let's say I have two 25G transit from two separate transit providers and both provide me full tables for IPv4 + IPv6. This means two full IPv4 tables and IPv6 tables which exceeds the “memory” limits of the switch chip/ASIC. Now let's say, I use a QSFP28 port for my internal network that where both egress/ingress traffic is averaging 40 Gigs from thousands of hosts/customers.

So Questions:
1. How does RouterOS v7.6-7.7 select what routes to offload to ASIC, and what routes to go via CPU?
2. How does complex routing filters affect the algorithm for route offloading selection? Can we influence the algorithm?
3. Assuming the network is fully 1500 MTU capable (because both transit providers are limited to that), what CPU usage would be the expected average for 40-50Gigs traffic?
4. What happens if I add another 25G transit provider with full tables for both IPv4 and IPv6? Will CPU start choking?

  1. If the entire routing table cannot fit in the hardware memory, routes with longer prefixes are offloaded to the switch chip while the shorter prefixes are left to the CPU.
  2. HW offloading of particular routes can be suppressed via routing filters (documentation)
  3. CPU usage depends on the software/hardware routing proportion. The more traffic is processed by the switch chip, the less CPU usage. So it can be anywhere in 0..100%
  4. Adding more routes means less (%) of total routes can be offloaded and, therefore, more CPU usage. However, you can fine-tune what routes are getting hw-offloaded (#2) to ensure that the heavy-duty routes are processed by the ASIC.
 
User avatar
nz_monkey
Forum Guru
Forum Guru
Posts: 2095
Joined: Mon Jan 14, 2008 1:53 pm
Location: Over the Rainbow
Contact:

Re: how does L3HW actually works?

Tue Dec 27, 2022 11:07 am

I have a dream of a Mikrotik router with hardware forwarding tables large enough to hold multiple full BGP tables.

One day soon I hope this will be reality !
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: how does L3HW actually works?

Tue Dec 27, 2022 3:59 pm

I have a dream of a Mikrotik router with hardware forwarding tables large enough to hold multiple full BGP tables.

One day soon I hope this will be reality !
even with other vendors that is difficult to achieve and quite expensive
 
User avatar
clambert
Member Candidate
Member Candidate
Posts: 120
Joined: Wed Jun 12, 2019 5:04 am

Re: how does L3HW actually works?

Tue Dec 27, 2022 4:25 pm

There are equipment with prices an order of magnitude higher that support in the order of 10,000 routes.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Tue Dec 27, 2022 7:20 pm

I have a dream of a Mikrotik router with hardware forwarding tables large enough to hold multiple full BGP tables.

One day soon I hope this will be reality !
even with other vendors that is difficult to achieve and quite expensive
It wouldn't be so difficult if MirkoTik and other proprietary vendors embrace XDP and/or DPDK for packet assembly, packet filtering, “route” offloading etc. And built hardware that is out-of-the-box compatible with XDP hardware offloading.

Even if it's not ASIC offloaded, such traffic using DPDK/XDP will be processed *before* sk_buff, therefore minimal CPU usage.

But no, let's continue using the bottlenecked net filter framework.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Tue Dec 27, 2022 7:22 pm

  1. If the entire routing table cannot fit in the hardware memory, routes with longer prefixes are offloaded to the switch chip while the shorter prefixes are left to the CPU.
  2. HW offloading of particular routes can be suppressed via routing filters (documentation)
  3. CPU usage depends on the software/hardware routing proportion. The more traffic is processed by the switch chip, the less CPU usage. So it can be anywhere in 0..100%
  4. Adding more routes means less (%) of total routes can be offloaded and, therefore, more CPU usage. However, you can fine-tune what routes are getting hw-offloaded (#2) to ensure that the heavy-duty routes are processed by the ASIC.
Is there a way to tell the algorithm "Hey I want you to offload only routes learnt from provider AS123 where the origin AS is AS563"?

It seems the way MikroTik is going about this, isn't ideal for heavy-duty full table use cases.
 
Guscht
Member Candidate
Member Candidate
Posts: 236
Joined: Thu Jul 01, 2010 5:32 pm

Re: how does L3HW actually works?

Fri Jan 06, 2023 5:22 pm

A question which is still not clarified for me.
We need IP/Firewall/Filter, NAT, Mangle, RAW + Bridge/Filter, NAT + Simple Queues. I assume from what I have read so far, L3 HW-Offload ist not achievable with this needs?
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Fri Jan 06, 2023 9:35 pm

A question which is still not clarified for me.
We need IP/Firewall/Filter, NAT, Mangle, RAW + Bridge/Filter, NAT + Simple Queues. I assume from what I have read so far, L3 HW-Offload ist not achievable with this needs?
You can offload some but all the traffic when queues are in play using FastTrack, but you need to implement it correctly while allowing queues to work as you intended. Some design and planning would be required.

NAT can be offloaded. RAW is prerouting chain, before conn_track, but it is after sk_buff, so it cannot be offloaded unless MikroTik supports XDP (with NIC/driver offloading). Filter table is after conn_track, by definition, it can never be offloaded, as state tracking is always CPU not ASIC.

Bridge filter, I think it's not much of an impact in conjunction with Bridge FastFoward/FastPath, I may be wrong. You should make use of STP parameters and bridge parameters to try and do what you want without bridge filters as much as possible.
 
blacksnow
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Feb 15, 2023 4:46 pm

Re: how does L3HW actually works?

Thu Feb 16, 2023 3:23 am

Sorry for hoping on an old thread, but I'm running the CCR2216 and also experience issues similar to this for reference here is my issue described a bit more in detail.

I'm using the CCR2216-1G-12XS-2XQ and I've setup vlans on a single bridge interface as documented. I have L3HW offload working but it only works in some directions and I just want to make sure this is expected functionality.

Router Version: 7.8rc1

Topology:

Single bridge
VLAN 3999 = WAN (On SFP28-12) (bridge and sfp28-12 are tagged)
VLAN 1-7 = LAN (On SFP28-5) (bridge and sfp28-5 are tagged)
SFP28-1 = LAN network with a server/workstation.

Current Functionality:
Going from SFP28-1 to the WAN or LAN on sfp28-12 or sfp28-5 is HW offloaded as expected. Bi-Directional.
Going from SFP28-5 (any VLAN) to SFP28-12 (WAN) is not offloaded.


So essentially, when the packet originates from a vlan that is coming through an interface that is on the bridge and is leaving through another interface on the bridge which is a different VLAN, it is not offloaded. If the packet originates from the a vlan on the bridge and goes to some other interface not on the bridge then it is offloaded and vice versa in terms of direction, for example sfp28-1 which is not on the bridge to any vlan on the bridge is offload. Is this expected?
When WAN has vlans & LAN has vlans. How to configure Fasttrack Connections Offloading + NAT ?
Below configurations are not working properly
/interface bridge
add ingress-filtering=no name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=vlan101 vlan-id=101
add interface=bridge1 name=vlan102 vlan-id=102
add interface=bridge1 name=vlan103 vlan-id=103
add interface=bridge1 name=vlan201 vlan-id=201
add interface=bridge1 name=vlan202 vlan-id=202
add interface=bridge1 name=vlan203 vlan-id=203
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 1 l3-hw-offloading=no
set 2 l3-hw-offloading=no
set 3 l3-hw-offloading=no
set 4 l3-hw-offloading=no
set 5 l3-hw-offloading=no
set 6 l3-hw-offloading=no
set 7 l3-hw-offloading=no
set 8 l3-hw-offloading=no
set 9 l3-hw-offloading=no
set 10 l3-hw-offloading=no
set 11 l3-hw-offloading=no
set 12 l3-hw-offloading=no
set 13 l3-hw-offloading=no
set 14 l3-hw-offloading=no
set 15 l3-hw-offloading=no
set 16 l3-hw-offloading=no
set 17 l3-hw-offloading=no
set 18 l3-hw-offloading=no
set 19 l3-hw-offloading=no
/interface bridge port
add bridge=bridge1 interface=qsfp28-1-1
add bridge=bridge1 interface=qsfp28-2-1
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,qsfp28-1-1 vlan-ids=100-110
add bridge=bridge1 tagged=bridge1,qsfp28-2-1 vlan-ids=200-210
/ip address
add address=192.168.11.1/24 interface=vlan101 network=192.168.11.0
add address=192.168.21.1/24 interface=vlan201 network=192.168.21.0
add address=192.168.22.1/24 interface=vlan202 network=192.168.22.0
add address=192.168.12.1/24 interface=vlan102 network=192.168.12.0
add address=192.168.13.1/24 interface=vlan103 network=192.168.13.0
add address=192.168.23.1/24 interface=vlan203 network=192.168.23.0
/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes protocol=tcp
add action=accept chain=forward connection-state=established,related
/ip firewall nat
add action=masquerade chain=srcnat out-interface=vlan101
add action=masquerade chain=srcnat out-interface=vlan102
add action=masquerade chain=srcnat out-interface=vlan103
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Sat Feb 18, 2023 5:54 pm

Sorry for hoping on an old thread, but I'm running the CCR2216 and also experience issues similar to this for reference here is my issue described a bit more in detail.

I'm using the CCR2216-1G-12XS-2XQ and I've setup vlans on a single bridge interface as documented. I have L3HW offload working but it only works in some directions and I just want to make sure this is expected functionality.

Router Version: 7.8rc1

Topology:

Single bridge
VLAN 3999 = WAN (On SFP28-12) (bridge and sfp28-12 are tagged)
VLAN 1-7 = LAN (On SFP28-5) (bridge and sfp28-5 are tagged)
SFP28-1 = LAN network with a server/workstation.

Current Functionality:
Going from SFP28-1 to the WAN or LAN on sfp28-12 or sfp28-5 is HW offloaded as expected. Bi-Directional.
Going from SFP28-5 (any VLAN) to SFP28-12 (WAN) is not offloaded.


So essentially, when the packet originates from a vlan that is coming through an interface that is on the bridge and is leaving through another interface on the bridge which is a different VLAN, it is not offloaded. If the packet originates from the a vlan on the bridge and goes to some other interface not on the bridge then it is offloaded and vice versa in terms of direction, for example sfp28-1 which is not on the bridge to any vlan on the bridge is offload. Is this expected?
When WAN has vlans & LAN has vlans. How to configure Fasttrack Connections Offloading + NAT ?
Below configurations are not working properly
/interface bridge
add ingress-filtering=no name=bridge1 vlan-filtering=yes
/interface vlan
add interface=bridge1 name=vlan101 vlan-id=101
add interface=bridge1 name=vlan102 vlan-id=102
add interface=bridge1 name=vlan103 vlan-id=103
add interface=bridge1 name=vlan201 vlan-id=201
add interface=bridge1 name=vlan202 vlan-id=202
add interface=bridge1 name=vlan203 vlan-id=203
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 1 l3-hw-offloading=no
set 2 l3-hw-offloading=no
set 3 l3-hw-offloading=no
set 4 l3-hw-offloading=no
set 5 l3-hw-offloading=no
set 6 l3-hw-offloading=no
set 7 l3-hw-offloading=no
set 8 l3-hw-offloading=no
set 9 l3-hw-offloading=no
set 10 l3-hw-offloading=no
set 11 l3-hw-offloading=no
set 12 l3-hw-offloading=no
set 13 l3-hw-offloading=no
set 14 l3-hw-offloading=no
set 15 l3-hw-offloading=no
set 16 l3-hw-offloading=no
set 17 l3-hw-offloading=no
set 18 l3-hw-offloading=no
set 19 l3-hw-offloading=no
/interface bridge port
add bridge=bridge1 interface=qsfp28-1-1
add bridge=bridge1 interface=qsfp28-2-1
/interface bridge vlan
add bridge=bridge1 tagged=bridge1,qsfp28-1-1 vlan-ids=100-110
add bridge=bridge1 tagged=bridge1,qsfp28-2-1 vlan-ids=200-210
/ip address
add address=192.168.11.1/24 interface=vlan101 network=192.168.11.0
add address=192.168.21.1/24 interface=vlan201 network=192.168.21.0
add address=192.168.22.1/24 interface=vlan202 network=192.168.22.0
add address=192.168.12.1/24 interface=vlan102 network=192.168.12.0
add address=192.168.13.1/24 interface=vlan103 network=192.168.13.0
add address=192.168.23.1/24 interface=vlan203 network=192.168.23.0
/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes protocol=tcp
add action=accept chain=forward connection-state=established,related
/ip firewall nat
add action=masquerade chain=srcnat out-interface=vlan101
add action=masquerade chain=srcnat out-interface=vlan102
add action=masquerade chain=srcnat out-interface=vlan103
WAN interface should not be in a bridge. Make it plain Ethernet port.

It should be on a bridge only if a single port is connected to a switch where both WAN and LAN VLANs are configured as trunk.
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 298
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: how does L3HW actually works?

Sat Feb 18, 2023 6:17 pm

On my 2116's, once I load up full routes from two providers, it mentions the HW table is full and it only keeps /25's or larger. The log shows something like 45 routes, although I have a hard time believing that there are only 45 /25's or larger in a table with 1.4M routes. It would be nice to have a quick visual as to how many routes are actually offloaded and how many are being processed by software. And instead of having a way to keep them out of HW tables, I'd like a way to encourage the system to keep certain routes in HW tables. I was under the impression that the algorithm switches less-used routes out of HW for those more commonly used.

I peer with three providers, two on one router and one on the other. A third router in the middle combines what it learns from the other two. To help make them fit, I set up an input filter allowing just 1-2 AS paths deep. For the most part that works well on 7.4.1 and the switch load drops to 5% on the borders (all of it going to routing processes) and 0% on the center router.

With releases after 7.4.1, I started seeing issues with L3HW offload getting out of sync with the routing table, so I've disabled it for now. We're not pushing enough traffic for the routers to go above 20%, and I prefer the improvements to BGP management offered by 7.5-7.7.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Sat Feb 18, 2023 6:30 pm

On my 2116's, once I load up full routes from two providers, it mentions the HW table is full and it only keeps /25's or larger. The log shows something like 45 routes, although I have a hard time believing that there are only 45 /25's or larger in a table with 1.4M routes. It would be nice to have a quick visual as to how many routes are actually offloaded and how many are being processed by software. And instead of having a way to keep them out of HW tables, I'd like a way to encourage the system to keep certain routes in HW tables. I was under the impression that the algorithm switches less-used routes out of HW for those more commonly used.

I peer with three providers, two on one router and one on the other. A third router in the middle combines what it learns from the other two. To help make them fit, I set up an input filter allowing just 1-2 AS paths deep. For the most part that works well on 7.4.1 and the switch load drops to 5% on the borders (all of it going to routing processes) and 0% on the center router.

With releases after 7.4.1, I started seeing issues with L3HW offload getting out of sync with the routing table, so I've disabled it for now. We're not pushing enough traffic for the routers to go above 20%, and I prefer the improvements to BGP management offered by 7.5-7.7.
I agree, that MikroTik is lacking visibility. Hard to know what's going where. And even the prefix count is still broken as of 7.7 stable.
 
blacksnow
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Feb 15, 2023 4:46 pm

Re: how does L3HW actually works?

Sun Feb 19, 2023 1:58 am

In my network topology, I have a managed switch that has access ports to some workstations and then I also have an access port for WAN (because I don't want to use a Ethernet to SFP+ module). So some networks are on VLAN 1-7 and then the WAN is on VLAN 4000. There are two uplinks from the single switch into the CCR router, trunk port one (sfp28-5) carries the tagged VLAN's 1-7 and then trunk port two (sfp28-12) carries the WAN traffic only.
WAN interface should not be in a bridge. Make it plain Ethernet port.

It should be on a bridge only if a single port is connected to a switch where both WAN and LAN VLANs are configured as trunk.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Sun Feb 19, 2023 2:25 am

In my network topology, I have a managed switch that has access ports to some workstations and then I also have an access port for WAN (because I don't want to use a Ethernet to SFP+ module). So some networks are on VLAN 1-7 and then the WAN is on VLAN 4000. There are two uplinks from the single switch into the CCR router, trunk port one (sfp28-5) carries the tagged VLAN's 1-7 and then trunk port two (sfp28-12) carries the WAN traffic only.
Okay, in this case on CCR side sfp28-5 and everything else will be in bridge as usual. sfp28-12 will be outside the bridge on the CCR side.

On the switch side, are you using MikroTik? If so, all ports will be on bridge as well on that side. And you should enable ingress filtering on both bridge and port to prevent possible leaks or misconfig or BUM traffic from leaking into the ports.
 
blacksnow
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Feb 15, 2023 4:46 pm

Re: how does L3HW actually works?

Sun Feb 19, 2023 2:57 am

Ok just to make sure I understand this: I have sfp28-1 through sfp28-4 as LAN networks (no VLANs), then sfp28-5 (VLAN 1-7 for some other LAN networks) and sfp28-12 (VLAN 4000 for WAN). According to MikroTik https://help.mikrotik.com/docs/display/ ... LANRouting we should not attach VLANs directly to the interfaces, all VLAN's should be on a bridge to take advantage of L3HW offloading and then just using IP firewall and such to enforce rules.

You are suggesting either:

A) Put interfaces sfp28 1-5 on a bridge, and put a VLAN directly on interface sfp28-12.
B) Don't touch sfp28 1-4, only put sfp28-5 on a bridge, and put a VLAN directly on interface sfp28-12.

Just to re-iterate in my tests, going from VLAN 1 (LAN) on sfp28-5 to VLAN 4000 (WAN) on sfp28-12 does not appear to be offloaded. But going from VLAN 1 on sfp28-5 to sfp28-1 (A non-VLAN network) is offloaded, and going from sfp28-1(A non-VLAN network) to either VLAN 1 on sfp28-5 or VLAN 4000 on sfp28-12 is offloaded. Both sfp28-5 and sfp28-12 are on the bridge as their own bridge ports and associated tagged VLANs.
Okay, in this case on CCR side sfp28-5 and everything else will be in bridge as usual. sfp28-12 will be outside the bridge on the CCR side.

On the switch side, are you using MikroTik? If so, all ports will be on bridge as well on that side. And you should enable ingress filtering on both bridge and port to prevent possible leaks or misconfig or BUM traffic from leaking into the ports.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Sun Feb 19, 2023 4:07 am

Ok just to make sure I understand this: I have sfp28-1 through sfp28-4 as LAN networks (no VLANs), then sfp28-5 (VLAN 1-7 for some other LAN networks) and sfp28-12 (VLAN 4000 for WAN). According to MikroTik https://help.mikrotik.com/docs/display/ ... LANRouting we should not attach VLANs directly to the interfaces, all VLAN's should be on a bridge to take advantage of L3HW offloading and then just using IP firewall and such to enforce rules.

You are suggesting either:

A) Put interfaces sfp28 1-5 on a bridge, and put a VLAN directly on interface sfp28-12.
B) Don't touch sfp28 1-4, only put sfp28-5 on a bridge, and put a VLAN directly on interface sfp28-12.

Just to re-iterate in my tests, going from VLAN 1 (LAN) on sfp28-5 to VLAN 4000 (WAN) on sfp28-12 does not appear to be offloaded. But going from VLAN 1 on sfp28-5 to sfp28-1 (A non-VLAN network) is offloaded, and going from sfp28-1(A non-VLAN network) to either VLAN 1 on sfp28-5 or VLAN 4000 on sfp28-12 is offloaded. Both sfp28-5 and sfp28-12 are on the bridge as their own bridge ports and associated tagged VLANs.
You're overcomplicating this discussion. CCR side, WAN PHYSICAL port, remove from bridge, attach VLAN directly to it. For EVERYTHING else physical port, leave it in the bridge as is.
 
blacksnow
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Feb 15, 2023 4:46 pm

Re: how does L3HW actually works?

Sun Feb 19, 2023 5:25 am

Ok I don't think we are on the same page. Posting my export so you can get a better idea of whats going on, for brevity I have removed all other interfaces to avoid confusion. My ultimate goal is to get traffic offloaded between VLANs regardless if one VLAN is carrying WAN traffic it is still considered a VLAN. So that means traffic from VLAN 3,4 or 6 to and from VLAN 4000 (and any other distinct combination like VLAN 6 to VLAN 4 etc..) should be able to be offloaded with firewall-compatible Inter-VLAN routing. I have a CCR2216 and it is able to offload inter-vlan forwarding with L3HW offload when setup per Mikrotik documentation. That is not occuring here.
/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge pvid=4094 vlan-filtering=\
    yes
/interface vlan
add interface=bridge name="IoT Wifi" vlan-id=6
add interface=bridge name="Mobile Wifi" vlan-id=4
add interface=bridge name="Secure Wifi" vlan-id=3
add interface=bridge name="WAN (4000)" vlan-id=4000
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 1 l3-hw-offloading=no
set 2 l3-hw-offloading=no
set 3 l3-hw-offloading=no
set 4 l3-hw-offloading=no
set 5 l3-hw-offloading=no
set 6 l3-hw-offloading=no
set 7 l3-hw-offloading=no
set 8 l3-hw-offloading=no
set 9 l3-hw-offloading=no
set 10 l3-hw-offloading=no
set 11 l3-hw-offloading=no
set 12 l3-hw-offloading=no
set 13 l3-hw-offloading=no
set 14 l3-hw-offloading=no
set 15 l3-hw-offloading=no
set 16 l3-hw-offloading=no
set 17 l3-hw-offloading=no
set 18 l3-hw-offloading=no
set 19 l3-hw-offloading=no
/interface bridge port
add bridge=bridge frame-types=admit-only-vlan-tagged interface=sfp28-12 pvid=\
    4094
add bridge=bridge frame-types=admit-only-vlan-tagged interface=sfp28-5 pvid=\
    4094
/interface bridge settings
set use-ip-firewall=yes use-ip-firewall-for-pppoe=yes \
    use-ip-firewall-for-vlan=yes
/interface ethernet switch l3hw-settings
set ipv6-hw=yes
/interface bridge vlan
add bridge=bridge tagged=bridge,sfp28-12 vlan-ids=4000
add bridge=bridge tagged=bridge,sfp28-5 vlan-ids=3-4,6
Please see the info from Raimondsp above (quoted the important part here below).

Before v7.2, RouterOS didn't support FastPath on vlan-filtered bridges, meaning no FastTrack for Inter-VLAN routing and no HW offloading. The feature has been introduced in RouterOS v7.2rc2, allowing Firewall-Compatible Inter-VLAN Routing.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Sun Feb 19, 2023 6:39 am

Ok I don't think we are on the same page. Posting my export so you can get a better idea of whats going on, for brevity I have removed all other interfaces to avoid confusion. My ultimate goal is to get traffic offloaded between VLANs regardless if one VLAN is carrying WAN traffic it is still considered a VLAN. So that means traffic from VLAN 3,4 or 6 to and from VLAN 4000 (and any other distinct combination like VLAN 6 to VLAN 4 etc..) should be able to be offloaded with firewall-compatible Inter-VLAN routing. I have a CCR2216 and it is able to offload inter-vlan forwarding with L3HW offload when setup per Mikrotik documentation. That is not occuring here.
/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge pvid=4094 vlan-filtering=\
    yes
/interface vlan
add interface=bridge name="IoT Wifi" vlan-id=6
add interface=bridge name="Mobile Wifi" vlan-id=4
add interface=bridge name="Secure Wifi" vlan-id=3
add interface=bridge name="WAN (4000)" vlan-id=4000
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 1 l3-hw-offloading=no
set 2 l3-hw-offloading=no
set 3 l3-hw-offloading=no
set 4 l3-hw-offloading=no
set 5 l3-hw-offloading=no
set 6 l3-hw-offloading=no
set 7 l3-hw-offloading=no
set 8 l3-hw-offloading=no
set 9 l3-hw-offloading=no
set 10 l3-hw-offloading=no
set 11 l3-hw-offloading=no
set 12 l3-hw-offloading=no
set 13 l3-hw-offloading=no
set 14 l3-hw-offloading=no
set 15 l3-hw-offloading=no
set 16 l3-hw-offloading=no
set 17 l3-hw-offloading=no
set 18 l3-hw-offloading=no
set 19 l3-hw-offloading=no
/interface bridge port
add bridge=bridge frame-types=admit-only-vlan-tagged interface=sfp28-12 pvid=\
    4094
add bridge=bridge frame-types=admit-only-vlan-tagged interface=sfp28-5 pvid=\
    4094
/interface bridge settings
set use-ip-firewall=yes use-ip-firewall-for-pppoe=yes \
    use-ip-firewall-for-vlan=yes
/interface ethernet switch l3hw-settings
set ipv6-hw=yes
/interface bridge vlan
add bridge=bridge tagged=bridge,sfp28-12 vlan-ids=4000
add bridge=bridge tagged=bridge,sfp28-5 vlan-ids=3-4,6
Please see the info from Raimondsp above (quoted the important part here below).

Before v7.2, RouterOS didn't support FastPath on vlan-filtered bridges, meaning no FastTrack for Inter-VLAN routing and no HW offloading. The feature has been introduced in RouterOS v7.2rc2, allowing Firewall-Compatible Inter-VLAN Routing.
We're on the same page but you keep going in circles. Remove sfp28-12 from the bridge. Make it a standalone Ethernet port. Put VLAN 4000 directly on it.

I mean, just test it. What have you got to lose?
 
blacksnow
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Feb 15, 2023 4:46 pm

Re: how does L3HW actually works?

Sun Feb 19, 2023 7:32 am

I did test your suggestion and it does not work. Furthermore, traffic from any non VLAN network is no longer offloaded when sfp28-12 is set as just a VLAN on top of an ethernet interface. And logically this makes sense, if you read above @Raimondsp and the MikroTik documentation clearly state that by doing it in this method (VLAN on an ethernet interface) you are not able to take advantage of the L3HW offload for VLANs. After re-reading through this thread it seems both @llag and @asaleh75 have identified the same issue as I have. I'm not sure if a support ticket is already out there or maybe a bug ticket but clearly this is a bug as it does not work as described in the documentation. Check the information provided in this thread and else where (some quotes below).

Hardware Inter-VLAN Routing requires a Hardware Bridge for VLAN tagging. More info here.




I have one question about L3 support: is L3 offloading for the firewall supposed to work between 2 VLANs?.

I have my WAN connection coming in on a VLAN.

I need to firewall (NAT) it to my internal router/firewall. All VLAN interface are defined on the bridge.
The fasttrack rule has hw-offload=yes.

When I tried with L3-HW-OFFLOADING=no on the port where the WAN vlan comes in and global forwarding on, I did no see any change in speed and connections don't have the H flag.

Is there something I missed?

Yes, FastTrack connection hw-offloading supports Inter-VLAN routing too. Does the connection have the F (Fasttrack) flag? Also, I highly recommend updating to RouterOS v7.6, as there have been bugfixes and improvements specifically for FastTrack/NAT connection hw-offloading.
 
dvdhngs
just joined
Posts: 6
Joined: Sun Feb 19, 2023 2:09 pm

Re: how does L3HW actually works?

Sun Feb 19, 2023 2:28 pm

Hello everybody! I hope that I can post my throuble here, if not, sorry...
I'm a little bit confused with l3hw, i think the first thing that i need to know if it's possible:

Where I work there was just a ccr1016, routing 400 ip cam and nating pppoe, my boss bought a crs326, to all cams route via l3hw to nvrs, only pppoe go to ccr1016, I "think" that I setting the l3hw right, but all rtsp cameras still going to ccr1016.

Its possible that rtsp goes from one vlan to another (where is nvrs) through l3hw?
You do not have the required permissions to view the files attached to this post.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: how does L3HW actually works?

Sun Feb 19, 2023 10:14 pm

If you want to HW offload routing between LANs and WAN, then indeed all relevant interfaces have to be members of same bridge.

Having two SFP28 connections between switch and CCR has potential for loops if both links are members of same bridge on both sides. And any xSTP except MSTP will detect it as such. So you have four possibilities:
  1. on both sides (switch and CCR) enable MSTP
  2. keep using (it's default on ROS) RSTP but configure both interfaces on both sides as edge ports. Beware of misconfigurations, loops can happen and with this setting loop won't be detected
  3. set bridge mode to none (disable xSTP) on both sides (one side is not enough, it might loop LLDP packets making the other side panic). Same warning as in preceeding bullet applies
  4. use both SFP28 links in a 802.3ad bond and play with transmit hash policy to steer traffic evenly between links
  5. simply use single SFP28 link between the two devices if combined throughput is not likely to congest the link capacity
 
blacksnow
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Feb 15, 2023 4:46 pm

Re: how does L3HW actually works?

Mon Feb 20, 2023 12:59 am

I appreciate your suggestions but none of them worked. Disabling STP on both the switch and the router (and rebooting both) did nothing. Putting all VLANs onto a single link and physically disconnecting the other and doing a complete reboot did nothing. And putting the switch and CCR into MSTP mode and complete reboot did nothing.

In all of these scenarios I was able to maintain non-VLAN to VLAN L3HW offload. But from VLAN's on the bridge to other VLAN's on the bridge none of it was offloaded. Hoping a MikroTik engineer can just replicate this and see if its a bug or expected functionality.
If you want to HW offload routing between LANs and WAN, then indeed all relevant interfaces have to be members of same bridge.

Having two SFP28 connections between switch and CCR has potential for loops if both links are members of same bridge on both sides. And any xSTP except MSTP will detect it as such. So you have four possibilities:
  1. on both sides (switch and CCR) enable MSTP
  2. keep using (it's default on ROS) RSTP but configure both interfaces on both sides as edge ports. Beware of misconfigurations, loops can happen and with this setting loop won't be detected
  3. set bridge mode to none (disable xSTP) on both sides (one side is not enough, it might loop LLDP packets making the other side panic). Same warning as in preceeding bullet applies
  4. use both SFP28 links in a 802.3ad bond and play with transmit hash policy to steer traffic evenly between links
  5. simply use single SFP28 link between the two devices if combined throughput is not likely to congest the link capacity
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Mon Feb 20, 2023 9:07 am

Ok I don't think we are on the same page. Posting my export so you can get a better idea of whats going on, for brevity I have removed all other interfaces to avoid confusion. My ultimate goal is to get traffic offloaded between VLANs regardless if one VLAN is carrying WAN traffic it is still considered a VLAN. So that means traffic from VLAN 3,4 or 6 to and from VLAN 4000 (and any other distinct combination like VLAN 6 to VLAN 4 etc..) should be able to be offloaded with firewall-compatible Inter-VLAN routing. I have a CCR2216 and it is able to offload inter-vlan forwarding with L3HW offload when setup per Mikrotik documentation. That is not occuring here.
/interface bridge
add frame-types=admit-only-vlan-tagged name=bridge pvid=4094 vlan-filtering=\
    yes
/interface vlan
add interface=bridge name="IoT Wifi" vlan-id=6
add interface=bridge name="Mobile Wifi" vlan-id=4
add interface=bridge name="Secure Wifi" vlan-id=3
add interface=bridge name="WAN (4000)" vlan-id=4000
/interface ethernet switch
set 0 l3-hw-offloading=yes
/interface ethernet switch port
set 0 l3-hw-offloading=no
set 1 l3-hw-offloading=no
set 2 l3-hw-offloading=no
set 3 l3-hw-offloading=no
set 4 l3-hw-offloading=no
set 5 l3-hw-offloading=no
set 6 l3-hw-offloading=no
set 7 l3-hw-offloading=no
set 8 l3-hw-offloading=no
set 9 l3-hw-offloading=no
set 10 l3-hw-offloading=no
set 11 l3-hw-offloading=no
set 12 l3-hw-offloading=no
set 13 l3-hw-offloading=no
set 14 l3-hw-offloading=no
set 15 l3-hw-offloading=no
set 16 l3-hw-offloading=no
set 17 l3-hw-offloading=no
set 18 l3-hw-offloading=no
set 19 l3-hw-offloading=no
/interface bridge port
add bridge=bridge frame-types=admit-only-vlan-tagged interface=sfp28-12 pvid=\
    4094
add bridge=bridge frame-types=admit-only-vlan-tagged interface=sfp28-5 pvid=\
    4094
/interface bridge settings
set use-ip-firewall=yes use-ip-firewall-for-pppoe=yes \
    use-ip-firewall-for-vlan=yes
/interface ethernet switch l3hw-settings
set ipv6-hw=yes
/interface bridge vlan
add bridge=bridge tagged=bridge,sfp28-12 vlan-ids=4000
add bridge=bridge tagged=bridge,sfp28-5 vlan-ids=3-4,6
Please see the info from Raimondsp above (quoted the important part here below).

Before v7.2, RouterOS didn't support FastPath on vlan-filtered bridges, meaning no FastTrack for Inter-VLAN routing and no HW offloading. The feature has been introduced in RouterOS v7.2rc2, allowing Firewall-Compatible Inter-VLAN Routing.

Hi there,

Try disabling the bridge firewall and see if that helps:
/interface bridge settings
set use-ip-firewall=no use-ip-firewall-for-pppoe=no use-ip-firewall-for-vlan=no
Using the bridge firewall prevents Fastpath, which, in turn, disables FastTrack and its hw-offloading. I guess you can still leave "use-ip-firewall-for-pppoe=yes" since PPPoE traffic is not offloaded anyway, but for the sake of testing, please entirely disable the firewall.

Consider using Switch Rules (ACL) for stateless firewalling. ACL rules are executed on the hardware level.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Mon Feb 20, 2023 9:32 am

Hello everybody! I hope that I can post my throuble here, if not, sorry...
I'm a little bit confused with l3hw, i think the first thing that i need to know if it's possible:

Where I work there was just a ccr1016, routing 400 ip cam and nating pppoe, my boss bought a crs326, to all cams route via l3hw to nvrs, only pppoe go to ccr1016, I "think" that I setting the l3hw right, but all rtsp cameras still going to ccr1016.

Its possible that rtsp goes from one vlan to another (where is nvrs) through l3hw?
Hi,

l3hw does not perform any "magic" under the hood - it just offloads the routing table to the hardware. Hence, to make l3hw work, you need to set up your CRS326 as a router - to route traffic between 192.168.0.0/24 and 192.168.1.0/24 VLANs. Usually, you want to configure a router with l3hw disabled (for better diagnostics); then enable l3hw for a speed boost.

It is hard to advise without seeing the entire network topology, but I guess the easiest solution is to move IP addresses 192.168.0.1 and 192.168.1.1 from CCR to CRS. It will make CRS route the traffic without the need to reconfigure network hosts. Then, I'm not sure if CCR needs to be a member of vlan20 and vlan30 anymore. Maybe CRS can use vlan1 for routing packets to the default gateway. Or make a separate VLAN between CCR and CRS to route traffic from vlan20/vlan30 and the outside world (if needed).
 
blacksnow
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Feb 15, 2023 4:46 pm

Re: how does L3HW actually works?

Mon Feb 20, 2023 9:33 am

Hey, I tried that just now and no difference.

Hi there,

Try disabling the bridge firewall and see if that helps:
/interface bridge settings
set use-ip-firewall=no use-ip-firewall-for-pppoe=no use-ip-firewall-for-vlan=no
Using the bridge firewall prevents Fastpath, which, in turn, disables FastTrack and its hw-offloading. I guess you can still leave "use-ip-firewall-for-pppoe=yes" since PPPoE traffic is not offloaded anyway, but for the sake of testing, please entirely disable the firewall.

Consider using Switch Rules (ACL) for stateless firewalling. ACL rules are executed on the hardware level.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 267
Joined: Mon Apr 27, 2020 10:14 am

Re: how does L3HW actually works?

Mon Feb 20, 2023 9:40 am

Hey, I tried that just now and no difference.

The rest of your posted config looks fine, so I'm unsure why it doesn't work on your side. Please create a support ticket, and we will try to reproduce your issue on our end.
 
User avatar
osc86
Member Candidate
Member Candidate
Posts: 197
Joined: Wed Aug 09, 2017 1:15 pm

Re: how does L3HW actually works?

Mon Feb 20, 2023 12:57 pm

Please share the solution here, once the problem is solved. I'm really interested if it's a configuration error or a bug.
I never could get L3 offloaded inter-vlan routing working on my 2116, reading the wiki page dozens of times, and trying every possible configuration.
IP communication worked, but fasttracked connections were never offloaded to the switch chip. It was easy ro see, no H-Flag in connection table and CPU usage went up when transferring multiple Gb/s.
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: how does L3HW actually works?

Mon Feb 20, 2023 3:15 pm

Please share the solution here, once the problem is solved. I'm really interested if it's a configuration error or a bug.
I never could get L3 offloaded inter-vlan routing working on my 2116, reading the wiki page dozens of times, and trying every possible configuration.
IP communication worked, but fasttracked connections were never offloaded to the switch chip. It was easy ro see, no H-Flag in connection table and CPU usage went up when transferring multiple Gb/s.

If you're relying on fasttracked connections, it sounds like you're wanting fw-offload with NAT which has some specific limitations.

Is that what you're trying to do or are you just looking for inter-vlan routing?
 
User avatar
osc86
Member Candidate
Member Candidate
Posts: 197
Joined: Wed Aug 09, 2017 1:15 pm

Re: how does L3HW actually works?

Mon Feb 20, 2023 4:03 pm

Hi @IPANetEngineer,
you are correct, I was trying to do fw-offloading, but without NAT.
The only interface I use NAT on is of type pppoe, which connections can't be offloaded anyway IIRC.
I need firewall filtering between all vlans, this is why I can't just enable full L3-Offloading, and fw-offloading (fasttracked connections) sounded like the perfect solution.
I didn't touch it in a while because currently the cpu can handle the traffic.
I'll update the device to 7.8rc2 later today and try again with a most simple config.
I think 7.5 or 7.6 was the last release I tried to get it working and once I added trunk interfaces to the bridge, fw-offloading stopped working, even after a reboot.
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: how does L3HW actually works?

Mon Feb 20, 2023 4:29 pm

I was trying to do fw-offloading, but without NAT.


fw-offload took a few versions to stabilize. We sent in at least one ticket to MikroTik support on it.

It's been a while since I tested it in the lab, but here were the limitations of fw-offload as of 6 months ago (discovered via testing and prod use - was undocumented at the time) - it might have improved since then.

Limitations of fw-offload (as of 7.6) - the original thread these came from (https://www.reddit.com/r/mikrotik/comme ... &context=3)

- No LACP
- No VLANs
- No hairpinning on a single port
- must have dedicated in and out physical interfaces

Config requirements (once you've removed limitations above:

Must have l3-hw-offloading=yes on the switch chip *but* disabled on all ports
Must create mangle rules to use fasttrack on connections that you want to offload to hardware

https://help.mikrotik.com/docs/display/ ... onnections

@raimondsp might be able to comment further on limitations and if any of them have been removed between 7.5/7.6 and current 7.8 rc versions.
 
blacksnow
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Feb 15, 2023 4:46 pm

Re: how does L3HW actually works?

Mon Feb 20, 2023 4:50 pm

I hope this is fixed relatively quickly. I mean the whole point if 2216 is having 100gb capability through the L3HW offload. Sure it's limited to 4.5k connections but if I can't reliably connect a VLAN trunk to the 100GB port and then have the traffic between vlans (inter-vlan) offloaded when possible then how will I achieve 100gb routing, the CPU isn't going to cut it and it wouldn't be preferred to have the CPU working that hard.

I was trying to do fw-offloading, but without NAT.


fw-offload took a few versions to stabilize. We sent in at least one ticket to MikroTik support on it.

It's been a while since I tested it in the lab, but here were the limitations of fw-offload as of 6 months ago (discovered via testing and prod use - was undocumented at the time) - it might have improved since then.

Limitations of fw-offload (as of 7.6) - the original thread these came from (https://www.reddit.com/r/mikrotik/comme ... &context=3)

- No LACP
- No VLANs
- No hairpinning on a single port
- must have dedicated in and out physical interfaces

Config requirements (once you've removed limitations above:

Must have l3-hw-offloading=yes on the switch chip *but* disabled on all ports
Must create mangle rules to use fasttrack on connections that you want to offload to hardware

https://help.mikrotik.com/docs/display/ ... onnections

@raimondsp might be able to comment further on limitations and if any of them have been removed between 7.5/7.6 and current 7.8 rc versions.
 
User avatar
osc86
Member Candidate
Member Candidate
Posts: 197
Joined: Wed Aug 09, 2017 1:15 pm

Re: how does L3HW actually works?

Mon Feb 20, 2023 4:53 pm

Wow, thank you for the list.
Well, I'm using LACP, VLANs, and most inter-vlan traffic is passing through the same interface (in/out), as most traffic is going over the LACP (2x10G) Link to the CRS328.
That's really unfortunate, and I couldn't find any of the mentioned limitations on the wiki page.
Given these limitations, I can't think of a single real-world use case for fw-offloading.. I mean these devices CCR2116/CCR2216 are clearly intended for business use.. Which business doesn't use vlans for L2 security, or lacp for redundancy?
 
blacksnow
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Wed Feb 15, 2023 4:46 pm

Re: how does L3HW actually works?

Mon Feb 20, 2023 5:00 pm

I created a bug ticket here if you want to add your relevant details/examples.

viewtopic.php?t=193770
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: how does L3HW actually works?

Mon Feb 20, 2023 5:02 pm

have the traffic between vlans (inter-vlan) offloaded when possible
To be clear. l3 hwoffload works without issue on CRS3xx and CCR2xxx (haven't tried on CRS5xx yet) and can use VLANs and LACP. Inter-VLAN routing, OSPF, BGP, static routes, etc. It's stateless and doesn't try to offload NAT or connections into hardware.

fw offload is a different mode the router/switch can operate in and push NAT and stateful connections into hardware (up to the chip limit of connections)
 
dvdhngs
just joined
Posts: 6
Joined: Sun Feb 19, 2023 2:09 pm

Re: how does L3HW actually works?

Sun Feb 26, 2023 11:02 am

Hello everybody! I hope that I can post my throuble here, if not, sorry...
I'm a little bit confused with l3hw, i think the first thing that i need to know if it's possible:

Where I work there was just a ccr1016, routing 400 ip cam and nating pppoe, my boss bought a crs326, to all cams route via l3hw to nvrs, only pppoe go to ccr1016, I "think" that I setting the l3hw right, but all rtsp cameras still going to ccr1016.

Its possible that rtsp goes from one vlan to another (where is nvrs) through l3hw?
Hi,

l3hw does not perform any "magic" under the hood - it just offloads the routing table to the hardware. Hence, to make l3hw work, you need to set up your CRS326 as a router - to route traffic between 192.168.0.0/24 and 192.168.1.0/24 VLANs. Usually, you want to configure a router with l3hw disabled (for better diagnostics); then enable l3hw for a speed boost.

It is hard to advise without seeing the entire network topology, but I guess the easiest solution is to move IP addresses 192.168.0.1 and 192.168.1.1 from CCR to CRS. It will make CRS route the traffic without the need to reconfigure network hosts. Then, I'm not sure if CCR needs to be a member of vlan20 and vlan30 anymore. Maybe CRS can use vlan1 for routing packets to the default gateway. Or make a separate VLAN between CCR and CRS to route traffic from vlan20/vlan30 and the outside world (if needed).


very thanks for the tips! I did what you recommended, if possible, I need another help, I think, one more time, I did something wrong...

Problem: the traffic yet counting on interface/vlan and cpu is almost 100% at crs.
# model = CRS326-24G-2S+

/interface bridge
add ingress-filtering=no name=Bridge1-Geral vlan-filtering=yes

/interface ethernet
set [ find default-name=sfp-sfpplus2 ] name=Sfp-sfpplus2-Link1CCR-uplinknvrs-offbridge
set [ find default-name=ether2 ] name=ether2-Link2CCR-linkppps-onbridge

#Keep only 4 vlans, but are total of 12
/interface vlan
add interface=Bridge1-Geral name=vlan60nvrs vlan-id=60
add interface=Bridge1-Geral name=vlan63laranja vlan-id=63
add interface=Bridge1-Geral name=vlan70verduras vlan-id=70
add interface=Bridge1-Geral name=vlan71folha vlan-id=71

/interface bonding
add mode=802.3ad name=bonding1 slaves=ether17,ether18
add mode=802.3ad name=bonding2 slaves=ether19,ether20

/interface ethernet switch
set 0 l3-hw-offloading=yes

#disabled on sfp-sfpplus2-Link1CCR-uplinknvrs-offbrigde (wan port)
/interface ethernet switch port
set 25 l3-hw-offloading=no

/interface bridge port
add bridge=Bridge1-Geral interface=bonding1
add bridge=Bridge1-Geral interface=bonding2
add bridge=Bridge1-Geral interface=ether2-Link2CCR-linkppps-onbridge


/interface bridge vlan
add bridge=Bridge1-Geral tagged="bonding2,Bridge1-Geral,bonding1,ether2-Link2CCR-linkppps-onbridge" vlan-ids=60-73

# a total of 100 ip address
# arp list have a 429 items	
/ip address
add address=172.20.0.10 interface=Sfp-sfpplus2-Link1CCR-uplinknvrs-offbridge network=172.20.0.9
add address=100.71.0.1/24 interface=vlan60nvrs network=100.71.0.0
add address=10.122.111.1/26 interface=vlan63laranja network=10.122.111.0
add address=10.123.105.1/26 interface=vlan70verduras network=10.123.105.0
add address=10.114.106.1/26 interface=vlan71folha network=10.114.106.0
add address=10.122.103.1/26 interface=Bridge1-Geral network=10.122.103.0

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=172.20.0.9 pref-src=\
    "" routing-table=main scope=30 suppress-hw-offload=no target-scope=10


#a total of 100 items, but copied only those
ip/route/print 
Flags: D - DYNAMIC; A - ACTIVE; c, s, y - COPY; H - HW-OFFLOADED
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#      DST-ADDRESS      GATEWAY                                     DISTANCE
0  As  0.0.0.0/0        172.20.0.9                                         1
  DAcH 100.71.0.0/24    vlan60nvrs                                         0
  DAcH 10.122.111.0/26  vlan63laranja                                      0
  DAcH 10.123.105.0/26  vlan70verduras                                     0
  DAcH 10.114.106.0/26  vlan71folha                                        0
  DAcH 10.122.103.0/26  Bridge1-Geral                                      0


Thanks in advanced!!!
 
User avatar
loloski
Member Candidate
Member Candidate
Posts: 276
Joined: Mon Mar 15, 2021 9:10 pm

Re: how does L3HW actually works?

Sun Feb 26, 2023 11:31 am

you are asking too much for this switch honestly, just make it layer2 switch and put a router in front, just my 0.2$
 
dvdhngs
just joined
Posts: 6
Joined: Sun Feb 19, 2023 2:09 pm

Re: how does L3HW actually works?

Sun Feb 26, 2023 11:47 am

you are asking too much for this switch honestly, just make it layer2 switch and put a router in front, just my 0.2$
oh... sorry then... i'll do that...
 
User avatar
loloski
Member Candidate
Member Candidate
Posts: 276
Joined: Mon Mar 15, 2021 9:10 pm

Re: how does L3HW actually works?

Sun Feb 26, 2023 11:56 am

for bonding ensure that you have H flag in the bridge if you add a bonding port otherwise bonding will be processed by CPU
You do not have the required permissions to view the files attached to this post.
 
dvdhngs
just joined
Posts: 6
Joined: Sun Feb 19, 2023 2:09 pm

Re: how does L3HW actually works?

Sun Feb 26, 2023 12:04 pm

for bonding ensure that you have H flag in the bridge if you add a bonding port otherwise bonding will be processed by CPU
interface/bridge/port/print
Flags: X, I - INACTIVE; H - HW-OFFLOAD
Columns: INTERFACE, BRIDGE, HW, PVID, PRIORITY, PATH-COST
#    INTERFACE                                   BRIDGE         HW   P  PRIO  PA
0  H bonding1                                    Bridge1-Geral  yes  1  0x80  10
1  H bonding2                                    Bridge1-Geral  yes  1  0x80  10
2  H ether2-Link2CCR-linkppps-onbridge           Bridge1-Geral  yes  1  0x80  10
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Wed Mar 01, 2023 11:25 pm

I've read this thread multiples over the months. The real problem here is complexities and unclear visibility of this L3 offloading, what gets offloaded (routes), why, etc.

We certainly don't have this much of a headache working with L3 offloading on other vendors. MikroTik needs to make some changes to their UI/CLI/UX logic and docs to help make L3 offloading as simple, straightforward and clear as possible.
 
User avatar
mozerd
Forum Veteran
Forum Veteran
Posts: 871
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: how does L3HW actually works?

Thu Mar 02, 2023 2:56 am

MikroTik needs to make some changes to their UI/CLI/UX logic and docs to help make L3 offloading as simple, straightforward and clear as possible.
DarkNate …
I could not agree more …
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: how does L3HW actually works?

Thu Mar 02, 2023 9:28 am

you are asking too much for this switch honestly, just make it layer2 switch and put a router in front, just my 0.2$

I'm not sure I agree with this statement. All of the CRS3xx series do a good job of L3 switching with either static or dynamic routing. I use it at my office, my home and in client ISP and DC networks.

Because NAT is on a CCR1016 in this specific setup, I don't see any reason why using l3-hw-offload should be a problem for a CRS326. You just can't use connection tracking or NAT in hardware on this platform.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: how does L3HW actually works?

Thu Mar 02, 2023 9:29 pm

I've read this thread multiples over the months. The real problem here is complexities and unclear visibility of this L3 offloading, what gets offloaded (routes), why, etc.

We certainly don't have this much of a headache working with L3 offloading on other vendors. MikroTik needs to make some changes to their UI/CLI/UX logic and docs to help make L3 offloading as simple, straightforward and clear as possible.
you are right
but
i think MikroTik in its roots started with a strange way to do bridging (version 5 etc) and many people started to using it and learned that way
i think in 6.41 MikroTik tryed to correct course with Bridge VLAN filtering with some sucess but had up to some extend still allow old style configurations working.
maybe MikroTik try to do things in a different way to avoid pattent hell

anyway, maybe what we have today is the result and remanents of that history, and that cannot be wiped From overnight

I was lucky, never learned old way of MikroTIk bridging so i embrace Bridge Vlan Filtering without a hassle, for me was a matter of add L3 and go
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Thu Mar 02, 2023 10:36 pm

you are right
but
i think MikroTik in its roots started with a strange way to do bridging (version 5 etc) and many people started to using it and learned that way
i think in 6.41 MikroTik tryed to correct course with Bridge VLAN filtering with some sucess but had up to some extend still allow old style configurations working.
maybe MikroTik try to do things in a different way to avoid pattent hell

anyway, maybe what we have today is the result and remanents of that history, and that cannot be wiped From overnight

I was lucky, never learned old way of MikroTIk bridging so i embrace Bridge Vlan Filtering without a hassle, for me was a matter of add L3 and go
Yeah, the old VLAN config method on MikroTik was rubbish, but it's not so different from how most people do it on Cisco, JunOS, Arista either as of 2023, so I wouldn't blame MikroTik for it, because this dumb approach predates even the birth of MikroTik. @IPANetEngineer can probably elaborate from his experience with Cisco/Juniper in the early ages of the internet.

The “new” bridge method that MikroTik implemented was based on DSA, which for some reason, only Linux programmers and nerds know about? Most network engineers have no clue this is the main backbone for the bridge config approach or rather VLAN bridge aware approach and in general most Cisco/Juniper affectionados learnt squat on Linux networking to begin with, so they shit on MikroTik without real solid technical arguments:
https://www.kernel.org/doc/Documentatio ... sa/dsa.txt

Cumulus Linux also recommends the same thing as MikroTik:
https://docs.nvidia.com/networking-ethe ... idge-Mode/
The reasons why you would use VLAN-aware mode for bridges are:

Scale: The VLAN-aware mode can support 2000 concurrent VLANs while the traditional mode supports only 200 concurrent VLANs.
Simplicity: VLAN-aware mode has a simpler configuration.
However, what differentiates MikroTik from Cumulus compatible hardware (or other vendors in general)
1. Bridge config is highly hardware dependent, one wrong config on a specific hardware model, and you're done for with offloading: https://help.mikrotik.com/docs/display/ ... +switching
2. Even if you do the bridge config correctly, if you have a large network with complex topology, you cannot use the bridge so easily on all the downlink ports if your hardware model have weird switch chip and non switch chip combo like those old CCR models or simply more than one switch chip for different set of ports, can't remember the models off the top of my head. So meaning, if your hardware has more than one switch chip or one switch chip for some ports but not the others, it adds complexity to the config. Especially for medium to large scale networks with a variety of topologies, especially unconventional ones.
3. As you already know, on MikroTik hardware offloading depends on proper bridge config, on Cisco/Juniper/Arista, you don't need a bridge config for hardware offloading.

With that being said, I personally prefer bridge config (also called IRB in Juniper/Cisco world) even on those big boys for the simplicity reason mentioned on Cumulus docs. I'm not sure why Cisco and Juniper certifications don't teach this bridge config approach. It simplifies config by a ton and helps achieve what some automation experts would call “unified config”. Some people just love complexity (hello MikroTik?) instead of KISS principle.

So in short, all this complexity MikroTik brought to itself, leads to the reality that only a few MikroTik or Linux specialist can truly do L3 offloading/bridge config correctly with any hardware model and in any network topology setup. Which as we can see from this thread, means 0.1%.

The real solution here for MikroTik, which we both know, isn't going to happen is:
1. MikroTik needs to correct their complexities and/or mistakes using the daemon/CLI/UI/UX. Which means a user only configures a single bridge on a single device, irrelevant of the hardware model and software will handle the correct mapping to ensure 100% native hardware offloading subject to ASIC capabilities.
2. Once the UI/UX/CLI config for bridge is hardware independent, the network topology dependency is simplified and people can then just work better with it.

The newer CCR models like CCR2116-12G-4S+ doesn't have this problem, even hAP ax2 doesn't have this problem, but CCR2004-16G-2S+ for example has this exact same dumb problem like their models from 2010 or something. You can verify this by looking at the block diagrams.

I'm not a hardware engineer, but I am curious as to how Juniper/Cisco achieves hardware offloading even if you use bridge or no bridge, or bridge some ports, and not others? MikroTik if possible, should implement a similar approach, but still push for bridge config like Cumulus did of course, it's still the best way forward.

I will give a real life example of how many ISPs and enterprise dump MikroTik because of “high CPU usage” where the actual reason is the lack of knowledge on proper bridge config for their specific hardware model:

This is a real example of one large scale ISP serving half a million customers with a bunch of CCR1072-1G-8S+, roughly 15k PPPoE customers on each CCR1072-1G-8S+ back in 2020-21.

What these idiots did was:
1. No bridge config, therefore no bridge/L2 fast-forward/fast-path
2. Created QinQ/VLANs using Cisco/Juniper style directly on the physical interfaces
3. Did their best to crap up on the MTU config for their MPLS/VPLS/PPPoE, therefore leading to CPU overhead for packet de-fragmentation and fragmentation

What they should've done:
1. Bridge all downlink ports together, enable bridge/L2 fast-forward/fast-path and then enable VLAN filtering
2. Properly configure L2/L3 MTU on the physical ports, therefore ensuring no crap up on MPLS/VPLS/PPPoE and hence zero CPU overhead for packet de-fragmentation and fragmentation
3. Follow hardware specific bridge config as per MikroTik docs in general.

So long story short, this is how MikroTik loses even loyal customers to other vendors like Edge-Core/Cumulus/IPInfusion(?)/Russian BisonRouter etc.

I don't know how Latvian culture is, but something about MikroTik's approach to the market feels very… “Free for all” type approach, which as we can see, lead them to these various dumb problems over the years. They should increase both hardware/software quality, increase pricing for CCR models etc, increase licensing fees for high-grade switches/routers which therefore will fund high quality software/hardware employees. It's 2023 and RouterOS v7.8 still fails to do BFD, still fails to show BGP prefix count in the UI/UX/CLI.

And oh, maybe launch 400G-2Tbps routers, with say 4M to 16M ASIC route offloading capacity? With “line cards”? MikroTik has potential to be a serious threat to so-called “big vendors”, but they for whatever reason fail to capitalise on that market. I know a lot of companies that would dump big vendors for MikroTik if they sold such products and fix the issues mentioned.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Thu Mar 02, 2023 10:53 pm

Here's another example of what MikroTik's poor approach leads to, a dumb problem:
viewtopic.php?t=194073

Folks like that or people from Cisco/Juniper world who sees threads like that, will assume it's a bug and stay away from MikroTik.
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 298
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: how does L3HW actually works?

Fri Mar 10, 2023 11:18 pm

Here's an odd one.

I've spent hours overnight and this morning trying to figure out why a newly-deployed 310 won't properly offload routed traffic.

I migrated the config from an RB4011 to the 310, similar to what I've done at other sites, which are working fine. The only difference is that this one's uplink port is connected directly to a 317, also doing L3HW offload, whereas all the other 310's are out in the network connected to each other or the 317 via wireless links, in a ring.

After rebooting the 310, or after disabling/enabling L3HW offload, all traffic stays on the CPU. I found this out by accident, but it's not until I bounce the 317's port that the 310 finally pushes the traffic to the ASIC and the CPU load drops.

All of the 300's are running 7.7. The 317 and 310 are connected via an SFP+ DAC. The 310's customer-facing port is a DAC connected to a 328, which distributes the VLANs to AP's and backhauls. The 310's out in the field (where this is working flawlessly) are connected to radios on the SFP+ (10G) ports and Netonix switches on the SFP (1G) ports.

Edit: For the sake of completeness, I disabled/enabled L3HW offload, forcing traffic back to the CPU. I tried bouncing the link on the 310's downlink port (SFP+2) from the 328's side. That didn't make a difference. It only works if I disable the link to the 310's upstream port (SFP+1) from the CRS317's side.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Sat Mar 11, 2023 3:47 am

Here's an odd one.

I've spent hours overnight and this morning trying to figure out why a newly-deployed 310 won't properly offload routed traffic.

I migrated the config from an RB4011 to the 310, similar to what I've done at other sites, which are working fine. The only difference is that this one's uplink port is connected directly to a 317, also doing L3HW offload, whereas all the other 310's are out in the network connected to each other or the 317 via wireless links, in a ring.

After rebooting the 310, or after disabling/enabling L3HW offload, all traffic stays on the CPU. I found this out by accident, but it's not until I bounce the 317's port that the 310 finally pushes the traffic to the ASIC and the CPU load drops.

All of the 300's are running 7.7. The 317 and 310 are connected via an SFP+ DAC. The 310's customer-facing port is a DAC connected to a 328, which distributes the VLANs to AP's and backhauls. The 310's out in the field (where this is working flawlessly) are connected to radios on the SFP+ (10G) ports and Netonix switches on the SFP (1G) ports.

Edit: For the sake of completeness, I disabled/enabled L3HW offload, forcing traffic back to the CPU. I tried bouncing the link on the 310's downlink port (SFP+2) from the 328's side. That didn't make a difference. It only works if I disable the link to the 310's upstream port (SFP+1) from the CRS317's side.
I'm 99% sure you did your bridge configuration incorrectly for your specific topology.
See here: viewtopic.php?t=183142#p987793
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 298
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: how does L3HW actually works?

Sat Mar 11, 2023 4:45 am

I'm 99% sure you did your bridge configuration incorrectly for your specific topology.
See here: viewtopic.php?t=183142#p987793
I'm 100% sure I didn't. I use this same bridge/VLAN config at 15 sites on 20+ routers, including hAP AC2, CCR1009, CCR2004, CCR2116, RB3011, RB4011, RB5009, and the aforementioned CRS317 and CRS310's, where it all works perfectly (for the most part; L3HW offload can be finicky when making changes).

This is a snippet of the config from the site router having the problem; aside from IP addresses and DHCP pools/leases, etc., it's identical to the other production sites, including the two CRS310's that work fine.
/interface bridge
add comment=defconf ingress-filtering=no name=bridge vlan-filtering=yes

/interface ethernet
# All unused interfaces are disabled
set [ find default-name=sfp-sfpplus1 ] l2mtu=10218 mtu=10218 name=sfpp1-crs317-uplink
set [ find default-name=sfp-sfpplus2 ] l2mtu=10218 mtu=10218 name=sfpp2-crs328-downlink

/interface vlan
add interface=bridge mtu=2020 name=vlan3331-core-1 vlan-id=3331
add interface=bridge name=vlan120 vlan-id=120
add interface=bridge name=vlan3999 vlan-id=3999
add interface=bridge name=vlan4000 vlan-id=4000
add interface=bridge name=vlan4001 vlan-id=4001
add interface=bridge name=vlan4002 vlan-id=4002
add interface=bridge name=vlan4003 vlan-id=4003
add interface=bridge name=vlan4004 vlan-id=4004
add interface=bridge name=vlan4005 vlan-id=4005
add interface=bridge name=vlan4006 vlan-id=4006
add interface=bridge name=vlan4007 vlan-id=4007
add interface=bridge name=vlan4008 vlan-id=4008
add interface=bridge name=vlan4009 vlan-id=4009
add interface=bridge name=vlan4010 vlan-id=4010
add interface=bridge name=vlan4011 vlan-id=4011
add interface=bridge name=vlan4012 vlan-id=4012
add interface=bridge name=vlan4013 vlan-id=4013
add interface=bridge name=vlan4014 vlan-id=4014
add interface=bridge name=vlan4015 vlan-id=4015

/interface ethernet switch
set 0 l3-hw-offloading=yes

# Skipping IP DHCP pools etc.

/interface bridge port
# The 317 - 310 link is configured as edge ports to avoid STP blocks (the 317 and 328 are trunked together)
add bridge=bridge comment=defconf edge=yes frame-types=admit-only-vlan-tagged interface=sfpp1-crs317-uplink
add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfpp2-crs328-downlink

/interface bridge vlan
add bridge=bridge comment="Customer-facing VLANs" tagged=bridge,sfpp2-crs328-downlink vlan-ids=3999,4000-4016
add bridge=bridge comment="Core Router" tagged=bridge,sfpp1-crs317-uplink vlan-ids=3331
add bridge=bridge comment="Management" tagged=bridge,sfpp2-crs328-downlink vlan-ids=120

/ip address
# Not all are included; Some samples here to give you an idea of the breakdown of VLANs.
add address=100.66.3.1/28 comment="Customer VLAN 4001" interface=vlan4001 network=100.66.3.0
add address=100.66.3.17/28 comment="Customer VLAN 4002" interface=vlan4002 network=100.66.3.16
add address=100.66.3.33/28 comment="Customer VLAN 4003" interface=vlan4003 network=100.66.3.32
add address=100.66.3.49/28 comment="Customer VLAN 4004" interface=vlan4004 network=100.66.3.48
add address=100.66.3.65/28 comment="Customer VLAN 4005" interface=vlan4005 network=100.66.3.64
# Uplink to the core 317
add address=100.100.3.10/30 interface=vlan3331-core-1 network=100.100.3.8

 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Sat Mar 11, 2023 4:51 am

I'm 100% sure I didn't. I use this same bridge/VLAN config at 15 sites on 20+ routers, including hAP AC2, CCR1009, CCR2004, CCR2116, RB3011, RB4011, RB5009, and the aforementioned CRS317 and CRS310's, where it all works perfectly (for the most part; L3HW offload can be finicky when making changes).

This is a snippet of the config from the site router having the problem; aside from IP addresses and DHCP pools/leases, etc., it's identical to the other production sites, including the two CRS310's that work fine.
/interface bridge
add comment=defconf ingress-filtering=no name=bridge vlan-filtering=yes

/interface ethernet
# All unused interfaces are disabled
set [ find default-name=sfp-sfpplus1 ] l2mtu=10218 mtu=10218 name=sfpp1-crs317-uplink
set [ find default-name=sfp-sfpplus2 ] l2mtu=10218 mtu=10218 name=sfpp2-crs328-downlink

/interface vlan
add interface=bridge mtu=2020 name=vlan3331-core-1 vlan-id=3331
add interface=bridge name=vlan120 vlan-id=120
add interface=bridge name=vlan3999 vlan-id=3999
add interface=bridge name=vlan4000 vlan-id=4000
add interface=bridge name=vlan4001 vlan-id=4001
add interface=bridge name=vlan4002 vlan-id=4002
add interface=bridge name=vlan4003 vlan-id=4003
add interface=bridge name=vlan4004 vlan-id=4004
add interface=bridge name=vlan4005 vlan-id=4005
add interface=bridge name=vlan4006 vlan-id=4006
add interface=bridge name=vlan4007 vlan-id=4007
add interface=bridge name=vlan4008 vlan-id=4008
add interface=bridge name=vlan4009 vlan-id=4009
add interface=bridge name=vlan4010 vlan-id=4010
add interface=bridge name=vlan4011 vlan-id=4011
add interface=bridge name=vlan4012 vlan-id=4012
add interface=bridge name=vlan4013 vlan-id=4013
add interface=bridge name=vlan4014 vlan-id=4014
add interface=bridge name=vlan4015 vlan-id=4015

/interface ethernet switch
set 0 l3-hw-offloading=yes

# Skipping IP DHCP pools etc.

/interface bridge port
# The 317 - 310 link is configured as edge ports to avoid STP blocks (the 317 and 328 are trunked together)
add bridge=bridge comment=defconf edge=yes frame-types=admit-only-vlan-tagged interface=sfpp1-crs317-uplink
add bridge=bridge comment=defconf frame-types=admit-only-vlan-tagged interface=sfpp2-crs328-downlink

/interface bridge vlan
add bridge=bridge comment="Customer-facing VLANs" tagged=bridge,sfpp2-crs328-downlink vlan-ids=3999,4000-4016
add bridge=bridge comment="Core Router" tagged=bridge,sfpp1-crs317-uplink vlan-ids=3331
add bridge=bridge comment="Management" tagged=bridge,sfpp2-crs328-downlink vlan-ids=120

/ip address
# Not all are included; Some samples here to give you an idea of the breakdown of VLANs.
add address=100.66.3.1/28 comment="Customer VLAN 4001" interface=vlan4001 network=100.66.3.0
add address=100.66.3.17/28 comment="Customer VLAN 4002" interface=vlan4002 network=100.66.3.16
add address=100.66.3.33/28 comment="Customer VLAN 4003" interface=vlan4003 network=100.66.3.32
add address=100.66.3.49/28 comment="Customer VLAN 4004" interface=vlan4004 network=100.66.3.48
add address=100.66.3.65/28 comment="Customer VLAN 4005" interface=vlan4005 network=100.66.3.64
# Uplink to the core 317
add address=100.100.3.10/30 interface=vlan3331-core-1 network=100.100.3.8

Your “uplink” port should be an independent Ethernet interface with plain L3 VLAN on top of it, not in the bridge. Remove it from the bridge. This is the issue right here, there are 1000 different ways of setting a network and also 1000 ways of doing bridge configuration for 1000 different topologies.

In your specific case, there's no need for the uplink to be bridged unless it's connected to another switch that acts as a trunk, and you require BPDUs to traverse aka RSTP/MSTP, and therefore it's chain switching topology and there's no “uplink/downlink” relationship.

Also, as already explained, different MikroTik hardware models requires different configuration. Of course, no guarantees for offloading though, you'll need to test my suggestion above and reboot.
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 298
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: how does L3HW actually works?

Sat Mar 11, 2023 8:05 am

Your “uplink” port should be an independent Ethernet interface with plain L3 VLAN on top of it, not in the bridge. Remove it from the bridge. This is the issue right here...

Um, no, it's not. And MikroTik recommends against that, especially for hardware offload to work on switch chips (which is literally the point of this thread).

From https://help.mikrotik.com/docs/display/ ... Offloading:

Inter-VLAN Routing
Since L3HW depends on L2HW, and L2HW is the one that does VLAN processing, Inter-VLAN hardware routing requires a hardware bridge underneath. Even if a particular VLAN has only one tagged port member, the latter must be a bridge member. Do not assign a VLAN interface directly on a switch port! Otherwise, L3HW offloading fails and the traffic will get processed by the CPU:

(The config you recommend is stricken through in the original) -> /interface/vlan add interface=ether2 name=vlan20 vlan-id=20

Assign VLAN interface to the bridge instead. This way, VLAN configuration gets offloaded to the hardware, and, with L3HW enabled, the traffic is subject to inter-VLAN hardware routing.

I'm routing between VLANs, plain and simple. It shouldn't matter which port they come in on or leave from, or how many VLANs are on a port. As I've now said two or three times, I have this exact same configuration working on two other CRS310's as well as a CRS317 and L3HW offload picks right up without issue.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: how does L3HW actually works?

Mon Mar 13, 2023 9:50 pm

You're only proving my argument that MikroTik has made it super complex.
 
mfedotov
just joined
Posts: 18
Joined: Mon Oct 25, 2021 3:32 am

Re: how does L3HW actually works?

Thu Mar 16, 2023 6:09 pm

Hi @sirbryan

Are all of the ports on your switch enabled for L3HW offloading? I had a configuration when I had a port with disabled L3HW, and if any VLANs defined on the port with the disabled L3HW are present on the other ports, those ports would be silently disabled for L3HW either. I asked on the forum and Mikrotik confirmed that this is expected behavior...
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 298
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: how does L3HW actually works?

Fri Mar 17, 2023 5:14 am

Hi @sirbryan

Are all of the ports on your switch enabled for L3HW offloading? I had a configuration when I had a port with disabled L3HW, and if any VLANs defined on the port with the disabled L3HW are present on the other ports, those ports would be silently disabled for L3HW either. I asked on the forum and Mikrotik confirmed that this is expected behavior...
No, and by default they're all on, so when you turn it on for the switch, they're on for the ports (at least that was the case with this router).

As I said before, I've used this config on multiple L3 switches now, and the only way to get this particular site to work was to bounce the port. Very strange.

It's humming along now, with 700+Mbps at the moment and 1-2% load.
 
akschu
Frequent Visitor
Frequent Visitor
Posts: 57
Joined: Thu Mar 15, 2012 2:09 am

Re: how does L3HW actually works?

Sun Apr 23, 2023 3:18 am

Quick question about l3HW offloading. I need to serve a bunch of customers with DHCP and rely on radius accounting to see how much traffic they are moving. If I configure radius accounting will RouterOS be able to get the bits per lease back from the switch chip and pass it up the stack to the radius client or will I just stop seeing radius accounting data when hardware is abled, or will it simply revert back to the CPU?

Also, I'm assuming if I assign a queue in radius that would for sure use the CPU.... correct?

schu
 
User avatar
sirbryan
Member Candidate
Member Candidate
Posts: 298
Joined: Fri May 29, 2020 6:40 pm
Location: Utah
Contact:

Re: how does L3HW actually works?

Sun Apr 23, 2023 1:38 pm

If I configure radius accounting will RouterOS be able to get the bits per lease back from the switch chip and pass it up the stack to the radius client or will I just stop seeing radius accounting data when hardware is abled, or will it simply revert back to the CPU?

Also, I'm assuming if I assign a queue in radius that would for sure use the CPU.... correct?
I assume that, based on the fact the counters per VLAN stop incrementing except when packets are forced to the CPU, you won't get accounting data per IP. Yes, anything that hits the queue will be CPU-bound. I haven't done anything with queues, so I don't know if any of the traffic will actually be queued, since the ASIC is handling all the work and the CPU doesn't see the packets to queue in the first place.
 
User avatar
chechito
Forum Guru
Forum Guru
Posts: 2989
Joined: Sun Aug 24, 2014 3:14 am
Location: Bogota Colombia
Contact:

Re: how does L3HW actually works?

Sun Apr 23, 2023 6:58 pm

talking about bandwidth management...
on a CRS-317 with ROS 7.6 runing a simple L3 HW offload with static routes all runing fine but ACL with Action=Rate (to do some bandwidth management) does not work
ACL to drop traffic works ok with L3 HW offload


similar ACL with Action=Rate (to do some bandwidth management) work OK without L3 HW offload (only L2 scenario) on ROS 7.6

Who is online

Users browsing this forum: coffee1978, Sailwebwifi and 60 guests