CRS309-1G-8S+ L3-HW-OFFLOADING bypasses a NAT rule

RouterOS General
1

I use a CRS309-1G-8S+ to connect to my ISP via a gpon sfp. The ISP is on vlan100 and the gpon also has a management/status outside the vlan. My setup is to put my home network on 192.168.1.0/24, the gpon management on 192.168.2.0/24 and NAT the 2:

/interface vlan
add interface=bridge name=vlan-main vlan-id=10
add interface=sfp-sfpplus3 name=vlan100 vlan-id=100

/ip address
add address=192.168.1.1/24 interface=vlan-main network=192.168.1.0
add address=192.168.2.2/24 interface=sfp-sfpplus3 network=192.168.2.0

/ip firewall nat
add action=src-nat chain=srcnat dst-address=192.168.2.0/24 src-address=\
    192.168.1.0/24 to-addresses=192.168.2.2

and sfp-sfplus3 is not on the bridge.

This works fine until I set l3-hw-offloading=yes in the switch, at which point the management interface drops away. I guess the hw is sending packets from 192.168.1.0/24 to 192.168.2.0/24 directly.

I have considered using the hw offloading for only fastrack connections, but the internet connection is so good as is that I didn't want to change it.

I tried adding sfp-sfpplus3 to the bridge, setting the port to accept-all with a pvid of 10, moving the vlan100 to the bridge and seting sfp-sfplus3 as tagged for 100 and untagged for 10. Unfortunately sfp-sfpplus3 would not show up in current-untagged for 10, so the sfp management interface would not be reachable. This would probably not been the best for security.

The sfp nat side is super simple, so I cannot set a default route there.

One thing that is close to working is seting the network on the sfp to 192.168.0.0/16 and deleting the nat. With that I can ping 192.168.2.1 from the router, but not from somewhere else in the network. Any suggestions on how to debug why ip forwarding is not working?

2

Possibly unrelated, but you are sure that you want the IP address on sfp-sfpplus3:

add address=192.168.2.2/24 interface=sfp-sfpplus3 network=192.168.2.0

and not on the VLAN interface, i.e. as opposed to:

add address=192.168.2.2/24 interface=vlan100 network=192.168.2.0

Apart from this, posting only the snippets you believe relevant is not enough for any of the more expert members to provide meaningful advice, you should post your FULL configuration (anonymized where needed).

3

Yes, it should be on sfpplus3. The vlan100 is where the dhcp-client goes. The complete config is attached.

I am starting to suspect that it is the gpon that is is not respecting the 255.255.0.0 mask and not replying. I will try putting it on a media converter after work to test if it replies to pings from both 192.168.1.2 and 192.168.2.2.

The original issue of l3 hardware offloading disabling the nat is, of course, in the router.

main.rsc (12.3 KB)

4

(NOTE: I've not read your post with the attached config export while writing this post!)

I think you should still add sfp-sfpplus3 to the main bridge. You can maybe try this approach:

First you'll need to update the L3HW configuration, to temporarily disable it on the switch before making changes to the bridge and VLANs:

/interface ethernet switch set 0 l3-hw-offloading=no

Now add sfp-sfpplus3 to the main bridge, with frame-types=admit-all and pvid=101.

/interface bridge port
add bridge=bridge interface=sfp-sfpplus3 frame-types=admit-all pvid=101

Then change the vlan100 interface to have bridge as parent interface (instead of sfp-sfpplus3),

/interface vlan
set [find vlan-id=100] interface=bridge

add an /interface bridge vlan entry:

/interface bridge vlan
add bridge=bridge tagged=bridge,sfp-sfpplus3 vlan-ids=100

Next add a new vlan-gpon interface with id 101:

/interface vlan
add interface=bridge name=vlan-gpon vlan-id=101

/interface bridge vlan
add bridge=bridge tagged=bridge untagged=sfp-sfpplus3 vlan-ids=101

Then edit the /ip address entry 192.168.2.2/24 that is currently on the sfp-sfpplus3 interface to be on vlan-gpon instead:

/ip address
set [find interface=sfp-sfpplus3] interface=vlan-gpon

Once this is done, vlan-gpon plays the role of the interface facing the management side of the GPON SFP (instead of sfp-sfpplus3).

Then you need to set l3-hw-offloading=no on sfp-sfpplus3

/interface ethernet switch port 
set sfp-sfpplus3 l3-hw-offloading=no

and yes on the other LAN ports (probably already done).

Also make sure the fasttrack rule (with hw-offload=yes, don't forget this flag!) and the "accept established,related" rule are at the top of the filter forward chain:

/ip firewall filter
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
add action=accept chain=forward connection-state=established,related,untracked

And of course check your source NAT rule for WAN as well as for the GPON management. Normally, you can just add vlan-gpon to the WAN list and benefit from the existing masquerade rule and don't need the extra SRCNAT rule:

/interface list member
add interface=vlan-gpon list=WAN

Update after reading export: However you have this rule:

add action=drop chain=forward dst-address-list=not_global_ipv4 log=yes \
    log-prefix=no-forward out-interface-list=WAN

so it might interfere with that.

Finally, you can re-enable L3HW on the switch chip:

/interface ethernet switch set 0 l3-hw-offloading=yes

EDIT: Another remark: You are using IPv6 but I think currently L3HW Fasttrack doesn't support IPv6 yet. So your IPv6 WAN traffic will be limited by the slow CPU.

5

CRS309-1G-8S+ L3-HW-OFFLOADING bypasses a NAT rule
Just another example of an out of control MT device.
Didn’t they provide a free whip at the MT Certification training?
Get your device under control!

6

Hi, thanks so much for everyone that helped. When using the sniffer I was filtering only icmp to reduce the noise. Today I disconnected the internet cable and collected everything. Looking at the dump in wireshark I found the problem. The gpon would get the pings, but not reply because it would never get an ARP reply. The solution in the end was simple:

Configure the gpon to use a network of 192.168.0.0/16.

Configure the router ips:

/ip address
add address=192.168.1.1/24 interface=vlan-main network=192.168.1.0
add address=192.168.2.2/24 interface=sfp-sfpplus3 network=192.168.2.0

proxy the ARP on the interface connected to the gpon

set [ find default-name=sfp-sfpplus3 ] arp=proxy-arp auto-negotiation=no \
    sfp-ignore-rx-los=yes speed=2.5G-baseX

And delete the old src-nat.

If the problem with the nat was that the l3 offloading was indeed sending packets from 192.168.1.0/24 to 192.168.2.0/24 directly, that is now fine as the gpon thinks that is still its network and the router proxies the ARPs.