Multiple bridge interfaces with different MTU causes out of memory problem

RouterOS General
1

I have a HAP AC2 router, in which the 2.4GHz and 5GHz wireless interfaces can only be set to at most 1600 for L2MTU, while the ether1 to ether5 can set the L2MTU up to 9214. In order to conserve the maximum L2MTU for all wired port as I want to enable jumbo frame, I configured two bridges:
bridge “A” is connecting all physical interfaces;
bridge “B” mainly for the wireless interfaces.

Thus, the L2MTU for bridge A and B can be managed separately.

However, this configuration is instable and sometimes crash the router and got rebooted. Here are some observations:
(i) After the reboot, a log “out of memory condition was detected” was displayed.
(ii) The CPU Profiler was done when the router was going to crazy, the process “SPI” consume 30% on one CPU core. The Disk utilization on all CPU also jumped to around 30 to 40%.
(iii) when the router hung, ping was not responding, winbox could not search it, webfig was not accessible. The only way to resolve this issue was to reboot the router manually.
(iv) the router hang even quicker if huge traffic is passing between bridge A to B.

The most strange behavior was the router didn’t hang if I changed back the L2MTU of all physical interfaces from 9214 back to 1600 so that the L2MTU on Bridge A fell back to 1600.
Is there any known issue on this setup? if multiple bridges is not recommended, is there any other solution that can achieve the same goal?

Additional Information:
Configuration which is stable:

/interface bridge> print
Flags: X - disabled, R - running
 0 R name="B" mtu=auto actual-mtu=1500 l2mtu=1600 arp=enabled arp-timeout=auto protocol-mode=rstp fast-forward=yes igmp-snooping=no auto-mac=yes ageing-time=5m priority=0x8000 max-message-age=12s forward-delay=9s transmit-hold-count=6 vlan-filtering=yes ether-type=0x8100 pvid=1 frame-types=admit-all ingress-filtering=no dhcp-snooping=yes add-dhcp-option82=no
     
 1 R name="A" mtu=auto actual-mtu=1500 l2mtu=1600 arp=enabled arp-timeout=auto protocol-mode=rstp fast-forward=yes igmp-snooping=no auto-mac=yes ageing-time=5m priority=0x8000 max-message-age=12s forward-delay=9s transmit-hold-count=6 vlan-filtering=yes ether-type=0x8100 pvid=1 frame-types=admit-all ingress-filtering=no dhcp-snooping=yes add-dhcp-option82=no

/interface ethernet> exp
/interface ethernet
set [ find default-name=ether2 ] l2mtu=1600
set [ find default-name=ether3 ] l2mtu=1600 
set [ find default-name=ether4 ] l2mtu=1600
set [ find default-name=ether1 ] l2mtu=1600
set [ find default-name=ether5 ] l2mtu=1600

Configuration which is INSTABLE:

/interface bridge> print
Flags: X - disabled, R - running
 0 R name="B" mtu=auto actual-mtu=1500 l2mtu=1600 arp=enabled arp-timeout=auto protocol-mode=rstp fast-forward=yes igmp-snooping=no auto-mac=yes ageing-time=5m priority=0x8000 max-message-age=12s forward-delay=9s transmit-hold-count=6 vlan-filtering=yes ether-type=0x8100 pvid=1 frame-types=admit-all ingress-filtering=no dhcp-snooping=yes add-dhcp-option82=no

 1 R name="A" mtu=auto actual-mtu=1500 l2mtu=9214 arp=enabled arp-timeout=auto protocol-mode=rstp fast-forward=yes igmp-snooping=no auto-mac=yes ageing-time=5m priority=0x8000 max-message-age=12s forward-delay=9s transmit-hold-count=6 vlan-filtering=yes ether-type=0x8100 pvid=1 frame-types=admit-all ingress-filtering=no dhcp-snooping=yes add-dhcp-option82=no

/interface ethernet> exp
/interface ethernet
set [ find default-name=ether2 ] l2mtu=9214
set [ find default-name=ether3 ] l2mtu=9214
set [ find default-name=ether4 ] l2mtu=9214
set [ find default-name=ether1 ] l2mtu=9214
set [ find default-name=ether5 ] l2mtu=9214

P.S. only the L2MTU changed, the MTU is still 1500, but it already causes trouble.

2

Use of jumbo frames between end devices is largely overrated these days with fast switches and capable NICs and you should have a very good reason for insisting on using them. Do some (real life!) benchmarks to assess the difference between using standard 1500-byte MTU and jumbo frames to see if difference is worth the trouble.
Keep in mind that routing packets between subnets with different MTUs causes additional load on both router (which performs packet fragmentation) and end devices (receiving side has to do de-fragmentation). If PMTUD works then MTU is lowered anyway.

If you use second bridge only for non-wired ports, then the additional overhead (compared to using single bridge) is low. If wireless devices could be in same subnet as wired devices, then using single bridge reduces load on CPU because packets skip routing process.

3

Physical interface MTU/L2 MTU should be maxed out on all physical ports identically.

L3 MTU on all physical interfaces and bridges/wireless bridges should be 1500. But if you’re running tunnelling protocols like VPLS, L3 MTU = 1600.

If you’re using /interface vlans on MikroTik, L3 MTU on it should match 1500 or 1600.

More details here on the MTU section: http://forum.mikrotik.com/t/how-to-edge-router-and-bng-optimization-for-isps/150007/1

4

My personal setup:


/interface ethernet
#For RFC4638 upstream L3=1520#
set [ find default-name=ether1 ] l2mtu=9214 mtu=1520

#For RFC4638 upstream L3=1520#
set [ find default-name=ether2 ] l2mtu=9214 mtu=1520

set [ find default-name=ether3 ] l2mtu=9214
set [ find default-name=ether4 ] l2mtu=9214
set [ find default-name=ether5 ] l2mtu=9214 

#Bridge L3 MTU = 1500 default#
/interface bridge
add admin-mac=############### arp=proxy-arp auto-mac=no comment=defconf name=bridge

/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
5

Thanks for your reply. I also alerted the overhead burdened with such multiple bridge setup, but I just didn’t realize that the “penalty” of fragmentation/defragmentation process in Mikrotik devices led to router hang / irresponsiveness. (I just added the additional information on post #1)

6

The main problem is the maximum L2MTU for all wireless interfaces are capped at 2290 (or 1600 if managed by CAPsMAN, I know there is L2MTU setting in CAPsMAN but it seems the clients simply neglect this setting, it can be discussed in other post), if I insist to bridge these wireless interfaces with the wired ports, the L2MTU of that bridge interface will be lowered to the L2MTU of the wireless interfaces. As a result, I created another bridge interface solely for wireless interfaces. (I just added the additional information on post #1)