Bridge: VPLS to ETH forwarding issue

RouterOS Wireless Networking
1

Guys, I’m busting myself and still cannot find where the problem is. Here is what the situation looks like and where the problem is:

Basiclly, I have a backbone running under 802.11n + NV2 (MIMO, VPLS in bridge with ETH, RSTP bridge protocol). Every thing works just fine up until R9. I get 100 Mbps TCP from internet up until R9. However on very next device I cannot push more than 60 (averige is about 45) mbps and I get a lot of packet losses (20%). The situation is the same in other direction. I did a test from R13 to R9 and I get 100 Mbps, yet on R7 and border router I get max 40 - 60 and a lot of packet losses. Something like this:

R13 ------- transparent bride ------- R9 ------ transparent bridge ------ BR
| | |
±------ 100 mbps / ping OK --------±------- 100 mbps / ping OK -----+
| |
±---------------------- 40 - 60 mbps / 20% loss ----------------------+


I checked the cables, ETH status (100mbps/FD), verified bridge (RSTP/VPLS+ETH) setup, firewall is off, connection tracking is off, compared it point-by-point with other similar configs and I cannot find what is causing this. MPLS-MTU set to 1526 on all devices. Traffic is going trough VPLS tunnels. All devices are RB433AH, running MT 5.11 or 5.12. Firmware upgraded.
It all points to a problem on forwarding packets within the bridge on R9. I even tried changing to WDS (in desperation) but I got noting. I added another device behind R9, putting R9 in transparent bridge and nothing.

Naturally, this is causing OSPF to be instabile, so I set NBMA on that part (Border router to R9).

HAS ANYONE HAD SUCH SITUATION? HELP !!! PLEASE!!!

Thanks in advance for any idea, direction, recommendation …

MUM entry tickets for most helpful forum users!
2

UPDATE:

As I mentioned, I added another device behind R9 and set R9 inside full transparent bridge with R8. So the traffic is passed to this new device. I swapped from VPLS to WDS transparent bridge. Now I have less packet losses (3% over 4 wireless bridges), but I still cannot push over last bridge more than 50 mbps. It looks like there’s something wrong with either RB433AH or 5.11.

I contacted support and am waiting for their reply.

Again, is or has anyone expirienced the similar situation?

Something is very wrong with passing traffic trough the bridge and I do not know what. Here is an output of the transparent bridge setup on R9:

BRIDGE STATUS:

Flags: X - disabled, R - running 
 0  R name="loopback" mtu=1500 l2mtu=65535 arp=enabled mac-address=02:AD:50:F2:78:05 protocol-mode=none priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m 

 1  R name="bridge1" mtu=1500 l2mtu=1522 arp=enabled mac-address=00:0C:42:92:AA:77 protocol-mode=rstp priority=0x8000 auto-mac=yes admin-mac=00:00:00:00:00:00 max-message-age=20s forward-delay=15s transmit-hold-count=6 ageing-time=5m

BRIDGE PORT STATUS:

Flags: X - disabled, I - inactive, D - dynamic 
 0    interface=2-ETH bridge=bridge1 priority=0x80 path-cost=10 edge=auto point-to-point=auto external-fdb=auto horizon=none 

 2  D interface=wlan1 bridge=bridge1 priority=0x80 path-cost=100 edge=no point-to-point=yes external-fdb=no horizon=none

IP ADDRESSES

Flags: X - disabled, I - invalid, D - dynamic 
 #   ADDRESS            NETWORK         INTERFACE
 0   172.16.0.1/27      172.16.0.0   ether3
 1   10.100.0.9/32      10.100.0.9   loopback
 2   10.100.2.14/30     10.100.2.12   wlan1
 3   172.16.0.206/28    172.16.0.192    bridge1

WIRELESS SETUP:

Flags: X - disabled, R - running 
 0  R name="wlan1" mtu=1500 mac-address=00:15:6D:84:9C:8F arp=enabled disable-running-check=no interface-type=Atheros 11N radio-name="00156D849C8F" mode=station-wds ssid="ECP-NLSO" area="" frequency-mode=superchannel country=no_country_set antenna-gain=0 frequency=5370 band=5ghz-onlyn channel-width=20/40mhz-ht-below scan-list=5300-5500 wireless-protocol=nv2 rate-set=configured supported-rates-a/g="" basic-rates-a/g="" max-station-count=2007 distance=dynamic tx-power-mode=default noise-floor-threshold=default nv2-noise-floor-offset=default periodic-calibration=default periodic-calibration-interval=60 dfs-mode=no-radar-detect wds-mode=dynamic wds-default-bridge=bridge1 wds-default-cost=100 wds-cost-range=50-150 wds-ignore-ssid=no update-stats-interval=disabled bridge-mode=enabled default-authentication=yes default-forwarding=yes default-ap-tx-limit=0 default-client-tx-limit=0 proprietary-extensions=post-2.9.25 wmm-support=disabled hide-ssid=no security-profile=default disconnect-timeout=3s on-fail-retry-time=100ms preamble-mode=both compression=no allow-sharedkey=no station-bridge-clone-mac=00:00:00:00:00:00 ht-ampdu-priorities=0,1 ht-guard-interval=long ht-supported-mcs=mcs-7,mcs-10,mcs-11,mcs-13,mcs-16,mcs-17,mcs-18,mcs-19,mcs-20,mcs-21,mcs-22,mcs-23 ht-basic-mcs=mcs-7,mcs-10,mcs-11 ht-txchains=0,1 ht-rxchains=0,1 ht-amsdu-limit=8192 ht-amsdu-threshold=8192 tdma-period-size=2 nv2-queue-count=2 nv2-qos=default nv2-cell-radius=30 nv2-security=disabled nv2-preshared-key="" hw-retries=12 frame-lifetime=0 adaptive-noise-immunity=client-mode hw-fragmentation-threshold=disabled hw-protection-mode=none hw-protection-threshold=0 frequency-offset=0 rate-selection=advanced

ETH STATUS

Flags: X - disabled, R - running, S - slave 
 0    name="ether1-PoE" mtu=1500 l2mtu=1526 mac-address=00:0C:42:92:AA:76 arp=enabled auto-negotiation=yes full-duplex=yes speed=100Mbps 

 1 R  name="2-ETH" mtu=1500 l2mtu=1522 mac-address=00:0C:42:92:AA:77 arp=enabled auto-negotiation=yes full-duplex=yes speed=100Mbps master-port=none bandwidth=unlimited/unlimited switch=switch1 

 2 R  name="ether3" mtu=1500 l2mtu=1522 mac-address=00:0C:42:92:AA:78 arp=enabled auto-negotiation=no full-duplex=yes speed=100Mbps master-port=none bandwidth=unlimited/unlimited switch=switch1

On the other side of the bridge, on R8 everything is similar. Wireless card in AP-bridge, dynamic WDS (used to be VPLS) in bridge with ETH, running under RSTP protocol. On top of the bridge, there is administrative IP address. ETH is set at 100 Mbps /FD. No firewall, conntrack off.

So, regardless I set VPLS or WDS, this points to a problem with Bridge and passing traffic trough it (I might got this discussion in a wrong topic … SORRY!!!). Anyhow, I cannot see why. From ethernet side I get 100 Mbps TCP inside the bridge. From wireless side I get also about 100 Mbps TCP (CPU slams at 100% above 93 - test done from Border router). But it just isn’t going trough!!!

Please, anyone … look over and see if you can spot anything.

In a meantime, I’m gonna change the routerboard tonight and hope it will solve the problem.

3

Seems that eth port on R9 is not working properly. Try to change port (e.g. from ether1 to ether2).

If you running this one on PoE you can buy “PoE ejector” - something like one of those: http://www.ted.net.pl/popup_image.php?pID=765

4

lukpiot, tnx for your post. However ETH is not an issue here. As I described in my posts I could get 100 mbps from wired side to this router. The problem actually proved to be on wireless side.

To share (regardless how trivial it might sound and obvious it should have been) here are my observations:

I’ve been exchanging emails with Maris from Mikrotik’s support team on this situation (tnx Maris) and his suggestion was:

But if you get the same BW with WDS, then it is possible that wireless link is
unable to forward more than that.Since TCP is bidirectional it will reduce
bandwidth.

Now, as much as odd this looked to me at first glance, I checked this morning (very, very early in the morning) if this just might be so. And here is where the problem lies:

First thing I noticed on this link is that regardless the signal levels (-62/-61 on 25 km) and noise level (-59 dbi) CCQ was dancing around pretty much comparing to other links. It still remained, most of the time above 80%, but it still ranged from 100/100% to 50%/50%. I checked freq usage, spectral history, spectral scan and found that there are a lot of interferences on complete 5Ghz range. I noticed this even earlier but as the point-to-point and within bridge TCP test showed that I can pull trough 100 Mbps, I simply ignored this. That is why I failed to mention that in my previous posts.
Anyway, as soon as that traffic was to be carried over VPLS tunnel (and WDS) this instability started to influence TCP traffic. Packet losses, ACK and higher latency caused traffic to get cut in half.
I moved freqs bit by bit to find one with the best (most stable) registration and CCQ and now I got 85 Mbps TCP up until private servers (5th hop). Not perfect, but comparing to 40 - 50 … very much satisfactory.

Again, thanks to Maris and Mikrotik’s support team. I know this should have been obvious to me at very start, but sometimes we simply ignore what is wright infront of us.

5

Hi denis,

RB 433ah supports 1526 MPLS-MTU only in the ether1. Try to change it to 1522 in all devices to make sure that will work all the time.

Regards.