Wireless backhaul best practice?

RouterOS Wireless Networking
1

Through ROS 4.17, when we wanted to create a (bridged) wireless backhaul link between RouterBoards, we would configure corresponding radios on each RB to ap-bridge, dynamic WDS, bridge the radio and Ethernet, enable RSTP, and they’re bridged.

As of of ROS 5.x, this same configuration results–sometimes immediately and sometimes after an hour or two of seemingly normal operation–in a packet storm of several Mbps between the APs that effectively makes the remote RBs unreachable. Sometimes even the local board becomes unreachable by Ethernet, and must be power-cycled to log on and disable the radios.

  1. I’m curious why this works effortlessly between two 4.x devices, but if either is running 5.x it inevitably trashes the network. But more importantly:

  2. Is there still a simple way to establish a wireless backhaul network?

  • Preferably symmetric (same configuration at each end)
  • Preferably multipoint
  • Sample ‘/ interface wireless export compact’ output welcome!

In the case of using a Mesh rather than a Bridge: my concern is that we have several non-ROS switches on our network that will support RTSP but not HWMP+, which I worry could result in a bridge loop with a radio link on one side and a wired link on the other side.

2

set one side to ap-bridge and the other to station-bridge. No need for rstp.

3

Ideally, every link in our backbone would be a strictly point-to-point connection, but sometimes we need to have multipoint at both ends.

(And as an aside: even with strictly point-to-point links, wouldn’t we still need rstp to prevent three inter-linked RBs from forming a bridge loop?)

4

ap-bridge <> station-bridge is still what you need to ptp.

as far as rstp, I guess I don’t see where the loop is coming in. If you could provide some more details as to where you are having a layer2 segment loop… Anyway, rstp wont hurt, its just some overhead so you can leave it on if you are concerned.

If you are putting in multiple links for redundancy, you might want to look at something more than rstp, like doing OSPF to load balance two links or bonding if the links are very similar in performance/latency.

5

But what is the established practice to connect two ap-bridge radios?

as far as rstp, I guess I don’t see where the loop is coming in. If you could provide some more details as to where you are having a layer2 segment loop… Anyway, rstp wont hurt, its just some overhead so you can leave it on if you are concerned.

Give three APs “A”, “B” and “C”, where A has bridged ptp links to both B and C; B has bridged ptp links to both A and C; and C has bridged ptp links to both B and C; then there is a cycle in which a broadcast packet (for example) could be forwarded from A, to B, to C, to A, etc. (RSTP detects the cycle and choose of the three two-way links above to disable.)

If you are putting in multiple links for redundancy, you might want to look at something more than rstp

Redundancy is the idea. In fact what we want is a 5GHz backhaul mesh, except I’m concerned that using a mesh rather than a bridge would result in problems where we use non-MT switches, which do not support the HWMP+ protocol. For example, suppose two APs are linked by both radio backhaul, and by Ethernet through a non-MT switch. Would HWMP+ detect the loop?

like doing OSPF to load balance two links…

I would love to set up OSPF, since our backbone will soon be too large to be a single broadcast domain. A working configuration is going to take some trial and error though, as my networking experience is limited (background in software development).

6

[quote=“syadnom”]
ap-bridge <> station-bridge is still what you need to ptp.
[/quote]

But what is the established practice to connect two ap-bridge radios?

ap-bridge to ap-bridge means full WDS, which can be somewhat troublesome. You can do it, but if it is a ptp link then ap-bridge to station-bridge is best

as far as rstp, I guess I don’t see where the loop is coming in. If you could provide some more details as to where you are having a layer2 segment loop… Anyway, rstp wont hurt, its just some overhead so you can leave it on if you are concerned.

Give three APs “A”, “B” and “C”, where A has bridged ptp links to both B and C; B has bridged ptp links to both A and C; and C has bridged ptp links to both B and C; then there is a cycle in which a broadcast packet (for example) could be forwarded from A, to B, to C, to A, etc. (RSTP detects the cycle and choose of the three two-way links above to disable.)

Ok, I see now. If it were me, I wouldn’t bridge the A-B, B-C, C-A links. I would run 3 separate L2 segments and route with OSPF. If that isn’t an option, RTSP will be fine with the following notes: RTSP keeps the current route until it fails. If one links gets slower, RTSP wont know/care. It’s a one way failover, meaning that once it fails over, it will stay that way until it fails again.


If you are putting in multiple links for redundancy, you might want to look at something more than rstp

Redundancy is the idea. In fact what we want is a 5GHz backhaul mesh, except I’m concerned that using a mesh rather than a bridge would result in problems where we use non-MT switches, which do not support the HWMP+ protocol. For example, suppose two APs are linked by both radio backhaul, and by Ethernet through a non-MT switch. Would HWMP+ detect the loop?

HWMP+ would detect the loop. Problem is that HWMP+ is a link state, not a link quality mesh routing protocal. So if a link is up at all, HWMP+ will consider it 100%. What I meant was that if you have 2 links, you might want to utilize the combined throughput. Redundancy is nice, but double capacity w/ failover is better.

like doing OSPF to load balance two links…

I would > love > to set up OSPF, since our backbone will soon be too large to be a single broadcast domain. A working configuration is going to take some trial and error though, as my networking experience is limited (background in software development).

You can setup the links with OSPF and then run an EoIP tunnel between. You can bridge the EoIP into the L2 on both sides. you get the throughput gains and maintain a L2 link.

7

I have been seeing the same issues with the wds mesh. I have yet to figure it but honestly I haven’t spent a huge amount of time looking into it other than I can describe what you are probably seeing is a large amount of rx packets and rx data. This is all coming from a arp storm on the wds bridge. I would like to know the fix if possible.

8

I have been experimenting with using Mesh instead of Bridge interfaces. Thus far, it looks like “dynamic mesh” mode eliminates the broadcast storm problem, but now I’m having a problem with the ethernet ports on the Mesh because we have non-MT switches between some of our equipment, which is causing all sort of problems with HWMP+. (And replacing all our switches with MT equipment is out of the question.)

As long as there is only one contiguous group of equipment on our network using Mesh interfaces, it works fine; as soon as any equipment with a Mesh interface is connected anywhere else on our network, all hell breaks loose: sometimes one or both groups of Meshed equipment stops responding (even by MAC address), or other non-Mesh equipment on the network becomes unreachable. Usually, disconnecting the Meshed equipment lets the others get back on, but some devices seem to require a reboot before their ethernet interfaces will respond again.

I want my simple bridged network back! Each RouterBoard has a bridge, backhaul links (wlan or ether) are ports on the bridge, and the RouterBoard’s admin IP is assigned to the bridge. Enable RSTP, and it just works–except for the packet storms coming from the AP-to-AP radio links. If MT would clear that up, I could get back to work.

Getting HWMP+ to work on our existing network is getting to be such a hassle, I’m about ready to bite the bullet and figure out how to configure routed links.

9

Follow-up: this WDS bug was fixed in ROS 5.18.

10

Why not to use ap-bridge to ap-bridge means full WDS?
To make a comment as to why to use Station WDS mode at one end and AP bridge at the other as opposed to the old way configuring corresponding radios on each RB to ap-bridge, dynamic WDS (ap-bridge to ap-bridge means full WDS, which can be somewhat troublesome).

Why is it troublesome? You will never get a link to register stay up (uptime) for days, most likely 5 or so link dropouts in a day, this due to the beacons being transmitted from both AP’s. When beacons happen to transmit together a link dropout occurs.

Back in the older days 2.x and 3.x the Station WDS option did not exist in mikrotik software so there was no other way to do this. Took me and other mikrotik experts maybe a year or so before we understood the reason and advantage of Station WDS. Station WDS basically emulates all the “Station” mode properties with no beacon and very stable, but with bridging added. Then it is well worth the extra effort /cost to place a second radio at a site with minimum 40cm between antenna’s to minimizing interference (link dropouts) and increase throughput too. Never put 3 radios in a RB433, keep the middle slot free for the same reason, minimizing interference and massively increasing throughput, separate boards are the best.

I do not use Station Bridge mode as it tends to create some strange network issues after a few hours, all be it, it is very handy to set a up a link to get initial connection till one can configure the Station WDS mode. A Mikrotik unit in Station Bridge does not need the other end AP mikrotik needing a WDS MAC address to be able to connect with the AP in a bridged setup.

11

what’s with the 18month old thread resurrection?

12

Actually I’m interested to hear about potential problems with the way I have been doing things, if it gives me a good reason to change them.

It’s an interesting theory. Looking through the half-dozen or so ap-bridge to ap-bridge links currently in my network, most of them have uptime of only a few hours–but one of them, the least-obstructed, has been going for 9 days straight. I chalk it up to the difficult environment: towers are not available, so we are going from rooftop-to-rooftop 50-60’ above sea level at most, in some cases across water, in all cases with trees and buildings near (if not within) the Fresnel zone.

I do occasionally see a “no beacons received” message in the logs, but it seems to me that any point-to-point radio link would be subject to brief interference, and that it should take several such “collisions” in a row before ROS would drop any connection. Is it in fact the case that ap-bridge to ap-bridge is more “brittle” than other types of connection, for this reason or some other?