Bonding 2 Wireless device to give more bandwidth

opalit · Tue Apr 21, 2020 10:40 am

I am trying to bond two PTP wireless links to a mountain site that I have and use to distribute broadband around the community, I have a MikroTik RB1100AHx4 ( soon to be replaced CCR 1009 ), one link uses two LHG 60G's and the other uses two Ubiquiti Lightbeam AC GEN 2's, this link is only used if the LHG's go down and manually switched, which usually only happens in Snow, however with this Corona Virus lockdown we are maxing out the LHG link, at the mountain end of the link is a Ubiquiti Edgeswitch 16 waterproof pole mounted POE Switch, ( It would be a MikroTik if you made one ), all the examples seem to show bonding of two internet WAN connections or links between two MikroTik routers and I have tried a number of them without success, can anyone help.

Zacharias · Tue Apr 21, 2020 7:58 pm

Well, you could use LACP, 802.3ad protocol between the Mikrotik and the Edge switch, if the Ubiquiti supports the protocol as well, you will have to check that out...
Also my concern is that CCR1009 will not support the protocol in hardware level... It would be best to use the CCR as router and a CRS328 as a switch where the Antennas are connected on...

Another thought, if you had a Mikrotik on the other side of the link as well instead of the Ubiquiti, to use OSPF and configure in a way that one PTP Link will be used for the Download Traffic and the Second PTP link will be used for the Upload traffic... But you do not have a Mikrotik there...

opalit · Tue Apr 21, 2020 9:38 pm

Well, you could use LACP, 802.3ad protocol between the Mikrotik and the Edge switch, if the Ubiquiti supports the protocol as well, you will have to check that out...
Also my concern is that CCR1009 will not support the protocol in hardware level... It would be best to use the CCR as router and a CRS328 as a switch where the Antennas are connected on...

Another thought, if you had a Mikrotik on the other side of the link as well instead of the Ubiquiti, to use OSPF and configure in a way that one PTP Link will be used for the Download Traffic and the Second PTP link will be used for the Upload traffic... But you do not have a Mikrotik there...

I Mikrotik made a 16 Port waterproof pole mounted switch I would have a one on the pole

sindy · Tue Apr 21, 2020 10:37 pm

The EdgeSwitch manual doesn't mention any type of port aggregation, and the LHG60 has only got a single GE port whose bandwidth matches the available bandwidth of the radio link, so there is no margin left for any additional traffic, so it makes no sense to create an L2 tunnel between the two LHG60 via the Ubiquiti link and aggregate it with the direct link via the LGH60's wireless interfaces.

Hence the only thing I can imagine is to use two VLANs on the EdgeSwitch, one per each radio link, and distribute the other devices on the mountain site into those two VLANs so that devices in each group would only share the bandwidth of that link. If done in the "properly wrong" way, i.e. if you just keep the EdgeSwitch port connected to the LHG an access port to the default VLAN 1, and the Edgeswitch port connected to the Ubiquiti radio an access port to some other one, whilst on the 1100 end both radios will be connected to the same (or none) VLAN, you won't need to change anything in IP configuration of the other devices on the mountain site as both the VLANs will be interconnected at the 1100. So the traffic distribution among the radio links will be static and you can fine-tune it by moving the other devices between the VLANs.

Whether this will work or not depends on whether the EdgeSwitch supports independent MAC address learning per VLAN. If not, you'd have to do some dirty tricks using /interface bridge nat at the 1100 so that the IP addresses of anything in that subnet that is physically connected at the base camp site would resolve to different MAC addresses depening on from where the ARP request comes.

And, of course, the share of traffic among the other devices on the mountain site must be negligible, as it will flow through the base camp (and if the independent VLAN learning doesn't work, such traffic will be impossible at all).

opalit · Wed Apr 22, 2020 12:00 am

OH, this will not be easy to do on a very hot live link, I think I will have to split them manually by installing a PowerBox Pro connected to one link and some of the radios connected in to that.

sindy · Wed Apr 22, 2020 12:34 am

OH, this will not be easy to do on a very hot live link, I think I will have to split them manually by installing a PowerBox Pro connected to one link and some of the radios connected in to that.

What complexity can you see in a runtime reconfiguration? You say the Ubiquiti link is currently not in use, so let's disconnect it at both ends.
Then, prepare an access port to VLAN 2 for it at the mountain site, and connect it to that port.
Remember that STP must be disabled everywhere, as spanning tree BPDUs are VLAN-agnostic (well, not in all modes, but those which live in independent VLANs are usually incompatible between vendors anyway). Next, connect the lower end of the Ubiquiti link at low traffic time - if doing so doesn't cause a loop, it means that the separation of the two links at the mountain site by VLAN IDs works, and the basic job is done by this, although all client traffic remains on the LHG60 link. If it causes a loop, which you notice by loss of ping to one of the client devices, you just disconnect the Ubiquiti radio from the 1100 again and that's it.

Then, to make a client device on the EdgeSwitch use the Ubiquiti link, it is enough to change the PVID of the port to which it is connected 1 to 2. Just a few packets may get dropped as the port will change mode. So the effective result will be the same as if you added the PowerBox Pro, except that you won't need to install the PowerBox Pro.

opalit · Wed Apr 22, 2020 12:48 am

Thanks, everytime I have played with VLAN's live, ( they seem to work on the bench ) I end up getting in a mess and locked out, so I have a couple of questions.
Do I have to add VLAN's to the AP Radios that are connecting to the Edgeswitch and the customers CPE's connecting to those AP's or just create the VLAN's in the EdgeSwitch itself

sindy · Wed Apr 22, 2020 1:02 am

No changes anywhere but on the edgeswitch. The whole exercise is just to partition the edgeswitch into two virtual switches using the vlan 2 which will imitate the powerbox.

opalit · Wed Apr 22, 2020 1:30 am

No changes anywhere but on the edgeswitch. The whole exercise is just to partition the edgeswitch into two virtual switches using the vlan 2 which will imitate the powerbox.

Just trying this at the moment, I have created a VLAN ID 100 on the EdgeSwitch, excluded the link from VLAN ID 1 and added Untagged to the port the link radio is connected too and as the MicroTik end is disconnected I cannot access the top end of the link which is to be expected, after connecting to MicroTik end of link to to the 1100 I can now get to the top of the link

Now I am a little bit stuck, do I assign the the ports connected to the AP radios to the new VLAN, I assume I do.

S**t, I think it is working, just tried it with one radio with just a couple of users on, MikroTik end disconnected, no access to radio, connect MikroTik end access to only that one Radio

***** A massive thank you

Zacharias · Wed Apr 22, 2020 1:35 pm

I turned off my simple QoS queue so I could get fast track back on. I thought that would help.

@sindy i said LACP between Mikrotik and Ubiquiti in case the later supports it... This makes us 4 Antennas and not 2 as you said... And the antennas would not play any role in the actual LACP process except the ptp connection of sites... We have share of bandwidth between links, not equal of corse but it is something and redundancy as well, which i consider super important...

Your idea of VLANs is nice but i do consider it as a temp solution... If a link fails someone should manually switch the VLANs... I would prefer a slower connection than no connection at all...
Imagine being at home without Internet, calling your ISP to see what's wrong, and the answer you get is "just a minute sir we are switching some VLANs" really ?

opalit · Wed Apr 22, 2020 1:57 pm

I turned off my simple QoS queue so I could get fast track back on. I thought that would help.
@sindy i said LACP between Mikrotik and Ubiquiti in case the later supports it... This makes us 4 Antennas and not 2 as you said... And the antennas would not play any role in the actual LACP process except the ptp connection of sites... We have share of bandwidth between links, not equal of corse but it is something and redundancy as well, which i consider super important...

Your idea of VLANs is nice but i do consider it as a temp solution... If a link fails someone should manually switch the VLANs... I would prefer a slower connection than no connection at all...
Imagine being at home without Internet, calling your ISP to see what's wrong, and the answer you get is "just a minute sir we are switching some VLANs" really ?

Yes it is a temporary solution to the excess traffic caused by the corona Virus Lockdown, while working on the EdgeSwitch I noticed it does LAG ( link Aggregation Groups ), this is similar to the Mikrotik bonding and supports load balancing and fail over, however experimenting with this on a hot unit is a no no for me, I am at the moment too ill with a virus ( no tests available for us UK minions ) to go up to the mast to fix if I screw up.

Zacharias · Wed Apr 22, 2020 1:59 pm

I am at the moment to ill to go up to the mast to fix if I screw up.

Yes that would be bad am sure

opalit · Wed Apr 22, 2020 2:02 pm

I am at the moment to ill to go up to the mast to fix if I screw up.
Yes that would be bad am sure

I have 10 Live AP's and one data only link on the S16 so cannot risk a screw up, I do have a Pico radio on the back of the S16, so I do have a way back in without climbing the mast if I scew up.

Zacharias · Wed Apr 22, 2020 2:08 pm

I see...
Can you sum up the overall changes you had to do as far as the VLANs are concerned ?

opalit · Wed Apr 22, 2020 2:21 pm

I see...
Can you sum up the overall changes you had to do as far as the VLANs are concerned ?

Switched S16 to Legacy mode, ( easier ), create a VLAN ID 100, change the U on the secondary link port to E ( Exclude ) on default VLAN ID 1, change the corresponding ports setting from E to U on VLAN ID 100, plug in to Mikrotik at bottom end and we now have two links to the S16 isolated from each other, no from traffic decide which AP's traffic you want going through each link and change the corresponding ports to E on VLAN ID 1 and U on VLAN ID 100, simples, see attached picture

Zacharias · Wed Apr 22, 2020 5:06 pm

@opalit the E for the Ubiquiti means it excluded that port from the default Group ?

@sindy, if i wanted to implement the exact same thing with a 24port Mikrotik switch,i would add all 24 ports in my Bridge, then i could let e.g. the 12 first ports with PVID 1 the rest 12 with PVID lets say 2 and i would enable then Bridge VLAN Filtering, thus splitting the switch in two... Correct ?

opalit · Wed Apr 22, 2020 5:36 pm

@opalit the E for the Ubiquiti means it excluded and my backup link port is excluded from the default Group ?

@sindy, if i wanted to implement the exact same thing with a 24port Mikrotik switch,i would add all 24 ports in my Bridge, then i could let e.g. the 12 first ports with PVID 1 the rest 12 with PVID lets say 2 and i would enable then Bridge VLAN Filtering, thus splitting the switch in two... Correct ?

The E does means excluded and the port is excluded from the default group, not tried it on a MicroTik, I would assume you would create a VLAN in the same way on the port and add the other ports to communicate with on the same VLAN

sindy · Wed Apr 22, 2020 7:28 pm

@Zacharias, responding to various points:

i said LACP between Mikrotik and Ubiquiti in case the later supports it... This makes us 4 Antennas and not 2 as you said... And the antennas would not play any role in the actual LACP process except the ptp connection of sites... We have share of bandwidth between links, not equal of corse but it is something and redundancy as well, which i consider super important...

Fun fact: as I am not familiar with Ubnt's feature set nor with the structure of their documentation, the first thing I've done was to look for LACP support in the EdgeSwitch documentation I could google up, but it didn't come to my mind that I should look for configuration items under dashboard->port status. So I've concluded it's not there.

Worse than that is that the link selection strategies in all that bonding/teaming/link aggregation assume that the member links of a bundle have the same bandwidth, and you can't explicitly state any load distribution rule (the link selection algorithms have to be simple so that they could be implemented in hardware), and here we have something like 1 Gbit/s full duplex on the LHG60 and at best 500 Mbit/s full duplex on the Ubiquity link. Plus the aim of those link selection strategies is to use the same link for all packets of the same application connection in order to avoid packet missequencing, but there is no coordination between directions, so the two directions of the sam application connection can use different links. So the ability to manually assign each device at the mountain site to one of the links allows to distribute the available bandwidth between them in a more controlled way.

4 point-to-point antennas make two links to me, so I don't understand what is your point here.

Your idea of VLANs is nice but i do consider it as a temp solution... If a link fails someone should manually switch the VLANs... I would prefer a slower connection than no connection at all...
Imagine being at home without Internet, calling your ISP to see what's wrong, and the answer you get is "just a minute sir we are switching some VLANs" really ?

This is no worse that the initial state @opalit has described:

the other uses two Ubiquiti Lightbeam AC GEN 2's, this link is only used if the LHG's go down and manually switched

So the whole thing is of course a temp solution, but the preference of higher throughput to redundancy has been clearly stated.

if i wanted to implement the exact same thing with a 24port Mikrotik switch,i would add all 24 ports in my Bridge, then i could let e.g. the 12 first ports with PVID 1 the rest 12 with PVID lets say 2 and i would enable then Bridge VLAN Filtering, thus splitting the switch in two... Correct ?

Correct.

opalit · Wed Apr 22, 2020 7:39 pm

@Zacharias, responding to various points:
i said LACP between Mikrotik and Ubiquiti in case the later supports it... This makes us 4 Antennas and not 2 as you said... And the antennas would not play any role in the actual LACP process except the ptp connection of sites... We have share of bandwidth between links, not equal of corse but it is something and redundancy as well, which i consider super important...
Fun fact: as I am not familiar with Ubnt's feature set nor with the structure of their documentation, the first thing I've done was to look for LACP support in the EdgeSwitch documentation I could google up, but it didn't come to my mind that I should look for configuration items under dashboard->port status. So I've concluded it's not there.

Worse than that is that the link selection strategies in all that bonding/teaming/link aggregation assume that the member links of a bundle have the same bandwidth, and you can't explicitly state any load distribution rule (the link selection algorithms have to be simple so that they could be implemented in hardware), and here we have something like 1 Gbit/s full duplex on the LHG60 and at best 500 Mbit/s full duplex on the Ubiquity link. Plus the aim of those link selection strategies is to use the same link for all packets of the same application connection in order to avoid packet missequencing, but there is no coordination between directions, so the two directions of the sam application connection can use different links. So the ability to manually assign each device at the mountain site to one of the links allows to distribute the available bandwidth between them in a more controlled way.

4 point-to-point antennas make two links to me, so I don't understand what is your point here.

Your idea of VLANs is nice but i do consider it as a temp solution... If a link fails someone should manually switch the VLANs... I would prefer a slower connection than no connection at all...
Imagine being at home without Internet, calling your ISP to see what's wrong, and the answer you get is "just a minute sir we are switching some VLANs" really ?
This is no worse that the initial state @opalit has described:
the other uses two Ubiquiti Lightbeam AC GEN 2's, this link is only used if the LHG's go down and manually switched
So the whole thing is of course a temp solution, but the preference of higher throughput to redundancy has been clearly stated.

if i wanted to implement the exact same thing with a 24port Mikrotik switch,i would add all 24 ports in my Bridge, then i could let e.g. the 12 first ports with PVID 1 the rest 12 with PVID lets say 2 and i would enable then Bridge VLAN Filtering, thus splitting the switch in two... Correct ?
Correct.

Initially I wanted to add a PowerBox pro to the mast and split the AP's between S16 and Powerbox, the VLAN's have done that job by splitting the S16 in to what is a effectively now two switches, this will do for the duration of the Lockdown, I have experimented before trying to get bonding, with failover, simple with mikrotik but not so with Ubiquiti, I have an 8 Port Ubiquiti Switch and a spare Mikrotik switch I am going to experiment with bonding and load balancing once I am happy I will do it on the live system.

Zacharias · Wed Apr 22, 2020 7:52 pm

so the two directions of the sam application connection can use different links

@sindy LACP 802.3ad does not split traffic accross links...
All packets associated with a given “conversation” are
transmitted on the same link to prevent mis-ordering
http://www.ieee802.org/3/hssg/public/ap ... 1_0407.pdf

The only thing i did not take into account, is that the correct implementation of a 802.3ad, needs the links to be of the same bandwidth capabilities...

sindy · Wed Apr 22, 2020 8:10 pm

so the two directions of the sam application connection can use different links
@sindy LACP 802.3ad does not split traffic accross links...
All packets associated with a given “conversation” are
transmitted on the same link to prevent mis-ordering

We talk about the same thing. The traffic as a whole is being distributed among the links, but the distribution rules take into account source and/or destination addresses and/or ports to ensure that packets belonging to the same "conversation" (or "application connection") use always the same link (as long as it is available). But I have to inisist that there is no coordination between the directions, each sender chooses the links independently, as the only aim of using the same link for all packets belonging to the same conversation is to prevent packets overtaking each other in the same direction.

Zacharias · Wed Apr 22, 2020 8:25 pm

Sure there is no coordination and the links obviously will not share the traffic equally. However am sure it would help to a certain point...

opalit · Wed Apr 22, 2020 8:58 pm

Sure there is no coordination and the links obviously will not share the traffic equally. However am sure it would help to a certain point...

My 8 Port Ubiquiti does not have LAG ( Link Aggregation Groups ) so cannot test Bonding and failover.

sindy · Wed Apr 22, 2020 9:27 pm

My 8 Port Ubiquiti does not have LAG ( Link Aggregation Groups ) so cannot test Bonding and failover.

Well, you could still test using the "real" switch up there, as the LAG can have a single active link (or even a single link at all, but the EdgeSwitch's configuration seems to require at least two ports when creating a LAG, so you'll have to include an unused port to the LAG there). So if you move everyone up there back to the VLAN fed by the LHG60 link, the Ubiquiti link becomes free again, and you can connect the test Mikrotik (or, in worst case, a separate bridge on the 1100) against the ES via the Ubiquity link. So the ES port connected to the Ubiquiti radio will become one of the LAG's member ports, and so will the physical port of the Mikrotik in the base camp. Using this setup, you can check whether both ends detect properly the unavailability of the remote switch when both radios stay up and connected on the Ethernet side but the link over the air breaks. This proper detection of a failed link is the key to successful failover, and some recent topic here deals with a case where Mikrotik's bond in 802.3ad (LACP) mode ignores the fact that the remote peer connected via an active transport equipment (i.e. not just a passive cable) is inaccessible.

opalit · Wed Apr 22, 2020 10:05 pm

My 8 Port Ubiquiti does not have LAG ( Link Aggregation Groups ) so cannot test Bonding and failover.
Well, you could still test using the "real" switch up there, as the LAG can have a single active link (or even a single link at all, but the EdgeSwitch's configuration seems to require at least two ports when creating a LAG, so you'll have to include an unused port to the LAG there). So if you move everyone up there back to the VLAN fed by the LHG60 link, the Ubiquiti link becomes free again, and you can connect the test Mikrotik (or, in worst case, a separate bridge on the 1100) against the ES via the Ubiquity link. So the ES port connected to the Ubiquiti radio will become one of the LAG's member ports, and so will the physical port of the Mikrotik in the base camp. Using this setup, you can check whether both ends detect properly the unavailability of the remote switch when both radios stay up and connected on the Ethernet side but the link over the air breaks. This proper detection of a failed link is the key to successful failover, and some recent topic here deals with a case where Mikrotik's bond in 802.3ad (LACP) mode ignores the fact that the remote peer connected via an active transport equipment (i.e. not just a passive cable) is inaccessible.

Not risking it, it is working well at the moment, already clicked the wrong port and screwed it up once today and I am too ill to be climbing masts, after a power off of the S16 it would not come back on, I had to remove all the POE connections to get it back on, then connect them to the AP's one at a time after it came back on.

sindy · Wed Apr 22, 2020 11:40 pm

There are pro's and cons to be the one and only. I'm lucky I've always had other people able to climb the mast instead of me when necessary so far. Get well soon. That thing is tricky, you think it's over and then it strikes the second time.

But I've just done a test here, and although @nerdafterdark is right in his finding that the fact that no LACP frames are coming from the remote is not a sufficient reason for RouterOS to declare the whole bonding interface (=LAG group) down even if you set min-links=1, the failover itself works well. When testing between two Mikrotiks, if I remove the remote port through which the traffic runs from the slaves list (so it stays up at L1 but doesn't send LACP frames any more), the traffic which was running through it migrates to the other port in a short while also at the local end where both ports remain bond slaves.

Zacharias · Thu Apr 23, 2020 12:27 am

Not risking

If you test it in the future let us know how it goes...

opalit · Thu Apr 23, 2020 6:16 am

That thing is tricky, you think it's over and then it strikes the second time.

And the third time over the last 5 weeks

opalit · Thu Apr 23, 2020 12:15 pm

There are pro's and cons to be the one and only. I'm lucky I've always had other people able to climb the mast instead of me when necessary so far. Get well soon. That thing is tricky, you think it's over and then it strikes the second time.

But I've just done a test here, and although @nerdafterdark is right in his finding that the fact that no LACP frames are coming from the remote is not a sufficient reason for RouterOS to declare the whole bonding interface (=LAG group) down even if you set min-links=1, the failover itself works well. When testing between two Mikrotiks, if I remove the remote port through which the traffic runs from the slaves list (so it stays up at L1 but doesn't send LACP frames any more), the traffic which was running through it migrates to the other port in a short while also at the local end where both ports remain bond slaves.

On thing I found when using Mikrotik normal bonding with wireless links, is it cannot detect the wireless link down because there is still a LAN link to the radio, have not found any solutions by googling yet.

sindy · Thu Apr 23, 2020 1:36 pm

That's exactly the scenario I wrote about. In the 802.3ad (LACP) mode, /interface bonding can detect unavailability of remote peer switch via a given link caused by non-transparency of an active transmission equipment (in your case, the microwave link) between them even though the physical slave interface of the bond stays up, because it doesn't receive the LACP control frames from the remote end on that slave interface, or it does receive them but the flags in the received ones indicate that the remote peer is not receiving any LACP frames from the local side (if the link is only broken in local->remote direction). The only issue is that even if less than min-links links are available, the bond interface itself still says it is active, but that only matters in specific cases, not in your one. But otherwise it does what is needed, as it only sends traffic via links that are are currently transparent. The state change of each link is detected with a delay of several seconds, though.

Just for the record, if the remote peer switch doesn't support LACP, you cannot use mode=802.3ad, and the only way to check link transparency is to use link-monitoring=arp and set a list of some remote IP addresses in a subnet accessible via the remote LAG interface into arp-ip-targets.

Zacharias · Thu Apr 23, 2020 1:57 pm

have not found any solutions by googling yet.

The solution is where it should be, in the Manual

ARP monitoring sends ARP queries and uses the response as an indication that the link is operational
https://wiki.mikrotik.com/wiki/Manual:I ... Monitoring

Notice though, that this mode is not recommended for all type of bondings e.g. for active-backup....

opalit · Thu Apr 23, 2020 2:52 pm

have not found any solutions by googling yet.
The solution is where it should be, in the Manual
ARP monitoring sends ARP queries and uses the response as an indication that the link is operational
https://wiki.mikrotik.com/wiki/Manual:I ... Monitoring

Notice though, that this mode is not recommended for all type of bondings e.g. for active-backup....

The solutions in the manual only work with direct cable connection not with wireless links, if a cable connection is down it fails over, i.e. you have unplugged it, if a wirless link is down it does not work because there is still a good cable connection, anyway not messing about, I tried to setup a LAG on the Ubiquiti S16 and it locked up.

opalit · Thu Apr 23, 2020 3:02 pm

The EdgeSwitch manual doesn't mention any type of port aggregation, the EP-S16 has LAG ( Link Aggregation Groups ), and supposedly with load balancing

Zacharias · Thu Apr 23, 2020 3:35 pm

The solutions in the manual only work with direct cable connection not with wireless links

That is actually wrong...
It all depends on the Bonding Mode used...
The bonding does not know if it has an Antenna connected to it or a wire, so it is up to you to perform a correct implementation of the bonding...
As posted earlier, you can use ARP Monitoring... And i am always talking about Mikrotik, i do not know what Ubiquiti does or doesnt...
LACP 802.3ad does not care if it is ethernet cable, optical, antenna or whatever, because it has other techniques to check for the operation of the link...
If you choose lets say balance-rr you can use ARP monitor and you will still be fine because you manually monitor the link...

Also, 802.3ad MUST be supported by both devices to work...

opalit · Thu Apr 23, 2020 3:43 pm

The solutions in the manual only work with direct cable connection not with wireless links
That is actually wrong...
It all depends on the Bonding Mode used...
The bonding does not know if it has an Antenna connected to it or a wire, so it is up to you to perform a correct implementation of the bonding...
As posted earlier, you can use ARP Monitoring... And i am always talking about Mikrotik, i do not know what Ubiquiti does or doesnt...
LACP 802.3ad does not care if it is ethernet cable, optical, antenna or whatever, because it has other techniques to check for the operation of the link...
If you choose lets say balance-rr you can use ARP monitor and you will still be fine because you manually monitor the link...

Also, 802.3ad MUST be supported by both devices to work...

I have experimented with all forms of the Mikrotik bonding and on every occasion turning off the the wifi does not cause it to failover, only disconnecting the LAN port causes it, I set up a test rig using 2 x Mikrotik switches and 4 x SXT radios in PTP bridge mode as to wire replacement links, turning off the ireless on one of the links did not failover

sindy · Thu Apr 23, 2020 4:15 pm

I have experimented with all forms of the Mikrotik bonding and on every occasion turning off the the wifi does not cause it to failover, only disconnecting the LAN port causes it, I set up a test rig using 2 x Mikrotik switches and 4 x SXT radios in PTP bridge mode as to wire replacement links, turning off the ireless on one of the links did not failover

That's really weird, my yesterday's tests show otherwise. Physical link up, piniging, sniffer shows which slave carries the pings; remove that link's interface from the list of slaves at the remote end, it stops sending LACP frames, the pings start being sent via the other link in a few seconds. With mode=802.3ad, link-monitoring=none. I was running the test on 6.45.7, what about you? But I admit that removal of slave interface from a bond may cause LACP restart, so I'll insert a dumb switch instead and report back.

A few induced thoughts:

on the radios, protocol-mode on the bridge between the Ethernet port and the wireless one must be set to none, otherwise the LACP frames are not forwarded. Are you aware of this?
since the Ethernet port is the only way to manage the radios, as soon as you connect the it to a switch interface which is a slave in a bond (of whatever kind), the management access to the radio may or may not be possible depending on the L2, L3, L4 criteria used to chose the slave link, so don't be surprised by this

I tried to setup a LAG on the Ubiquiti S16 and it locked up.

So you had to climb the mast after all???

sindy · Thu Apr 23, 2020 6:39 pm

Update, instead of the dumb switch I've decided to insert another Mikrotik into one link between the two ones with bonding, and connected the cables in such a way that ping requests took the path through the inserted Mikrotik and the ping responses took the direct cable while both paths were transparent. And then I used bridge filter rules on the "mid-cable" Mikrotik to drop LACP frames, first in one direction and then in the opposite one. In either case, i.e. both if the "local" (pinging) Mikrotik was not receiving LACP frames from the "remote" (pinged) one as well as when it received them but they carried an information that the remote one was getting no LACP frames from the local one, the ping requests moved to the direct link, and once the controlled one became transparent again, they moved back.

So as for me, everything works as it should. With mode=802.3ad and link-monitoring=none, the LACP control traffic alone is enough to determine the link transparency, and the failover works properly.

Zacharias · Thu Apr 23, 2020 7:05 pm

on the radios, protocol-mode on the bridge between the Ethernet port and the wireless one must be set to none

You mean disable RSTP on the Bridge ?

sindy · Thu Apr 23, 2020 7:39 pm

You mean disable RSTP on the Bridge ?

Ναί, but that's secondary in this case. The primary reason is that protocol-mode=none not only switches off any STP flavors on the bridge, but also makes it break the 802.1D requirement that frames with destination MAC address prefix 01:80:c2 must never be forwarded by a switch/bridge, as they are reserved for communication with adjacent elements. And LACP is one of these protocols.

Zacharias · Thu Apr 23, 2020 7:46 pm

Ναί

Where are you from? if i may ask...

As far as i ve tested LACP works just fine with RSTP enabled on the Bridge where the bonding exists... Nor i can find any reference that RSTP must be disabled...
So, i don't really understand...

sindy · Thu Apr 23, 2020 8:11 pm

Where are you from? if i may ask...

Take the road to Berlin and stop in the previous country on the way.

As far as i ve tested LACP works just fine with RSTP enabled on the Bridge where the bonding exists... Nor i can find any reference that RSTP must be disabled...

I haven't said that the protocol-mode=none must be set on the bridge whose member port the bond interface is; this requirement is relevant for the bridges on the active network path between the two devices running LACP. Each of the two LACP-enabled switches (bridges) is connected to an Ethernet interface of one of the LHG60, and inside each LHG60, the Ethernet interface and the wireless interface must be bridged together for this application case (L2 transparent channel between the Ethernets over the air). And to make them transparent also for the LACP frames, both these bridges inside the LHG60 must be configured with protocol-mode=none. Details in the manual: "Since RouterOS v6.43 it is possible to forward Reserved MAC addresses that are in 01:80:C2:XX:XX:XX range, this can be done by setting the protocol-mode to none."

Zacharias · Thu Apr 23, 2020 8:25 pm

Poland i guess...

this requirement is relevant for the bridges on the active network path between the two devices running LACP

Ok that makes it clear to me now...
I made a quick capture with Wireshark under GNS3 and i could see the reserved MAC address range in the LLDP packets...

sindy · Thu Apr 23, 2020 8:31 pm

Poland i guess...

A narrow miss, keep a bit more to the west while driving

I made a quick capture with Wireshark under GNS3 and i could see the reserved MAC address range in the LLDP packets...

Yes, LLDP is the same case like STP BPDUs and LACP frames, all of these should normally not get past the adjacent device.

Zacharias · Thu Apr 23, 2020 8:35 pm

Ok, nice to know that...

opalit · Fri Apr 24, 2020 6:10 am

]
So you had to climb the mast after all???
[/quote]

Yes and at 72 years old, and suffering with a virus infection, no mean feat I can tell you.

I have just dug out the old SXT's and Sextants to create a simulation of the links to the mountain, so I can practise what you say in some of these posts

Mikrotik are supposed to be bringing out a Netpower 16P , a 16Port pole mounted POE waterproof switch, similar to the Ubiquiti S16, I think this will solve my problems, the links will then be MikroTik to MikroTik

Zacharias · Fri Apr 24, 2020 2:16 pm

Mikrotik are supposed to be bringing out a Netpower 16P

You can already find and purchase this product...

opalit · Fri Apr 24, 2020 8:22 pm

Mikrotik are supposed to be bringing out a Netpower 16P
You can already find and purchase this product...

Not yet seen the 16P for sale

Who is online