I Mikrotik made a 16 Port waterproof pole mounted switch I would have a one on the poleWell, you could use LACP, 802.3ad protocol between the Mikrotik and the Edge switch, if the Ubiquiti supports the protocol as well, you will have to check that out...
Also my concern is that CCR1009 will not support the protocol in hardware level... It would be best to use the CCR as router and a CRS328 as a switch where the Antennas are connected on...
Another thought, if you had a Mikrotik on the other side of the link as well instead of the Ubiquiti, to use OSPF and configure in a way that one PTP Link will be used for the Download Traffic and the Second PTP link will be used for the Upload traffic... But you do not have a Mikrotik there...
What complexity can you see in a runtime reconfiguration? You say the Ubiquiti link is currently not in use, so let's disconnect it at both ends.OH, this will not be easy to do on a very hot live link, I think I will have to split them manually by installing a PowerBox Pro connected to one link and some of the radios connected in to that.
Just trying this at the moment, I have created a VLAN ID 100 on the EdgeSwitch, excluded the link from VLAN ID 1 and added Untagged to the port the link radio is connected too and as the MicroTik end is disconnected I cannot access the top end of the link which is to be expected, after connecting to MicroTik end of link to to the 1100 I can now get to the top of the linkNo changes anywhere but on the edgeswitch. The whole exercise is just to partition the edgeswitch into two virtual switches using the vlan 2 which will imitate the powerbox.
@sindy i said LACP between Mikrotik and Ubiquiti in case the later supports it... This makes us 4 Antennas and not 2 as you said... And the antennas would not play any role in the actual LACP process except the ptp connection of sites... We have share of bandwidth between links, not equal of corse but it is something and redundancy as well, which i consider super important...I turned off my simple QoS queue so I could get fast track back on. I thought that would help.
Yes it is a temporary solution to the excess traffic caused by the corona Virus Lockdown, while working on the EdgeSwitch I noticed it does LAG ( link Aggregation Groups ), this is similar to the Mikrotik bonding and supports load balancing and fail over, however experimenting with this on a hot unit is a no no for me, I am at the moment too ill with a virus ( no tests available for us UK minions ) to go up to the mast to fix if I screw up.@sindy i said LACP between Mikrotik and Ubiquiti in case the later supports it... This makes us 4 Antennas and not 2 as you said... And the antennas would not play any role in the actual LACP process except the ptp connection of sites... We have share of bandwidth between links, not equal of corse but it is something and redundancy as well, which i consider super important...I turned off my simple QoS queue so I could get fast track back on. I thought that would help.
Your idea of VLANs is nice but i do consider it as a temp solution... If a link fails someone should manually switch the VLANs... I would prefer a slower connection than no connection at all...
Imagine being at home without Internet, calling your ISP to see what's wrong, and the answer you get is "just a minute sir we are switching some VLANs" really ?
Yes that would be bad am sureI am at the moment to ill to go up to the mast to fix if I screw up.
I have 10 Live AP's and one data only link on the S16 so cannot risk a screw up, I do have a Pico radio on the back of the S16, so I do have a way back in without climbing the mast if I scew up.Yes that would be bad am sureI am at the moment to ill to go up to the mast to fix if I screw up.
Switched S16 to Legacy mode, ( easier ), create a VLAN ID 100, change the U on the secondary link port to E ( Exclude ) on default VLAN ID 1, change the corresponding ports setting from E to U on VLAN ID 100, plug in to Mikrotik at bottom end and we now have two links to the S16 isolated from each other, no from traffic decide which AP's traffic you want going through each link and change the corresponding ports to E on VLAN ID 1 and U on VLAN ID 100, simples, see attached pictureI see...
Can you sum up the overall changes you had to do as far as the VLANs are concerned ?
The E does means excluded and the port is excluded from the default group, not tried it on a MicroTik, I would assume you would create a VLAN in the same way on the port and add the other ports to communicate with on the same VLAN@opalit the E for the Ubiquiti means it excluded and my backup link port is excluded from the default Group ?
@sindy, if i wanted to implement the exact same thing with a 24port Mikrotik switch,i would add all 24 ports in my Bridge, then i could let e.g. the 12 first ports with PVID 1 the rest 12 with PVID lets say 2 and i would enable then Bridge VLAN Filtering, thus splitting the switch in two... Correct ?
Fun fact: as I am not familiar with Ubnt's feature set nor with the structure of their documentation, the first thing I've done was to look for LACP support in the EdgeSwitch documentation I could google up, but it didn't come to my mind that I should look for configuration items under dashboard->port status. So I've concluded it's not there.i said LACP between Mikrotik and Ubiquiti in case the later supports it... This makes us 4 Antennas and not 2 as you said... And the antennas would not play any role in the actual LACP process except the ptp connection of sites... We have share of bandwidth between links, not equal of corse but it is something and redundancy as well, which i consider super important...
This is no worse that the initial state @opalit has described:Your idea of VLANs is nice but i do consider it as a temp solution... If a link fails someone should manually switch the VLANs... I would prefer a slower connection than no connection at all...
Imagine being at home without Internet, calling your ISP to see what's wrong, and the answer you get is "just a minute sir we are switching some VLANs" really ?
So the whole thing is of course a temp solution, but the preference of higher throughput to redundancy has been clearly stated.the other uses two Ubiquiti Lightbeam AC GEN 2's, this link is only used if the LHG's go down and manually switched
Correct.if i wanted to implement the exact same thing with a 24port Mikrotik switch,i would add all 24 ports in my Bridge, then i could let e.g. the 12 first ports with PVID 1 the rest 12 with PVID lets say 2 and i would enable then Bridge VLAN Filtering, thus splitting the switch in two... Correct ?
Initially I wanted to add a PowerBox pro to the mast and split the AP's between S16 and Powerbox, the VLAN's have done that job by splitting the S16 in to what is a effectively now two switches, this will do for the duration of the Lockdown, I have experimented before trying to get bonding, with failover, simple with mikrotik but not so with Ubiquiti, I have an 8 Port Ubiquiti Switch and a spare Mikrotik switch I am going to experiment with bonding and load balancing once I am happy I will do it on the live system.@Zacharias, responding to various points:Fun fact: as I am not familiar with Ubnt's feature set nor with the structure of their documentation, the first thing I've done was to look for LACP support in the EdgeSwitch documentation I could google up, but it didn't come to my mind that I should look for configuration items under dashboard->port status. So I've concluded it's not there.i said LACP between Mikrotik and Ubiquiti in case the later supports it... This makes us 4 Antennas and not 2 as you said... And the antennas would not play any role in the actual LACP process except the ptp connection of sites... We have share of bandwidth between links, not equal of corse but it is something and redundancy as well, which i consider super important...
Worse than that is that the link selection strategies in all that bonding/teaming/link aggregation assume that the member links of a bundle have the same bandwidth, and you can't explicitly state any load distribution rule (the link selection algorithms have to be simple so that they could be implemented in hardware), and here we have something like 1 Gbit/s full duplex on the LHG60 and at best 500 Mbit/s full duplex on the Ubiquity link. Plus the aim of those link selection strategies is to use the same link for all packets of the same application connection in order to avoid packet missequencing, but there is no coordination between directions, so the two directions of the sam application connection can use different links. So the ability to manually assign each device at the mountain site to one of the links allows to distribute the available bandwidth between them in a more controlled way.
4 point-to-point antennas make two links to me, so I don't understand what is your point here.
This is no worse that the initial state @opalit has described:Your idea of VLANs is nice but i do consider it as a temp solution... If a link fails someone should manually switch the VLANs... I would prefer a slower connection than no connection at all...
Imagine being at home without Internet, calling your ISP to see what's wrong, and the answer you get is "just a minute sir we are switching some VLANs" really ?So the whole thing is of course a temp solution, but the preference of higher throughput to redundancy has been clearly stated.the other uses two Ubiquiti Lightbeam AC GEN 2's, this link is only used if the LHG's go down and manually switched
Correct.if i wanted to implement the exact same thing with a 24port Mikrotik switch,i would add all 24 ports in my Bridge, then i could let e.g. the 12 first ports with PVID 1 the rest 12 with PVID lets say 2 and i would enable then Bridge VLAN Filtering, thus splitting the switch in two... Correct ?
@sindy LACP 802.3ad does not split traffic accross links...so the two directions of the sam application connection can use different links
We talk about the same thing. The traffic as a whole is being distributed among the links, but the distribution rules take into account source and/or destination addresses and/or ports to ensure that packets belonging to the same "conversation" (or "application connection") use always the same link (as long as it is available). But I have to inisist that there is no coordination between the directions, each sender chooses the links independently, as the only aim of using the same link for all packets belonging to the same conversation is to prevent packets overtaking each other in the same direction.@sindy LACP 802.3ad does not split traffic accross links...so the two directions of the sam application connection can use different links
All packets associated with a given “conversation” are
transmitted on the same link to prevent mis-ordering
My 8 Port Ubiquiti does not have LAG ( Link Aggregation Groups ) so cannot test Bonding and failover.Sure there is no coordination and the links obviously will not share the traffic equally. However am sure it would help to a certain point...
Well, you could still test using the "real" switch up there, as the LAG can have a single active link (or even a single link at all, but the EdgeSwitch's configuration seems to require at least two ports when creating a LAG, so you'll have to include an unused port to the LAG there). So if you move everyone up there back to the VLAN fed by the LHG60 link, the Ubiquiti link becomes free again, and you can connect the test Mikrotik (or, in worst case, a separate bridge on the 1100) against the ES via the Ubiquity link. So the ES port connected to the Ubiquiti radio will become one of the LAG's member ports, and so will the physical port of the Mikrotik in the base camp. Using this setup, you can check whether both ends detect properly the unavailability of the remote switch when both radios stay up and connected on the Ethernet side but the link over the air breaks. This proper detection of a failed link is the key to successful failover, and some recent topic here deals with a case where Mikrotik's bond in 802.3ad (LACP) mode ignores the fact that the remote peer connected via an active transport equipment (i.e. not just a passive cable) is inaccessible.My 8 Port Ubiquiti does not have LAG ( Link Aggregation Groups ) so cannot test Bonding and failover.
Not risking it, it is working well at the moment, already clicked the wrong port and screwed it up once today and I am too ill to be climbing masts, after a power off of the S16 it would not come back on, I had to remove all the POE connections to get it back on, then connect them to the AP's one at a time after it came back on.Well, you could still test using the "real" switch up there, as the LAG can have a single active link (or even a single link at all, but the EdgeSwitch's configuration seems to require at least two ports when creating a LAG, so you'll have to include an unused port to the LAG there). So if you move everyone up there back to the VLAN fed by the LHG60 link, the Ubiquiti link becomes free again, and you can connect the test Mikrotik (or, in worst case, a separate bridge on the 1100) against the ES via the Ubiquity link. So the ES port connected to the Ubiquiti radio will become one of the LAG's member ports, and so will the physical port of the Mikrotik in the base camp. Using this setup, you can check whether both ends detect properly the unavailability of the remote switch when both radios stay up and connected on the Ethernet side but the link over the air breaks. This proper detection of a failed link is the key to successful failover, and some recent topic here deals with a case where Mikrotik's bond in 802.3ad (LACP) mode ignores the fact that the remote peer connected via an active transport equipment (i.e. not just a passive cable) is inaccessible.My 8 Port Ubiquiti does not have LAG ( Link Aggregation Groups ) so cannot test Bonding and failover.
If you test it in the future let us know how it goes...Not risking
And the third time over the last 5 weeksThat thing is tricky, you think it's over and then it strikes the second time.
On thing I found when using Mikrotik normal bonding with wireless links, is it cannot detect the wireless link down because there is still a LAN link to the radio, have not found any solutions by googling yet.There are pro's and cons to be the one and only. I'm lucky I've always had other people able to climb the mast instead of me when necessary so far. Get well soon. That thing is tricky, you think it's over and then it strikes the second time.
But I've just done a test here, and although @nerdafterdark is right in his finding that the fact that no LACP frames are coming from the remote is not a sufficient reason for RouterOS to declare the whole bonding interface (=LAG group) down even if you set min-links=1, the failover itself works well. When testing between two Mikrotiks, if I remove the remote port through which the traffic runs from the slaves list (so it stays up at L1 but doesn't send LACP frames any more), the traffic which was running through it migrates to the other port in a short while also at the local end where both ports remain bond slaves.
The solution is where it should be, in the Manualhave not found any solutions by googling yet.
The solutions in the manual only work with direct cable connection not with wireless links, if a cable connection is down it fails over, i.e. you have unplugged it, if a wirless link is down it does not work because there is still a good cable connection, anyway not messing about, I tried to setup a LAG on the Ubiquiti S16 and it locked up.The solution is where it should be, in the Manualhave not found any solutions by googling yet.
ARP monitoring sends ARP queries and uses the response as an indication that the link is operational
https://wiki.mikrotik.com/wiki/Manual:I ... Monitoring
Notice though, that this mode is not recommended for all type of bondings e.g. for active-backup....
The EdgeSwitch manual doesn't mention any type of port aggregation, the EP-S16 has LAG ( Link Aggregation Groups ), and supposedly with load balancing
That is actually wrong...The solutions in the manual only work with direct cable connection not with wireless links
I have experimented with all forms of the Mikrotik bonding and on every occasion turning off the the wifi does not cause it to failover, only disconnecting the LAN port causes it, I set up a test rig using 2 x Mikrotik switches and 4 x SXT radios in PTP bridge mode as to wire replacement links, turning off the ireless on one of the links did not failoverThat is actually wrong...The solutions in the manual only work with direct cable connection not with wireless links
It all depends on the Bonding Mode used...
The bonding does not know if it has an Antenna connected to it or a wire, so it is up to you to perform a correct implementation of the bonding...
As posted earlier, you can use ARP Monitoring... And i am always talking about Mikrotik, i do not know what Ubiquiti does or doesnt...
LACP 802.3ad does not care if it is ethernet cable, optical, antenna or whatever, because it has other techniques to check for the operation of the link...
If you choose lets say balance-rr you can use ARP monitor and you will still be fine because you manually monitor the link...
Also, 802.3ad MUST be supported by both devices to work...
That's really weird, my yesterday's tests show otherwise. Physical link up, piniging, sniffer shows which slave carries the pings; remove that link's interface from the list of slaves at the remote end, it stops sending LACP frames, the pings start being sent via the other link in a few seconds. With mode=802.3ad, link-monitoring=none. I was running the test on 6.45.7, what about you? But I admit that removal of slave interface from a bond may cause LACP restart, so I'll insert a dumb switch instead and report back.I have experimented with all forms of the Mikrotik bonding and on every occasion turning off the the wifi does not cause it to failover, only disconnecting the LAN port causes it, I set up a test rig using 2 x Mikrotik switches and 4 x SXT radios in PTP bridge mode as to wire replacement links, turning off the ireless on one of the links did not failover
So you had to climb the mast after all???I tried to setup a LAG on the Ubiquiti S16 and it locked up.
You mean disable RSTP on the Bridge ?on the radios, protocol-mode on the bridge between the Ethernet port and the wireless one must be set to none
Ναί, but that's secondary in this case. The primary reason is that protocol-mode=none not only switches off any STP flavors on the bridge, but also makes it break the 802.1D requirement that frames with destination MAC address prefix 01:80:c2 must never be forwarded by a switch/bridge, as they are reserved for communication with adjacent elements. And LACP is one of these protocols.You mean disable RSTP on the Bridge ?
Where are you from? if i may ask...Ναί
Take the road to Berlin and stop in the previous country on the way.Where are you from? if i may ask...
I haven't said that the protocol-mode=none must be set on the bridge whose member port the bond interface is; this requirement is relevant for the bridges on the active network path between the two devices running LACP. Each of the two LACP-enabled switches (bridges) is connected to an Ethernet interface of one of the LHG60, and inside each LHG60, the Ethernet interface and the wireless interface must be bridged together for this application case (L2 transparent channel between the Ethernets over the air). And to make them transparent also for the LACP frames, both these bridges inside the LHG60 must be configured with protocol-mode=none. Details in the manual: "Since RouterOS v6.43 it is possible to forward Reserved MAC addresses that are in 01:80:C2:XX:XX:XX range, this can be done by setting the protocol-mode to none."As far as i ve tested LACP works just fine with RSTP enabled on the Bridge where the bonding exists... Nor i can find any reference that RSTP must be disabled...
Ok that makes it clear to me now...this requirement is relevant for the bridges on the active network path between the two devices running LACP
A narrow miss, keep a bit more to the west while drivingPoland i guess...
Yes, LLDP is the same case like STP BPDUs and LACP frames, all of these should normally not get past the adjacent device.I made a quick capture with Wireshark under GNS3 and i could see the reserved MAC address range in the LLDP packets...
You can already find and purchase this product...Mikrotik are supposed to be bringing out a Netpower 16P
Not yet seen the 16P for saleYou can already find and purchase this product...Mikrotik are supposed to be bringing out a Netpower 16P