Community discussions

MikroTik App
 
fred974
newbie
Topic Author
Posts: 37
Joined: Fri Jul 15, 2016 3:39 pm

Pfsense - Mikrotik switch High Availability setup

Wed Jan 04, 2023 2:43 pm

Hi,

I have the feeling that my setup isn’t right…
At the head of the network, we have 2x pfsense firewall setup in HA . The firewalls have 2x sfp+ ports and 3 1G RJ45 ports

WAN – sfpplus1
LAN – sfpplus2
LAN2 – RJ45-1
pfSync – RJ45-2
Unused – RJ45-3

The LAN from firewall1 is connected to mk_switch1 and the LAN from firewall 2 is connected to mk_switch2.

Image
The hypervisor, storage and backup servers has 2x sfp+ interfaces and are physically connected to both Mikrotik switches (1 port per switch). At the software level, we setup the 2 ports as a bonded interface using the LACP protocol and we setup the mikrotik port interfaces across both switches using MLAG.

I cannot help but think that something is wrong but I cannot figure out what yet… :thinking:
Do I need to swap the wan/lan interface so i have 2x sfp+ in lag in firewall and then every firewall has 1 lan port in each swith?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Pfsense - Mikrotik switch High Availability setup

Wed Jan 04, 2023 3:46 pm

Having configured MLAG on hypervisor is fine. However, the problem is that it is expected that any destination (beyond MLAG peers) is accessible via any of MLAG links. In your case that's not the case, primary opensense is only accessible via left MT switch but not via right MT switch. I'm assuming that even though opensense are in some sort of HA configuration, single connection still needs to go through same opensense server entirely. MLAG doesn't ensure that by itself.
So either you need connection between MT switches (to complement the MLAG configuration) or you need also opensense both connected to both MT switches (in MLAG configuration).
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Posts: 1492
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: Pfsense - Mikrotik switch High Availability setup

Wed Jan 04, 2023 4:08 pm

Somewhat related to your topic "Pfsense - Mikrotik switch High Availability setup"

Never use a Mikrotik CHR in a VmWare ESXi Cluster High Availability setup.

The Mikrotik license on a CHR will self destruct when there is a failure in the Cluster which results in a Mikrotik CHR router getting auto relocated to a different physical VmWare ESXi Hypervisor.

PfSense virtual routers work well in a H-A Cluster , Mikrotik CHR license self-destructs in a H-A Cluster when the Cluster moves anything.

Why Mikrotik chooses to have a self-destruct in their CHR license is way beyond stupid.

North Idaho Tom Jones
 
fred974
newbie
Topic Author
Posts: 37
Joined: Fri Jul 15, 2016 3:39 pm

Re: Pfsense - Mikrotik switch High Availability setup

Wed Jan 04, 2023 6:03 pm

Hi,

Thank you very much for taking the time to replying to my post :) I also forgot to mention that my switches are 2x Mikrotik CRS317-1G-16S+RM

The Mikrotik license on a CHR will self destruct when there is a failure in the Cluster which results in a Mikrotik CHR router getting auto relocated to a different physical VmWare ESXi Hypervisor.
Can you please tell me what CHR mean?

So either you need connection between MT switches (to complement the MLAG configuration) or you need also opensense both connected to both MT switches (in MLAG configuration).
The OPNsense firewall had 5 ports in total:
ax0 - SFP+ (10GE)
ax1 - SFP+ (10GE)
igb0 - 1000BASE-T (1GE)
igb1 - 1000BASE-T (1GE)
igb2 - 1000BASE-T (1GE)
Will it work if I then swap the interface around like this:
WAN - igb0
pfSync - igb2
LAN - ax0 + ax1 in lagg bond

Then I can connect ax0 to left MT switch and ax1 to right MT switch?
Or
Option2 is to connect the 2 switches via a dac cable MT1 sfp16 to MT2 sfp16

Which is the better option? At the moment I don't need or use a 10G WAN I set it up this way for furtur proofing.

Thank you
 
fred974
newbie
Topic Author
Posts: 37
Joined: Fri Jul 15, 2016 3:39 pm

Re: Pfsense - Mikrotik switch High Availability setup

Wed Jan 04, 2023 6:06 pm

Never use a Mikrotik CHR in a VmWare ESXi Cluster High Availability setup.
I just figure out that it stand for Cloud Hosted Router.. the Mikrotik are not virtualised but real physical hardware
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Pfsense - Mikrotik switch High Availability setup

Wed Jan 04, 2023 7:55 pm

Option2 is to connect the 2 switches via a dac cable MT1 sfp16 to MT2 sfp16
Since your opensense boxes are in redundant configuration, you could go with option #2 ... because if you go with option #1, you'll bump into same problem if you ever connect another box to only one of MT switches.
 
fred974
newbie
Topic Author
Posts: 37
Joined: Fri Jul 15, 2016 3:39 pm

Re: Pfsense - Mikrotik switch High Availability setup

Fri Jan 06, 2023 7:07 pm

Option2 is to connect the 2 switches via a dac cable MT1 sfp16 to MT2 sfp16
Since your opensense boxes are in redundant configuration, you could go with option #2 ... because if you go with option #1, you'll bump into same problem if you ever connect another box to only one of MT switches.
Thank you very much.
If I go with option2, do I need to setup Spanning Tree Protocol (STP) between the 2 switches?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Pfsense - Mikrotik switch High Availability setup

Fri Jan 06, 2023 8:07 pm

I don't have first-hand experience with MLAG. As far as I understand, the link between switches is specifically configured as MLAG peer link. Have a look at help page.

I'm not sure MLAG is supported in SwOS.
 
fred974
newbie
Topic Author
Posts: 37
Joined: Fri Jul 15, 2016 3:39 pm

Re: Pfsense - Mikrotik switch High Availability setup

Mon Jan 09, 2023 2:10 pm

I just had a call with the datacentre and the setup fee to reconfigure the system from 10G uplink to 1G uplink is fairly significant as they said, I requires a hardware and configuration modification....

Right now the servers are being told they can go to either switch to reach the default gateway, but the active firewall is only present on one of them, requiring traffic that hits the wrong switch to traverse an extra hop.

If I don't change the existing wirering for the WAN and now that I have both switches directly connected via sfp+16, how much of a performance impact do I have when the traffic that hits the wrong switch has when it has to traverse an extra hop?

I am aware that is is probably not best practice but is it that bad to leave it as is?

Thank you
 
fred974
newbie
Topic Author
Posts: 37
Joined: Fri Jul 15, 2016 3:39 pm

Re: Pfsense - Mikrotik switch High Availability setup

Mon Jan 09, 2023 2:38 pm

I don't have first-hand experience with MLAG. As far as I understand, the link between switches is specifically configured as MLAG peer link. Have a look at help page.

I'm not sure MLAG is supported in SwOS.
No it isn't, I am doing this on RouterOS, but the hardware are switches... So do you thing the MLAG only cares about the 'peer ports'?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Pfsense - Mikrotik switch High Availability setup

Mon Jan 09, 2023 5:38 pm

I guess that performance hit when packet goes via "wrong switch" is slightly increased delay ... every ethernet switch adds it because most work in "store and forward" mode. Exact delay depends on port speed, for 1Gbps port it's rougly 15us (for standard MTU size). This can be increased if the switch interconnect becomes congested ... which makes the idea of connecting most (high traffic) LAN hosts to both switches sensible (because that way traffic between different hosts doesn't have to pass switch interconnect). The switch interconnect will be used only in case one host only connects to one switch (either it's by design or if one link of MLAG bond fails).

As far as I understand, MLAG is more or less the same as "normal" bonds (e.g. 802.3 ad), but bond peer (on one or both sides) is not single logical entity (connecting multiple links to a stack of switches is still bonding). However, devices on bond link end need to cooperate somehow to ensure proper frame delivery. In case of stacked switches that's done via stack interconnect, in MLAG case that's over MLAG peer ports. When a "simple device" (e.g. a server with two NICs) is connected to two switches in MLAG config, it sees it as simple LACP bond, the MLAG details are hidden from it. Similar for switch that connects to two MLAG-configured switches (e.g. virtual switch of hypervisor).
 
fred974
newbie
Topic Author
Posts: 37
Joined: Fri Jul 15, 2016 3:39 pm

Re: Pfsense - Mikrotik switch High Availability setup

Mon Jan 09, 2023 7:09 pm

MKK, thank you for your reply. So do you think it is worth me downgrading the 10G uplink for 1G to create the interconnect stack connected the 2 switches to each firewall?
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11381
Joined: Thu Mar 03, 2016 10:23 pm

Re: Pfsense - Mikrotik switch High Availability setup

Mon Jan 09, 2023 9:34 pm

I'm not sure where this "reconfigure from 10G to 1G to establish MLAG peer link" comes from, I missed exact model of Mikrotik switch and otger details which might explain this mistery to me.
If those switches lack free 10G ports, then you should think about how to solve the problem. Potentially the MLAG peer link xan get used quite heavily if there are many "high speed hosts" talking between each other via "wrong" switch. On the other extreme that link might be utilized really low if most hosts talk to each other via correct switch. So try to get an educated guess about the amount of traffic which will use it. You might get away with 1Gbps link.

If switches have a free 10Gbps port, then use them for MLAG peer link and you're done ... no need to change pfsense connections IMO since they are in HA config anyway (a bit reduced because you loose internet if fails one of switches and the other pfsense, if both firewalls had MLAG connections, then internet would keep working in this case).

Who is online

Users browsing this forum: pe1chl and 20 guests