Community discussions

MikroTik App
 
UPyqbvQZ
just joined
Topic Author
Posts: 3
Joined: Wed May 05, 2021 12:57 am

LAGG with pfsense Setup

Wed May 05, 2021 1:08 am

I have a bit of a problem and before I purchase anything, I was hoping somebody can let me know if this will work or not...

I have a cable modem with a 2.5G port on it. From what I've seen, the 10G ethernet SFP's support this speed. I then have a pfsense box with a couple of gigabit nic's in it. What I want to do is get the full speed of that 2.5G port (well, really 1.5G since that's what I'm provisioned from the provider) over to my pfsense box.

To do this, I'm thinking of putting a CRS305-1G-4S+IN between the cable modem and my pfsense box. Get 3 S+RJ10 SFP's, then create a LAGG group for two of those connections and use the third to the cable modem.

I did a bit of research and I'm presuming the command to do this is "interface bonding add name=P01 mode=802.3ad transmit-hash-policy=layer2-and-3 slaves=ether1,ether2". Would doing this essentially bond ether1 and ether2 together to form a 2gb link from my pfsense box to the switch? Is this the right way to go about doing this or am I totally off?

Thank you.
 
mkx
Forum Guru
Forum Guru
Posts: 6021
Joined: Thu Mar 03, 2016 10:23 pm

Re: LAGG with pfsense Setup

Wed May 05, 2021 7:27 pm

Something in that line.

There's just a gotcha with LAG in general (and MT can't be any different): all packets belonging to single connection will pass same bond member, hence single connection throughput is limited to speed of bond member (in your case 1Gbps). Same may apply to muktiple connections depending on transmit hash policy. When l2-l3 policy is used, this means ethernet MAC address (internet connection will bear MAC address of your router) and IP address (internet server). When running speedtest with paralkel streams, they all share same l2-l3 combination and will thzs share same physical link of LAG. There's non-standard l3-l4 transmit hash policy which (statistically) solves the problem because it uses L4 info as well (with TCP or UDP that's port number and those will be different for parallel connections) ... but it's not standard and your combo (switch-pfsense) might support it or not.

Just wanted to mention it for you to adjust expectations ....
BR,
Metod
 
UPyqbvQZ
just joined
Topic Author
Posts: 3
Joined: Wed May 05, 2021 12:57 am

Re: LAGG with pfsense Setup

Wed May 05, 2021 10:16 pm

Something in that line.

There's just a gotcha with LAG in general (and MT can't be any different): all packets belonging to single connection will pass same bond member, hence single connection throughput is limited to speed of bond member (in your case 1Gbps). Same may apply to muktiple connections depending on transmit hash policy. When l2-l3 policy is used, this means ethernet MAC address (internet connection will bear MAC address of your router) and IP address (internet server). When running speedtest with paralkel streams, they all share same l2-l3 combination and will thzs share same physical link of LAG. There's non-standard l3-l4 transmit hash policy which (statistically) solves the problem because it uses L4 info as well (with TCP or UDP that's port number and those will be different for parallel connections) ... but it's not standard and your combo (switch-pfsense) might support it or not.

Just wanted to mention it for you to adjust expectations ....
Thanks for the reply. I don't believe pfsense can do the l3-l4 hash policy.

If I'm reading this right.... My pfsense box has 2 NIC's that will have the same mac address. The cable modem NIC has the same mac address. With the l2-l3 policy, I'm always going to see a top speed of a gig because the mac addresses are staying the same? On a speedtest page, if I enable "multi-threaded" for the speedtest, it will still be a gig?

I'm thinking it because I do nat on my pfsense box. I have an internal lan, goes into my pfsense box, and then out to the cable modem. Where the switch would be is between my pfsense box and the cable modem. On the switch, it's going to see the same mac addresses (2 from my pfsense box, 1 from the cable modem), so it's going to just use that gigabit speed because that's what lacp does. Or am I totally off on my thinking because even though it's the same mac address, it's different destination IP's so it will be unique...?
 
mkx
Forum Guru
Forum Guru
Posts: 6021
Joined: Thu Mar 03, 2016 10:23 pm

Re: LAGG with pfsense Setup

Wed May 05, 2021 11:02 pm

Switch between pfsense and cable modem will always see only 2 MAC addresses (1 of cable modem and very probably only 1 of pfsense - linux bonding always uses MAC address of first active bond member as bond MAC - for all bond members, I'm not sure about other implementations but they are probably the same with this regard). It will see the same IP address on pfsense end (the public IP address) but generally different IP addresses on cable modem end (IP addresses of internet servers). Thus traffic will be spread over all bond members if there will be mix of connections towards different internet hosts. With multi-connection test towards single speedtest server cable modem end will see same IP address for all connections hence all will use same bond member and will thus be limited to 1Gbps (sum of all connections).

Remember that gateway's IP address (cable modom in this case) only matters to find out MAC address of gateway's interface[*]. The real payload frames don't contain gateway's IP address at all.

[*]gateway's IP address is needed for management as well (in that case payload packets will contain gateway's IP address), but that's out of scope of this discussion.

I'm thinking: if the whole purpose of yet-to-be-purchased switch is to enable higher wan speed of pfsense, wouldn't be replacing pfsense's NIC with a 10Gbps part actually cheaper? And would allow full speed for connections targeting same internet server.
BR,
Metod
 
UPyqbvQZ
just joined
Topic Author
Posts: 3
Joined: Wed May 05, 2021 12:57 am

Re: LAGG with pfsense Setup

Thu May 06, 2021 4:58 pm

I'm thinking: if the whole purpose of yet-to-be-purchased switch is to enable higher wan speed of pfsense, wouldn't be replacing pfsense's NIC with a 10Gbps part actually cheaper? And would allow full speed for connections targeting same internet server.
You are 100% correct. Unfortunately, I've hit a few problems... My pfsense box is a protectli, which is a small form factor PC without any expansion. It has usb3 but that's really about it. It has six gigabit ports on it and that's really about it. When I bought it I thought I would max it out at 1 gig, but I was wrong.

I was thinking about swapping the hardware, but even if I do, driver support under freebsd (pfsense) is sketchy. It can support 10g, but the cable modem only has gigabit ports and one 2.5G port on it. There are a few 2.5G NIC's out there, and some SFP+'s that support 2.5G, but the drivers for this look like they are just being introduced and just really buggy.

Ubiquiti has their product that can support 2.5G, but one of my requirements is an OpenVPN client and server running on that box, which isn't supported on the Ubiquiti side. I guess if I really wanted to, I could replace my pfsense with Ubiquiti, then get a seperate box to run OpenVPN, but that is a lot of work and more power. So that's why I'm trying to think of different ways to do this...which, in the end, I'm not sure if it's really going to work.
 
mkx
Forum Guru
Forum Guru
Posts: 6021
Joined: Thu Mar 03, 2016 10:23 pm

Re: LAGG with pfsense Setup

Fri May 07, 2021 1:23 pm

Well, the setup you outlined in your original post will work ... but as I described, certain connections will be capped at 1Gbps. If there are many connections, their cumulative throughput will likely hit the cap your ISP is (or will be) provisioning to you.
BR,
Metod

Who is online

Users browsing this forum: No registered users and 4 guests