Hey, kinda newbie here, but it seems that I can’t find the answer by my self.
It works, but when I do some iperf3 tests between different VLANs all the traffic seems to go through the L009UiGS’s CPU and sometimes the CRS326-24G-2S+'s CPU.
I got a L009UiGS, it’s in the DMZ of my ISP router via ether1.
And I configured a bond between it and a CRS326-24G-2S+ with 4 ports.
Proxmox Host on SFP+ with 10G NIC.
L3 hw offload on the CRS326-24G-2S+ enabled.
Just tell me what configs you need to see please.
Iperf3 between random Linux host on random VLAN and Proxmox host:
300~400Mb/s
L009UiGS’s CPU 80+%
Iperf3 between random Proxmox VM on random VLAN and Proxmox host:
300Mb/s
L009UiGS’s CPU 70+%
Iperf3 between random Proxmox VM on Proxmox host’s VLAN:
20Gb/s
This is my first post, should I upload my entire config file so you can look a it?
I’m sorry in advance if I didn’t explain something well, just, braining isn’t braining anymore rn
Some basics: L2 offload works between different ports within same VLAN. Router is needed to pass between different VLANs. Only a few devices can do L3 (routing) HW ofgload and it’ll work if that device is set up as router and other devices use it as their gateway.
So it won’t work by simply dropping CRS into a network … it has to be configured as default gateway for all devices in all VLANs.
BTW traffic during the last test likely doesn’t even reach the cable which connects PVE host with the rest of network.
Yup, totally suspected that about my latest test.
I’m really grateful for the response.
How do you think I should approach it?
Should I configure something like OSPF (hadn’t look at it enough yet) to make the CRS know the routes?
I tried adding some static routes on the CRS but seems that I don’t know what I’m doing or it isn’t working.
Or should I create a specific bridge for the VLANs that need to exchange traffic between them on the CRS.
You’ll have to add IP address to every VLAN where you want CRS to route between. And set devices in those VLANs to use CRS’s address as default gateway. In simple flat network (all VLANs passing through same CRS) you don’t need any of routing protocols, no need to distribute routing information … other than to your edge router (L009). If your IP network topology won’t be too dynamic, you can even live with static settings on L009, simply set static routes toward all the LAN IP subnets via CRS. If your VLAN subnets use adjacent address spaces, you can even use single route with shorter subnet mask (to cover all VLAN subnets) on L009.
Your L009 will only know about CRS (it won’t be member of any of VLANs where CRS will do the routing), they will interconnect using dedicated subnet and will likely be the only two devices in that subnet. Any other device in same subnet would suffer from routing triangle at some point in time and that has potential of upseting connection tracking machinery in L009 (which is necessary for firewall functions).
So what you’re up to is definitely not “kindergarten rated” and you have to understand IP routing to certain degree to get things working correctly and at right speed.
Alr, I think I understand it, I’ll try to set it up this weekend and I’ll post updates.
Really appreciate your time man, fast and clear instructions .
One last question, if I setup all like that, does the firewall rules on the L009 still apply between the CRS VLANs?
No, inter-VLAN traffic will bypass L009. If you want to control inter-VLAN traffic, you have to do it on CRS .. either routing rules (these are pretty coarse, but consume way less resources) or using firewall (and you’ll want to establish fasttrack offloading, not sure if your CRS supports that though). Beware that devices support a limited number of routes/connections offloaded and you don’t want to end in a state where limits get exceeded.
But you do have an option to route certain VLANs via L0009 .. for those VLANs CRS will only act as a switch (without corresponding vlan interface and IP address) and L009 will be member of those VLANs. Routing table on L009 will be slightly more populated and that’s all. But be prepared to fare with L009’s slow routing/firewalling speed, it’s nowhere near wirespeed.
Alright I understand it, thanks.
Should I use the L009 to forward my services to the outside?
I got ~800Mb/s with my ISP, but when I did some speedtests it’s CPU was pretty high (also the CRS) and I got around ~600Mb/s download
Your L009 is still slightly faster than CRS when it comes to CPU-based routing/firewalling (according to official test results around 40%), so it still makes sense to use it as border gateway for your home network (while using CRS as core router). Keep in mind that number of L3HW offloaded connections is limited and your “slow internet connections” might eat into limits, thus forcing local (inter-VLAN) connections to pass CPU. Which would kill your inter-VLAN performance on the spot.
Also, depending on the exact style of internet access, offered by ISP, it might not be offloaded at all (e.g. PPPoE).
Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAd 0.0.0.0/0 192.168.18.1 1
DAc 192.168.18.0/24 ether1 0
DAc 192.168.88.0/24 bridge 0
DAc 10.1.0.0/30 bridge 0
As I told you, I’m a newbie in all of this, but if I ping lets say 1.1.1.1 from the CRS, It won’t go out without the 0.0.0.0/0 route.
But why the f… I got my laptop connected via ethernet to the CRS and I get the DHCP lease from my L009.
Flags: D - DYNAMIC
Columns: ADDRESS, MAC-ADDRESS, HOST-NAME, SERVER, STATUS, LAST-SEEN
# ADDRESS MAC-ADDRESS HOST-NAME SERVER STATUS LAST-SEEN
1 D 192.168.88.225 something L-Laptop dhcp1 bound 13m24s
My Proxmox sits in the vlan20, inter-VLAN traffic seems to work, but it doesn’t go out.
[admin@MikroTik] > ip/address/print
Flags: D - DYNAMIC; S - SLAVE
Columns: ADDRESS, NETWORK, INTERFACE
# ADDRESS NETWORK INTERFACE
0 192.168.88.1/24 192.168.88.0 bridge
1 D 192.168.18.254/24 192.168.18.0 ether1
2 S 10.1.0.1/30 10.1.0.0 bonding1
[admin@MikroTik] > ip/address/print
Flags: S - SLAVE
Columns: ADDRESS, NETWORK, INTERFACE
# ADDRESS NETWORK INTERFACE
;;; Home
0 10.1.2.1/26 10.1.2.0 vlan2
;;; IoT
1 10.1.3.1/24 10.1.3.0 vlan3
;;; Proxmox
2 10.1.20.1/28 10.1.20.0 vlan20
;;; Home Multimedia
3 10.1.4.1/26 10.1.4.0 vlan4
;;; Home Servers
4 10.1.5.1/26 10.1.5.0 vlan5
;;; Home Lab
5 10.1.10.1/24 10.1.10.0 vlan10
;;; Testing
6 10.1.11.1/24 10.1.11.0 vlan11
;;; Home Lab Servers
7 10.1.21.1/26 10.1.21.0 vlan21
;;; Home Public Servers
8 10.1.22.1/28 10.1.22.0 vlan22
;;; Home Game Servers
9 10.1.23.1/28 10.1.23.0 vlan23
10 S 10.1.0.2/30 10.1.0.0 bonding1
What is setup of port to which laptop is connected? Access or trunk? If access, which VLAN? Where is DHCP server which serves that VLAN. Or do you have DHCP relay on CRS?
Any reason for two addresses on bridge on L009? I’d remove pirt, connecting CRS, from bridge and set address directly. Or run connection as tagged and have that address on a particular vlan interface.
The laptop was untagged on ether5 with PVID 1, now I just moved it to vlan10.
The L009 was the only DHCP server but my brain don’t comprehend why all the traffic from my laptop was “like” bypassing the CRS, until I didn’t add the static routes to the L009 it couldn’t ping let’s say 10.1.20.2, was like if I was directly connected to the L009.
I’ll post you the config, I think you’ll see it better than with my words.
Also when I do some speedtests the L009’s CPU goes pretty high, but I think there’s nothing much to do there(?).
L009:
[admin@MikroTik] > ip/address/print
Flags: D - DYNAMIC; S - SLAVE
Columns: ADDRESS, NETWORK, INTERFACE
# ADDRESS NETWORK INTERFACE
0 192.168.88.1/24 192.168.88.0 bridge
1 S 10.1.0.1/30 10.1.0.0 bonding1
2 D 192.168.18.254/24 192.168.18.0 ether1
I think I did what you told me to do, but I’m not sure.
Bridge it’s default config, the ether1 it’s getting DHCP lease from my ISP router and the bonding1 goes to the CRS (CRS got the 10.1.0.2/30).
[admin@MikroTik] > ip/route/print
Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, s - STATIC, d - DHCP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
# DST-ADDRESS GATEWAY DISTANCE
DAd 0.0.0.0/0 192.168.18.1 1
0 As 10.1.10.0/24 10.1.0.2 1
DAc 192.168.18.0/24 ether1 0
DAc 192.168.88.0/24 bridge 0
1 As 10.1.21.0/26 10.1.0.2 1
2 As 10.1.20.0/28 10.1.0.2 1
DAc 10.1.0.0/30 bridge 0
(I didn’t add all static routes to the CRS VLANs yet)
Post output of /export command (redact sensitive information, such as serial number) … print’s show running config but not how it ended up being like that.
Re. L009 CPU load: L009 has moderate routing capacity (for today’s standards) of something between 300Mbps and 2Gbps depending on the actual config … with anything in firewall it’ll be in the lower part of that range. So yes, when L009 has to route traffic, its CPU will likely be the bottleneck. And the point of setting CRS to route between VLANs is to bypass slow L009 with (fast due to L3HW offload) CRS to do the inter-VLAN routing.
Well, you’ll have to. You can’t expect the “jolly new roundabout” fully functional if you’re letting traffic reach it via some old goat path. And even if traffic does flow somehow (partly via new roundabout, partly old goat path), you can’t assess how well the new path works. So until you finish rebuilding your paths/routing, your only concern should be to retain management access to your devices.
The biggest issue: bridge1 CPU-facing port has to be tagged member of all VLANs where switch is supposed to route between.
Same goes for bond1 (I’m guessing thete will be some tagged VLANs passing between CRS and L009).
Thanks for sharing that tutorial, finally got it working.
I think another issue it’s the 4 port bond, seems like it’s struggling, since I only need less than 1G I’ll just set 2 ports as active backup and I’ll update if it works better or not.
.