now I'm the owner of a CSR326-24G-S+, which is connected on port 24 to my fritzbox router.
I want to have ports 2 - 8 into a bridge called "server". (done)
I also want to have ports 1, 9 - 24 into a bridge called "bridge" (done)
Bridge "bridge" is receiving ip addresses via dhcp through my fritzbox router, which is connected to port 24 on the switch. (checked via ip > dhcp client)
On port 1 (bridge "bridge) of the switch I connected a pfsense server which only has one nic. I configured the NIC (WAN) that it'll get an ip address of my main router (fritzbox) through the bridge / switch. (done)
Now the pfsense has a V-LAN interface with a dhcp server running on it.
Now I want that my pfsense, which is connected on port 1 (bridge), that it will give vlan dhcp to the bridge: "server" (ports 2 - .
I tried to create a vlan interface id 10 in the switch and put "ether1" on it.
vlan10 interface then gets ip addresses from pfsense vlan dhcp server.
From switch I can ping my pfsense on my WAN and VLAN:
But all servers, connected to port 2 - 8 (bridge: "server") I can't ping (timeout)
The servers are well pre-configured, so I just need a way to figure these things on the switch out
It’s hard to know what kind of config your CRS is actually running. But since you mentioned multiple bridges it seems to be wrong … at least from performance point of view (only one bridge can be offloaded to switch chip, the rest are handled by slow CPU).
So start over by using single bridge (can span all ports) and use VLANs to “partition” your CRS.
interface bridge gives switch dhcp client an ip address (192.168.178.25/24)
interface vlan10 gives switch dhcp client an ip address (172.16.0.16/24)
I can ping 172.16.0.1 (pfsense vlan) from my vlan10 interface, but I cannot ping my servers connected on port 3, 4 and 5 (172.16.0.4 - 6).
Servers are pre-configures, there is no issue on the server side.
I’m not going to look into your config as long as you have multiple bridges. You may succeed in making this eagle move, but it’s not going to fly, it’s going to crawl.
Ok. I tried to bring this to work only by creating one bridge.
Looks like I can now ping my servers from my vlan interface and also access them from the vlan.
But I’m still confused, because the “vlan10” interface (172.16.0.0/24) should not be able to ping my main routers ip address (192.168.178.24/1).
192.168.178.0/24 = no vlan
172.16.0.1/24 = vlan id 10
Could you take a look into my new config, please?
What I try to realise is that my pfsense on port 1 is getting an dhcp ip address on port 1 from my main router through the bridge, so it can route everything into internet. (works)
Also pfsnse on port 1 should be able to give his vlan dhcp ip addresses to vlan10 interface (works)
Now every server on port 2 - 8 should be in vlan 10 (seems to work) but all servers can ping my main router?
Probably it’s because ROS looks at packet and if destination address is any of its own, it will treat the packet the same regardless the ingress interface (bridge vs. vlan). It is possible to block use of “non-native” router’s IP address but it’s purely aesthetics, nothing about security.
… but all servers can ping my main router?
Your CRS has IP address in both of your subnets. That’s not necessary if device is used only as switch (and another router is made part of both subnets and can thus route between subnets). It’s a mystery if it’s actually CRS making the accessibility possible, CRS has only DHCP clients running so use of CRS as gateway towards main router surely comes from elsewhere (static settings on servers? DHCP server?). Even if you do intend to use CRS as router between subnets, you have to be aware that CRS can indeed route but without special care it’ll do it using slow CPU.
Well, when I configured pfSenses WAN and LAN Port then I guess it’s the pfsense which routes everything, because of it’s default gateway (192.168.178.1).
So should I switch to SWOS instead of ROS for my purpose?
Since my mainrouter has the internet access I needed pfsense to get an ip address (and route) from the mainrouter. Servers (and VMs on it) on ether3-5 should be like “behind the pfsense”, isolated in the vlan (but they still should have access to internet). I’m pretty new to network technology so I don’t know very much about all of that, I’m still learning and try to figure everything out.
Besides of that what you told, one should never user more than 2 bridges? Now I only configured 1 bridge as I said before.
From performance point of view it doesn’t matter if your CRS runs ROS or SwOS … if you configure things under ROS right. ROS is more versatile … and thus allows user to screw the config more easily.
When mentioning multiple addresses, I was referring to this part of config:
You only need IP address in single subnet, whichever you want to use for management. I’ll assume you’ll want to have it in “native” subnet, which is referred to as “interface=bridge” in the code snippet above. I assume you want to have VLAN 10 only for communication between virtual servers. So you can
remove DHCP server, which is bound to interface=vlan10
As per specifications from your opening post, you’ll want to add ports 9-24 to bridge as access ports without PVID set. This will add them as access ports to VLAN ID 1 (which is default config on ROS). ether1 is already there, so that will create the “1,9-24” part of the switch.
As to connection between pfsense and VLAN 10: the best way would be to convert ether1 (and port on pfsense) to “hybrid” port with “native VLAN” (untagged) and tagged VLAN 10. The untagged would continue to work with ports 9-24 and tagged would give pfsense access to virtual servers’ subnet.
Later on you will want to “harden” the VLAN config by setting frame-types=admit-only-untagged-and-priority-tagged ingress-filtering=yes on access ports (regardless the PVID setting; ether2-ether24) and ingress-filtering=yes on trunk/hybrid ports (ether1).
Now my switch only has one ip from the native subnet (192.168.178.2/24).
The dhcp-server (vlan10) is running on pfsense, not on the switch. I also removed vlan10 interface now on the switch.
Since my english is not that good I made a picture of my scenario for better understanding.
In the future I will replace the fritzbox with the pfsense, as I currently do not have a DSL modem for the pfsense and unfortunately I cannot configure the fritzbox accordingly
This scenario is only for testing and “pre-setup” my future network.
Seems like my “virtual machines” still get an ip address vom pfsense vlan10 interface (dhcp server) (which should be and is good).
If i directly log in into my pfsense on 192.168.178.254 (ether 1 (bridge) - wan interface, native subnet) I can’t ping my virtual servers from both subnets.
(ping -S 192.168.178.254 172.16.0.4 / ping -S 172.16.0.1 172.16.0.4). Looks like vlan is working / vms are isolated.
My “problem” now is that I also can’t ping pve 1 - 3 servers (proxmox host systems) on 172.16.0.4 - 6 (ether3 - ether5),
but on the other side this is also ok for me and should be this way.
My next step would be to want to access this vlan10 vom my native subnet from a specific ip address / port (port 9 - my personal pc in native subnet)
(192.168.178.29 → 172.16.0.1/24). Or setup a management vlan?
Do I need to make this happen by firewall-rules or some configuration on the mikrotik switch?
Even I guess I have to add:
bridge vlan - VLAN IDs 1 - tagged port: ether 9
bridge vlan - VLAN IDs 10 - tagged port: ether 9
And also make static routes between mainrouter ↔ pfsense (+ some firewall-rules, right).
I also wonder if I have to manage 2 devices now for firewall rules (switch AND pfsense). I would love to manage my firewall only from my pfsense.
Would love to getting supported to make this happen!
I’m not sure if you have made conceptual decision: which device should do the routing between your two subnets?
In principle you could have two routers in your network: main router which will route between your “LAN” subnets and edge router which will connect your network towards internet. In small networks it’s customary to have single router doing both tasks and that means that all subnets have to be connected to that router. I’m not sure that your pfSense is configured to connect to both 192.168.178.0/24 and 172.16.0.1/24 .
It is possible to configure CRS as main router. While it has a slow CPU it can offload routing to hardware (so it’s done wirespeed … but number of routes/connections offloaded to hardware is limited.
But if you will go this way, then your network (routing part) has to be redesigned quite substantially.
If you decide to go with single router, then CRS will be only a switch. Which is, BTW, the way it’s configured right now. But you’ll have to forget about setting it up as router/firewall (e.g. /interface bridge settings set use-ip-firewall=yes has potential to kill all switching performance).
And you’ll have to configure connection between CRS and pfSense as trunk port … to give pfSense ability to talk to VLAN 10 … yes, setting ether9 as tagged for VLAN 10 is good. Setting anything as tagged VLAN 1 is a bad idea, many vendors use VLAN 1 as default VLAN ID setting, often that setting is not shown in configuration exports. I recommend you to use a different VLAN ID for “normal LAN” … set ether 9 tagged for that VLAN … and set PVID on other ports to that ID. Also add vlan interface with corresponding VLAN ID to make switch management work over that VLAN.
And keep one port at its current config to be able to do the management until management over the new VLAN is working.
Well, the pfsense should route everything between these 2 subnets.
The main router is only for 192.168.178.1/24 subnet and direct internet access.
I want my pve 1 - 3 and vms on it to be on the vlan10, traffic shold then be routed through pfsense and for internet access then through the main router.
pve x → vm → pfsense → mainrouter
I know this is not an optimal set up and it is just for testing and pre-setup, because in a while I will replace my mainrouter with my pfsense.
Switch should only do switch work and I want pfsense to do the firewall.
The setting /interface bridge settings set use-ip-firewall=yes is disabled for now.
Isn’t pfsenses port 1 already configures as trunk port as on my screenshot above?
I guess I miscomunicated the vlan1 and pvid1 because seems like these are 2 different things.
You have currently complication because you have two routers (fritz and pfsense) in your network. And if I understand you correctly only pfsense knows about PVE subnet. So when machines, which are part of main subnet (192.168.178.0/24) want to communicate with PVE subnet, they will use their default gateway (fritz). And if fritz is not configured to use pfsense (192.168.178.254) as gateway towards PVE subnet (172.16.0.0/24), then packets will get mis-routed.
You can solve this problem by setting pfsense as default gateway on all main LAN devices … but then you’ll have routing triangle for LAN devices (they will send packets internet-bound to pfsense, it’ll pass them to fritz, fritz will send them to internet; return packets will arrive at fritz and it will deliver them directly to LAN devices … which will upset connection tracking machinery on pfsense). And this still won’t give access to internet for PVE subnet. Both problems can be solved by performing SRC-NAT on pfsense (so fritz will only see pfsense as source of all traffic and will direct return traffic to pfsense) … but then all devices in main LAN will see pfsense as source of all traffic originating from PVE subnet.
Etc.
You network is not exactly trivial from routing point of view. It’s not hard either … (but that assessment depends on your routing knowledge).
ether9 as tagged member of VLAN 1 could be correct or wrong, depending on how exactly it’s configured on pfsense side. If pfsense is configured explicitly to work with vlan-tagged frames for VLAN 1, then your latest CRS config is fine (but be careful to set pvid on ether to something other than 10 or 1 and set “frame-types=admit-only-vlan-tagged”). If pfsense is configured to to work with untagged frames for LAN 1, then latest CRS config is not OK.
As I already mentioned: using VLAN 1 as tagged is not wrong by itself. The problem is that many vendors use VLAN 1 as default for untagged (they call it “native”) over the wire … and that inplicit config is not apparent. That’s why I’d use a different VLAN ID for “main LAN” … and have it tagged on connections between infrastructure devices (which includes PVE physical machines … you may end up using many more VLANs and different virtual machines on same PVE hardware members of different VLANs).
You don’t have to configure VLANs on fritz … it’s connected to access port of CRS … which can switch it over to tagged port towards pfsense (and other LAN devices). And if you decide to go with another VID for main LAN, you’ll again have plenty of access ports to that VLAN, just like you have a few access ports to VLAN 10.
The problem with fritz right now is the interaction with pfsense (and PVE subnet). And that’s L3 (IP layer) problem, has nothing to do with VLANs. As long as fritz is acting both as edge router (you call it “main router” and partially core router - it’s gateway to your main LAN), there will be problems.
If you can’t get rid of fritz for technical reasons, you could move your main LAN to a segment off pfsense … and use 192.168.178.0/24 only as connection segment between pfsense and fritz. Of course you’d have to perform NAT (both SRC-NAT and DST-NAT) on pfsense … which you’ll have to after you replace fritz with pfsense … but until you do, it’ll be double NAT (on fritz between public IP address and 192.168.178.254 … and pfsense between 192.168.178.254 and 172.16.0.0/24 … and 192.168.178.254 and whatever IP subnet you might choose for main LAN).
You can start creating the new “main LAN” segment off pfsense/CRS and move devices after you’re satisfied with inter-LAN connectivity. After you move all devices from your current “main LAN” to the new one, you’re basically ready to remove fritz … on pfsense you’ll just have to change its WAN IP address (currently it’s 192.168.178.254) … and that will be mostly it.
.
