I’m trying to get my server cluster up and running. I have a network showed bellow:
My initial goal is to create vlans on the servers and allow communication across servers on their respective VLANs without manually setting new VLANs on the router. My only way of allowing communication was to manually add vlan id to bridge vlan.
You can make it behave as you wish. Search for “filtering” on this documentation page. There’s more than just the option you’ve shown in your /export output. I believe you want to set ingress-filter, if I understand your purposes.
Something else to think about: does the CRS do the inter-VLAN routing, or does that get promoted up to the RB4011 and go back down through the switch? The latter is much better suited as a router, being 3.5× faster on a pure clock rate times core count basis, but that means choking your bonds (presumably dual-redundant 10G?) down to a single 10G link. Yet if you make the CRS do it, can you do so without needing so much filtering that you overrun the CRS’s CPU?
The thing is that with setting vlan-filtering=yes on bridge, ROS enforces certain level of security. One notable setting is subtree /interface bridge vlan which defines egress filtering.
If you want to make CRS transparent to VLANs (and agree to move VLAN security to connected devices), then set vlan-filtering=no in which case CRS will act as dumb switch not processing 802.1Q headers at all.
Setting ingress filtering on individual bridge ports basically is = to stating if the vlan is not defined on this port then discard it from this port
Settng ingress filtering on the bridge itself = to stating if the vlan is not defined anywhere on the bridge then discard it from any port
According to the manual it is used to limit the allowed VLANs that can access the CPU port in specific..
The ingress-filtering can be used on the CPU port (bridge) as well, this can be used to prevent some possible attack vectors and limit the allowed VLANs that can access the CPU.
I’ve not used ingress filtering that much, so i’m not sure if it blocks any VLAN that is not defined in any port in general or it just protects the Management VLAN and access to CPU only…
Have you tested @anav that setting ingress filtering to the Bridge it self will drop any VLAN not defined from any port ?
Turning off vlan-filtering on the bridge drops the connection from PVE nodes.
This is my config as of today:
# jul/15/2021 13:17:56 by RouterOS 6.48.2
#
# model = CRS317-1G-16S+
/interface bridge
add name=BR
add name=DATA protocol-mode=none vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp-sfpplus16 ] l2mtu=1592 name=TRUNK
set [ find default-name=ether1 ] l2mtu=1592
set [ find default-name=sfp-sfpplus1 ] l2mtu=1592
set [ find default-name=sfp-sfpplus2 ] l2mtu=1592
set [ find default-name=sfp-sfpplus3 ] l2mtu=1592
set [ find default-name=sfp-sfpplus4 ] l2mtu=1592
set [ find default-name=sfp-sfpplus5 ] l2mtu=1592
set [ find default-name=sfp-sfpplus6 ] l2mtu=1592
set [ find default-name=sfp-sfpplus7 ] l2mtu=1592
set [ find default-name=sfp-sfpplus8 ] l2mtu=1592
set [ find default-name=sfp-sfpplus9 ] l2mtu=1592
set [ find default-name=sfp-sfpplus10 ] l2mtu=1592
set [ find default-name=sfp-sfpplus11 ] l2mtu=1592
set [ find default-name=sfp-sfpplus12 ] l2mtu=1592
set [ find default-name=sfp-sfpplus13 ] l2mtu=1592
set [ find default-name=sfp-sfpplus14 ] l2mtu=1592
set [ find default-name=sfp-sfpplus15 ] l2mtu=1592
/interface vlan
add interface=BR name=MANAGEMENT_VLAN vlan-id=10
/interface bonding
add mode=802.3ad name=PVE01 slaves=sfp-sfpplus1,sfp-sfpplus2
add mode=802.3ad name=PVE02 slaves=sfp-sfpplus3,sfp-sfpplus4
add mode=802.3ad name=PVE03 slaves=sfp-sfpplus5,sfp-sfpplus6
add mode=802.3ad name=PVE04 slaves=sfp-sfpplus7,sfp-sfpplus8
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/user group
set full policy=local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,dude,tikapp
/interface bridge port
add bridge=BR hw=no interface=ether1
add bridge=DATA interface=sfp-sfpplus9
add bridge=DATA interface=sfp-sfpplus10
add bridge=DATA interface=TRUNK
add bridge=DATA interface=sfp-sfpplus15 pvid=10
add bridge=DATA interface=PVE01
add bridge=DATA interface=PVE02
add bridge=DATA interface=PVE03
add bridge=DATA interface=PVE04
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/interface bridge vlan
add bridge=DATA tagged=TRUNK untagged=sfp-sfpplus15 vlan-ids=10
add bridge=DATA tagged=TRUNK,PVE01,PVE02,PVE03,PVE04 vlan-ids=2912
add bridge=DATA tagged=PVE01,PVE02,PVE03,PVE04 vlan-ids=2913
add bridge=DATA tagged=TRUNK,PVE01,PVE02,PVE03,PVE04 vlan-ids=2730
add bridge=DATA tagged=TRUNK,PVE01,PVE02,PVE03,PVE04 vlan-ids=2830
add bridge=DATA tagged=PVE01,PVE02,PVE03,PVE04 vlan-ids=2914
/ip address
add address=172.29.10.4/24 interface=MANAGEMENT_VLAN network=172.29.10.0
/ip cloud
set update-time=no
/ip dns
set servers=1.1.1.1
/ip route
add check-gateway=ping distance=1 gateway=172.29.10.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh address=172.29.10.0/24
set api disabled=yes
set winbox address=172.29.10.0/24
set api-ssl disabled=yes
/ip ssh
set strong-crypto=yes
/system clock
set time-zone-name=Europe/Zagreb
/system identity
set name=RB02
/system routerboard settings
set boot-os=router-os
/tool bandwidth-server
set authenticate=no enabled=no
/tool mac-server
set allowed-interface-list=none
/tool mac-server mac-winbox
set allowed-interface-list=none
/tool mac-server ping
set enabled=no
@
On LAG ports on each node there will be the following traffic: CEPH Cluster network, CEPH Public network, Proxmox migration network, VM networks behind VyOS router.
CEPH cluster network and Proxmox migration network should be limited to only the switch chip on the CRS since they don’t need to go out of their respective networks.
CEPH public network should be limited to only be accessed by the VM networks trough VyOS router.
VM networks that are behind VyOS will be accessed by the clients CSS326-24G-2S+ (see picture bellow) that will be connected to the CRS.
There is only one client that could take advantage of 10G that would have to go up to RB4011. All other clients are 1G conencted to CSS326-24G-2S+.
