Switchports with multi VLANs

ivanobuffa · Fri Jan 07, 2022 8:02 pm

Hi Guys,

I broke my head enough and this is honestly my last resort. I hope someone can really assist on this.

CURRENT CONFIGURATION:

1 x MikroTik router: CRS326-24G-2S+
3 x Supermicro server with 2 NICs for a total of 6 ports in each server (exactly the same hardware and software configuration)
OS running on the MikroTik router: RouterOS 6.49.2
OS running on the Supermicro servers: ESXi 6.7 UP3

CURRENT TOPOLOGY:

TARGET CONFIGURATION:

MGMT VLAN has been successfully configured on the MikroTik router
vMotion VLAN has been successfully configured on the MikroTik router
Servers can vmkping eahother on these two networks
Packets coming from servers are tagged (VLAN 10 from MGMT network and VLAN40 from vMotion network)

However for a correct configuration on the ESXi servers side, I have used 2 NICs for failover (see configuration below for the second ESXi but the other ones are exactly the same):
- Uplink vmnic0 for MGMT traffic (on VLAN 10)
- Uplink vmnic6 for vMotion traffic (on VLAN 40)
The problem is that the failover does not work. If I power off - for instance - ether3 on the router, the NIC for vMotion must take over also for the MGMT traffic on the ESXi 01 but that's not happening
Please find below some screenshots explaining the way I configured my MikroTik router:
It seems like configuring the very same port to accept tagged packets from two (or more) different VLANs doesn't work (at least for me).

Any assistance on this is much appreciated.

Regards.

Ivano Buffa

sindy · Fri Jan 07, 2022 8:40 pm

I understand this is not possible with VMware, but post the text export of the CRS configuration - see the mini-howto in my automatic signature just below.

The basic mode of Ethernet redundancy on VMware doesn't use port-channel or LACP but simply uses one physical NIC systematically for one group of MAC addresses and the other one for another group of MAC addresses, and it only changes this mapping if one of the NICs goes down.

Also, you use the "old" way of configuring VLANs on Mikrotik (one bridge per VLAN), which effectively means the CRS forwards all VLANs but one in software, and it's CPU is by far not sufficient for wirespeed forwarding. This has nothing to do with your issue, though.

What is the ESXi version? I had issues with NIC redundancy with some older ESXi versions.

ivanobuffa · Fri Jan 07, 2022 9:33 pm

Hi Sindy,

Thanks for your reply.

>>I understand this is not possible with VMware, but post the text export of the CRS configuration - see the mini-howto in my automatic signature just below
>>The basic mode of Ethernet redundancy on VMware doesn't use port-channel or LACP but simply uses one physical NIC systematically for one group of MAC addresses and the other one for another group of MAC addresses, and it only changes this mapping if one of the NICs goes down.
I'm not using LACP on these ports and this is what I'm trying to achieve. I have a Dell PowerConnect 5548 and in there (Cisco like) I easily configured many ports accepting tagged packets from multiple VLANs and also sometimes accepting untagged packets with the very same ESXi servers and everything worked (and still works in case of failover). So I believe it should be possible with MikroTik routers as well. Or perhaps you mean that this is not possible with MikroTik router/switches?

Anyhow, please find below my current configuration. There's also a configuration for an iSCSI VLAN as these ESXi servers are also linked to a SAN storage (I have replaced information to important IP addresses with things like XXX.XXX.XXX.XXX, etc...):

# jan/07/2022 03:48:26 by RouterOS 6.49.2
# software id = 410F-1UJM
#
# model = CRS326-24G-2S+
# serial number = 945509308C46
/interface bridge
add comment="Ports: 3 - 6 - MGMT Traffic - VLAN 10" name=bridge-VLAN10 pvid=\
    10 vlan-filtering=yes
add comment="Ports: 7 - 14 - iSCSI Traffic - VLAN 20" name=bridge-VLAN20 \
    pvid=20 vlan-filtering=yes
add comment="Ports: 20 - 22 - vMotion - VLAN 40" name=bridge-VLAN40 pvid=40 \
    vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] comment="Management port connected to ts-sw02\
    .ireland.tm on gi1/0/4 with private static IP address: 10.26.11.19 (bonded\
    \_with ether 2)" speed=100Mbps
set [ find default-name=ether2 ] comment="Management port connected to ts-sw02\
    .ireland.tm on gi2/0/4 with private static IP address: 10.26.11.19 (bonded\
    \_with ether 1)" speed=100Mbps
set [ find default-name=ether3 ] comment=\
    "Connected to MGMT port of prod01.ectr.lab" speed=100Mbps
set [ find default-name=ether4 ] comment=\
    "Connected to MGMT port of prod02.ectr.lab" speed=100Mbps
set [ find default-name=ether5 ] comment=\
    "Connected to MGMT port of prod03.ectr.lab" speed=100Mbps
set [ find default-name=ether6 ] comment=\
    "Connected to MGMT port of ds.ectr.lab" speed=100Mbps
set [ find default-name=ether7 ] comment=\
    "Connected to 1st iSCSI port of prod01.ectr.lab" l2mtu=9092 mtu=9000 \
    speed=100Mbps
set [ find default-name=ether8 ] comment=\
    "Connected to 2nd iSCSI port of prod01.ectr.lab" l2mtu=9092 mtu=9000 \
    speed=100Mbps
set [ find default-name=ether9 ] comment=\
    "Connected to 1st iSCSI port of prod02.ectr.lab" l2mtu=9092 mtu=9000 \
    speed=100Mbps
set [ find default-name=ether10 ] comment=\
    "Connected to 2nd iSCSI port of prod02.ectr.lab" l2mtu=9092 mtu=9000 \
    speed=100Mbps
set [ find default-name=ether11 ] comment=\
    "Connected to 1st iSCSI port of prod03.ectr.lab" l2mtu=9092 mtu=9000 \
    speed=100Mbps
set [ find default-name=ether12 ] comment=\
    "Connected to 2nd iSCSI port of prod03.ectr.lab" l2mtu=9092 mtu=9000 \
    speed=100Mbps
set [ find default-name=ether13 ] comment=\
    "Connected to 1st iSCSI port of ds.ectr.lab" l2mtu=9092 mtu=9000 speed=\
    100Mbps
set [ find default-name=ether14 ] comment=\
    "Connected to 2nd iSCSI port of ds.ectr.lab" l2mtu=9092 mtu=9000 speed=\
    100Mbps
set [ find default-name=ether15 ] disabled=yes speed=100Mbps
set [ find default-name=ether16 ] disabled=yes speed=100Mbps
set [ find default-name=ether17 ] disabled=yes l2mtu=9092 mtu=9000 speed=\
    100Mbps
set [ find default-name=ether18 ] disabled=yes speed=100Mbps
set [ find default-name=ether19 ] disabled=yes speed=100Mbps
set [ find default-name=ether20 ] speed=100Mbps
set [ find default-name=ether21 ] speed=100Mbps
set [ find default-name=ether22 ] speed=100Mbps
set [ find default-name=ether23 ] speed=100Mbps
set [ find default-name=ether24 ] comment="UP-Link port serving VLAN 10 and 30\
    \_connected to Nova Telecom switch on ETH4 with public static IP address: \
    XXX.XXX.XXX.XXX" speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] disabled=yes speed=10Gbps
set [ find default-name=sfp-sfpplus2 ] disabled=yes speed=10Gbps
/interface vlan
add comment="MGMT traffic - VLAN 10" interface=bridge-VLAN10 name=vlan10 \
    vlan-id=10
add comment=" iSCSI traffic - VLAN 20" interface=bridge-VLAN20 name=vlan20 \
    vlan-id=20
add comment="vMotion Traffic - VLAN 40" interface=bridge-VLAN40 name=vlan40 \
    vlan-id=40
/interface bonding
add comment="Link Aggregation for ports connected to iSCSI / NFS datastore ds.\
    ectr.lab" mode=802.3ad mtu=9000 name=LAGG-13-14 slaves=ether13,ether14 \
    transmit-hash-policy=layer-2-and-3
add comment="Management interface (eth1 and eth2 in bonding)" name=\
    MGMT-Bonding slaves=ether1,ether2
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add comment="DHCP Pool range for hosts in VLAN 10 (MGMT network)" name=\
    dhcp_pool_vlan10 ranges=192.168.10.100-192.168.10.200
/ip dhcp-server
add address-pool=dhcp_pool_vlan10 disabled=no interface=bridge-VLAN10 name=\
    dhcp-for-VLAN10
/user group
set full policy="local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,pas\
    sword,web,sniff,sensitive,api,romon,dude,tikapp"
/interface bridge port
add bridge=bridge-VLAN10 interface=ether3 pvid=10
add bridge=bridge-VLAN10 interface=ether4 pvid=10
add bridge=bridge-VLAN10 interface=ether5 pvid=10
add bridge=bridge-VLAN10 interface=ether6 pvid=10
add comment=defconf interface=ether15
add comment=defconf interface=ether16
add comment=defconf interface=ether17
add comment=defconf interface=ether18
add comment=defconf interface=ether19
add bridge=bridge-VLAN40 comment=defconf interface=ether20 pvid=40
add bridge=bridge-VLAN40 comment=defconf interface=ether21 pvid=40
add bridge=bridge-VLAN40 comment=defconf interface=ether22 pvid=40
add bridge=bridge-VLAN10 interface=ether23 pvid=10
add bridge=bridge-VLAN20 interface=ether7 pvid=20
add bridge=bridge-VLAN20 interface=ether8 pvid=20
add bridge=bridge-VLAN20 interface=ether10 pvid=20
add bridge=bridge-VLAN20 interface=ether9 pvid=20
add bridge=bridge-VLAN20 interface=ether11 pvid=20
add bridge=bridge-VLAN20 interface=ether12 pvid=20
add bridge=bridge-VLAN20 interface=LAGG-13-14 pvid=20
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/interface bridge vlan
add bridge=bridge-VLAN20 comment="Switchports where the iSCSI ports of all ESX\
    i servers and the iSCSI ports of the datastore are hooked are tagged as th\
    ey are in the same VLAN but packets are coming tagged from them" \
    untagged=ether7,ether8,ether9,ether10,ether11,ether12,ether13,ether14 \
    vlan-ids=20
add bridge=bridge-VLAN40 comment="Switchports where the vMotion port of all ES\
    Xi servers are hooked are tagged as packets coming from them are tagged an\
    d - of course - they are WITHIN the same VLAN" tagged=\
    ether20,ether21,ether22,ether3,ether4,ether5 vlan-ids=40
add bridge=bridge-VLAN10 comment="Switchports where the MGMT port of the ESXi \
    servers are hooked are tagged. Switchport where the MGMT port of the datas\
    tore is hooed is untagged. However ALL devices (ESXi servers and datastore\
    ) are WITHIN the same VLAN" tagged=\
    ether3,ether4,ether5,ether20,ether21,ether22 untagged=ether6,ether23 \
    vlan-ids=10
/ip address
add address=10.26.11.19/22 comment=\
    "Management interface (eth1 and eth2 in bonding)" interface=MGMT-Bonding \
    network=10.26.8.0
add address=XXX.XXX.XXX.XXX/29 comment=\
    "Public IP address (within Nova Telecom network) of up-link" interface=\
    ether24 network=HHH.HHH.HHH.HHH
add address=192.168.10.1/24 comment=\
    "Gateway for VLAN 10: IP address VLAN 10 MGMT network" interface=\
    bridge-VLAN10 network=192.168.10.0
add address=192.168.20.1/24 comment="IP address VLAN 20 iSCSI network" \
    interface=bridge-VLAN20 network=192.168.20.0
add address=192.168.40.1/24 comment="IP address VLAN 40 vMotion network" \
    interface=vlan40 network=192.168.40.0
/ip dhcp-client
add interface=ether1
/ip dhcp-server network
add address=192.168.10.0/24 gateway=192.168.10.1
/ip dns
set allow-remote-requests=yes servers=91.142.110.5,91.142.110.20
/ip firewall filter
add action=accept chain=input comment=\
    "Access tthis router through Internet from WinBox" disabled=yes \
    dst-address=XXX.XXX.XXX.XXX dst-port=8291 protocol=tcp
add action=drop chain=forward comment="Block any traffic from VLAN 10 to MGMT \
    ports (ether1 & ether2) and viceversa" in-interface=bridge-VLAN10 log=yes \
    out-interface=MGMT-Bonding
add action=drop chain=forward comment=\
    "Block any traffic from VLAN 10 to VLAN 20 and viceversa" in-interface=\
    bridge-VLAN10 log=yes out-interface=bridge-VLAN20
add action=drop chain=input comment="Block any public access for incoming conn\
    ections on IP address: XXX.XXX.XXX.XXX on eth24" dst-address=XXX.XXX.XXX.XXX \
    log=yes
add action=drop chain=input comment=\
    "Block any private access to router from VLAN 10 network (MGMT)" \
    dst-address=192.168.10.1 log=yes src-address=192.168.10.0/24
add action=drop chain=input comment=\
    "Block any private access to router from VLAN 20 network (iSCSI)" \
    dst-address=192.168.20.1 log=yes src-address=192.168.20.0/24
add action=drop chain=input comment=\
    "Block any private access to router from VLAN 10 network (iSCSI)" \
    dst-address=192.168.20.1 in-interface=bridge-VLAN10 log=yes
add action=drop chain=input comment=\
    "Block any private access to router from VLAN 30 network (Test)" \
    dst-address=192.168.30.1 log=yes src-address=192.168.30.0/24
add action=drop chain=input comment=\
    "Block any private access to router from VLAN 40 network (vMotion)" \
    disabled=yes dst-address=192.168.40.1 log=yes src-address=192.168.40.0/24
/ip firewall mangle
add action=mark-routing chain=prerouting comment=\
    "Marked route through port eth4 of Nova Telecom switch" log=yes \
    new-routing-mark=Nova-Router-through-eth4 passthrough=yes src-address=\
    192.168.10.0/24
/ip firewall nat
add action=src-nat chain=srcnat comment="Outgoing NAT from vLAN 10" log=yes \
    out-interface=ether24 src-address=192.168.10.0/24 to-addresses=\
    XXX.XXX.XXX.XXX
add action=dst-nat chain=dstnat comment="It allows incoming connections to por\
    t 21 of router from any jump server in EDC2" disabled=yes dst-address=\
    XXX.XXX.XXX.XXX dst-port=21 log=yes protocol=tcp src-address=\
    YYY.YYY.YYY.YYY/24 to-addresses=192.168.10.1 to-ports=21
add action=dst-nat chain=dstnat comment="It allows incoming connections to por\
    t 80 of router from any jump server in EDC2" disabled=yes dst-address=\
    XXX.XXX.XXX.XXX dst-port=80 log=yes protocol=tcp src-address=\
    YYY.YYY.YYY.YYY/24 to-addresses=192.168.10.1 to-ports=21
add action=dst-nat chain=dstnat comment="VLAN10 - From (XXX.XXX.XXX.XXX:31022) T\
    o (192.168.10.31:22): Incoming SSH access to ESXi ectrprd01.ireland.tm fro\
    m any jump server in EDC2" dst-address=XXX.XXX.XXX.XXX dst-port=31022 log=\
    yes protocol=tcp src-address=YYY.YYY.YYY.YYY/24 to-addresses=192.168.10.31 \
    to-ports=22
add action=dst-nat chain=dstnat comment="VLAN10 - From (XXX.XXX.XXX.XXX:21022) T\
    o (192.168.10.21:22): Incoming SSH access to vCSA ffrom any jump server in\
    \_EDC2" dst-address=XXX.XXX.XXX.XXX dst-port=21022 log=yes protocol=tcp \
    src-address=YYY.YYY.YYY.YYY/24 to-addresses=192.168.10.21 to-ports=22
add action=dst-nat chain=dstnat comment="VLAN10 - From (XXX.XXX.XXX.XXX:3389) To\
    \_(192.168.10.30:3389): Incoming RDP access to ECTR lab Jump MGMT server f\
    rom any jump server in EDC2" dst-address=XXX.XXX.XXX.XXX dst-port=3389 log=\
    yes protocol=tcp src-address=YYY.YYY.YYY.YYY/24 to-addresses=192.168.10.30 \
    to-ports=3389
add action=dst-nat chain=dstnat comment="RDP access to laptop in server room c\
    onnected to port 23 of router on mgmt network" dst-address=XXX.XXX.XXX.XXX \
    dst-port=33389 log=yes protocol=tcp src-address=YYY.YYY.YYY.YYY/24 \
    to-addresses=192.168.10.200 to-ports=3389
add action=dst-nat chain=dstnat comment="VLAN10 - From (XXX.XXX.XXX.XXX:443) To \
    (192.168.10.21:443): Incoming Web Access (TCP) to vCSA from any jump serve\
    r in EDC2" dst-address=XXX.XXX.XXX.XXX dst-port=21443 log=yes protocol=tcp \
    src-address=YYY.YYY.YYY.YYY/24 to-addresses=192.168.10.21 to-ports=443
add action=dst-nat chain=dstnat comment="VLAN10 - From (XXX.XXX.XXX.XXX:21443) T\
    o (192.168.10.21:443): Incoming Web Access (TCP) to vCSA from any jump ser\
    ver in EDC2" dst-address=XXX.XXX.XXX.XXX dst-port=31443 log=yes protocol=\
    tcp src-address=YYY.YYY.YYY.YYY/24 to-addresses=192.168.10.31 to-ports=443
add action=dst-nat chain=dstnat comment="VLAN10 - From (XXX.XXX.XXX.XXX:41022) T\
    o (192.168.10.41:22): Incoming SSH access to ESXi ectrprd02.ireland.tm fro\
    m any jump server in EDC2" dst-address=XXX.XXX.XXX.XXX dst-port=41022 log=\
    yes protocol=tcp src-address=YYY.YYY.YYY.YYY/24 to-addresses=192.168.10.41 \
    to-ports=22
add action=dst-nat chain=dstnat comment="VLAN10 - From (XXX.XXX.XXX.XXX:41443) T\
    o (192.168.10.41:443): Incoming Web Access (TCP) to ESXi ectrprd02.ireland\
    .tm from any jump server in EDC2" dst-address=XXX.XXX.XXX.XXX dst-port=\
    41443 log=yes protocol=tcp src-address=YYY.YYY.YYY.YYY/24 to-addresses=\
    192.168.10.41 to-ports=443
add action=dst-nat chain=dstnat comment="VLAN10 - From (XXX.XXX.XXX.XXX:51022) T\
    o (192.168.10.51:22): Incoming SSH access to ESXi ectrprd03.ireland.tm fro\
    m any jump server in EDC2" dst-address=XXX.XXX.XXX.XXX dst-port=51022 log=\
    yes protocol=tcp src-address=YYY.YYY.YYY.YYY/24 to-addresses=192.168.10.51 \
    to-ports=22
add action=dst-nat chain=dstnat comment="VLAN10 - From (XXX.XXX.XXX.XXX:51443) T\
    o (192.168.10.51:443): Incoming Web Access (TCP) to ESXi ectrprd03.ireland\
    .tm from any jump server in EDC2" dst-address=XXX.XXX.XXX.XXX dst-port=\
    51443 log=yes protocol=tcp src-address=YYY.YYY.YYY.YYY/24 to-addresses=\
    192.168.10.51 to-ports=443
add action=dst-nat chain=dstnat comment="VLAN10 - From (XXX.XXX.XXX.XXX:61022) T\
    o (192.168.10.61:22): Incoming SSH access to TrueNAS ectrsto.ireland.tm  f\
    rom any jump server in EDC2" dst-address=XXX.XXX.XXX.XXX dst-port=61022 \
    log=yes protocol=tcp src-address=YYY.YYY.YYY.YYY/24 to-addresses=\
    192.168.10.61 to-ports=22
add action=dst-nat chain=dstnat comment="VLAN10 - From (XXX.XXX.XXX.XXX:61443) T\
    o (192.168.10.61:443): Incoming Web Access (TCP) to TrueNAS ectrsto.irelan\
    d.tm from any jump server in EDC2" dst-address=XXX.XXX.XXX.XXX dst-port=\
    61443 log=yes protocol=tcp src-address=YYY.YYY.YYY.YYY/24 to-addresses=\
    192.168.10.61 to-ports=443
/ip route
add comment="Route to access (and being accessible) from the office network in\
    \_Cork (10.26.0.0/22)" disabled=yes distance=1 dst-address=10.26.0.0/22 \
    gateway=10.26.8.1 routing-mark=NOVA
add comment="It sends any default packets to gateway of Nova Telecom" \
    distance=1 gateway=PPP.PPP.PPP.PPP pref-src=0.0.0.0
/system clock
set time-zone-name=Europe/Dublin
/system identity
set name=ts-router2
/system routerboard settings
set boot-os=router-os

>>Also, you use the "old" way of configuring VLANs on Mikrotik (one bridge per VLAN), which effectively means the CRS forwards all VLANs but one in software, and it's CPU is by far not sufficient for wirespeed forwarding. This has nothing to do with your issue, thoughI have been "struggling" in finding the best (and most elegant) way to configure VLANs in MikroTik routers like mine but this is only the one I was able to put in place. If you can spare sometime to provide some documentation or point me to a better direction, that would be highly appreciated

>>What is the ESXi version? I had issues with NIC redundancy with some older ESXi versionsIt's in the original post: 6.7 U3

Regards.

Ivano

sindy · Fri Jan 07, 2022 11:03 pm

You have done almost everything the correct way for the "all VLANs on a single bridge" approach, but broke it all by doing that on three independent bridges. And I am surprised the configuration doesn't flash red and beep, because you cannot make a single Ethernet interface a member port of multiple bridges, which is exactly what you've done.

So can you confirm that you use 10.26.11.19 to manage the switch, i.e. you won't lose management access to it no matter what happens to VLANs 10, 20, and 40?

I can give you a script to copy-paste line by line, or interpreted manually using mouse clicks if you prefer, to fix the mess, but I need to know whether some extra steps are required to migrate the management access safely or whether the fix can be done the most straightforward way.

Also please clarify whether the CRS itself needs to have an own IP address in each of these 3 VLANs because it is used as a gateway for other devices in those VLANs/subnets.

ivanobuffa · Tue Feb 15, 2022 10:00 am

Hi Sindy,
Apologies for the late reply. Forgot to reply a while ago (actually the day after you sent me your last post).
After reviewing my configuration, everything is correct and I'm surprised you said it was a mess. I repeat, everything was correctly configured.
The root cause was a bug in VMware which was fixed in vSphere 7.
Thanks for your help btw. You were very helpful and that was highly appreciated.
You can close this topic.
Regards.
~Ivano

sindy · Tue Feb 15, 2022 2:34 pm

I'm surprised you said it was a mess. I repeat, everything was correctly configured.

Well, my concern was that you use the "one bridge per VLAN" approach, which normally excludes hardware-accelerated bridging - but maybe on CRS, this is not an issue as long as each such bridge uses a distinct set of Ethernet ports. So if you can see the H indicator on all rows of /interface bridge port print, indicating that hardware acceleration is indeed active, it's OK.

What is against all what the manuals say is that you've configured the same Ethernet interfaces to be member ports of two distinct bridges under /interface bridge vlan, but I'd assume that what prevails is the /interface bridge port section, so the interface names in red in the lists are actually ignored.

/interface bridge port
add bridge=bridge-VLAN10 interface=ether3 pvid=10
add bridge=bridge-VLAN10 interface=ether4 pvid=10
add bridge=bridge-VLAN10 interface=ether5 pvid=10
add bridge=bridge-VLAN10 interface=ether6 pvid=10
...
add bridge=bridge-VLAN40 comment=defconf interface=ether20 pvid=40
add bridge=bridge-VLAN40 comment=defconf interface=ether21 pvid=40
add bridge=bridge-VLAN40 comment=defconf interface=ether22 pvid=40
add bridge=bridge-VLAN10 interface=ether23 pvid=10

/interface bridge vlan
...
add bridge=bridge-VLAN40 tagged=ether3,ether4,ether5,ether20,ether21,ether22 vlan-ids=40
add bridge=bridge-VLAN10 tagged=ether3,ether4,ether5,ether20,ether21,ether22 untagged=ether6,ether23 vlan-ids=10

You can close this topic.

This is not a support portal, just a peer forum, so technically I cannot as I am not an admin

Switchports with multi VLANs

Switchports with multi VLANs

Re: Switchports with multi VLANs

Re: Switchports with multi VLANs

Re: Switchports with multi VLANs

Re: Switchports with multi VLANs

Re: Switchports with multi VLANs

Who is online