Community discussions

MikroTik App
 
TopHatProductions115
newbie
Topic Author
Posts: 30
Joined: Fri Dec 10, 2021 2:44 am

Redoing Bridge VLAN Setup

Fri Apr 07, 2023 2:31 am

I'm currently in a strange scenario. I have an ESXi server that needs to run on its own private network(s), and requires Internet access. The rest of my household relies on an ISP-provided router/WAP, with a limited feature set. That (ISP-provided) router is also on a different floor than the ESXi server, and I can't run any network cables to it since it sits in the living room. The only way that I've been able to connect to it is wirelessly. I currently have a RB4011iGS+5HacQ2HnD-IN acting as my router, but I doubt that this device is a good pick for long-term use.

I currently have the following:
  • Chateau 5G ax
  • RB4011iGS+5HacQ2HnD-IN
  • CCR2004-1G-12S+2XS

And am considering acquiring this:
  • CRS317-1G-16S+RM

I want to have a bridge VLAN setup for my ESXi server and its VMs. However, I can't really rely on (tagging) physical access ports for this setup due to the way vSphere handles networking. There would always be multiple VLANs going over any single physical port on the server. It's also impractical to assign physical NICs to each VM due to the number of running VMs (vs the no. PCIe slots in the physical server). The one exception is HPE iLO, which isn't controlled or managed by vSphere and has its own dedicated port on the back of the server.

My first guess is that I could use either the CCR2004-1G-12S+2XS or the CRS317-1G-16S+RM to host the bridge VLANs themselves. From what I've read thus far, the CRS317-1G-16S+RM seemed to be potentially more performant for this task because of its high-end switch chip. I think the access port that iLO connects to can be tagged, since it would be the only host connected (though I could be wrong in my logic). On the vSphere side, the portgroup for my VMs is tagged with VLAN ID 4095 and the VMs handle tagging (VGT). But, how would I connect to the ISP WAP? Could I use the Chateau 5G ax as a wireless uplink in this setup?

What are your thoughts?

P.S. I will be uploading my current config in a bit, for review.
You do not have the required permissions to view the files attached to this post.
 
User avatar
Frederick88
newbie
Posts: 49
Joined: Thu Jun 24, 2021 12:34 pm

Re: Redoing Bridge VLAN Setup

Fri Apr 07, 2023 6:57 pm

depends what you want responsible for the routing of the “private network” within the ESXI host…

if it’s your ISP router, you need to trunk from router to ESXI host and then assign each vlan network to your guest OSes.

otherwise you could virtualise a router within the ESXI host, create virtual interfaces to assign to each guest OS, then uses existing ISP router as its gateway. double NAT’ing effectively.
 
TopHatProductions115
newbie
Topic Author
Posts: 30
Joined: Fri Dec 10, 2021 2:44 am

Re: Redoing Bridge VLAN Setup

Mon Apr 10, 2023 10:19 am

if it’s your ISP router, you need to trunk from router to ESXI host and then assign each vlan network to your guest OSes.
I'd rather not increase my reliance on that for the rest of my equipment. I've been less than impressed with its reliability as of late. Also, the ISP router does not appear to have user-defined VLANs as a feature. I'm assuming that I'd end up creating the VLANs on the vSphere side instead.

otherwise you could virtualise a router within the ESXI host, create virtual interfaces to assign to each guest OS, then uses existing ISP router as its gateway. double NAT’ing effectively.
Would the virtual router require a wireless NIC (PCI Passthrough) to connect to the ISP router? How will it reach the ISP router (gateway) in this scenario? Would it be impractical for me to use the CCR2004-1G-12S+2XS (that I already have) as my router, instead of creating a router VM in ESXi? Just curious.

According to VMware, there are three major variations for VLAN tagging:
Who would be responsible for VLAN tagging in both of these scenarios? OS/driver, vSwitch, or MikroTik equipment?
 
User avatar
Frederick88
newbie
Posts: 49
Joined: Thu Jun 24, 2021 12:34 pm

Re: Redoing Bridge VLAN Setup

Mon Apr 10, 2023 4:12 pm

well i think you have quite a few different options and ways you can do this…

i guess first question is, how much “control” or influence you have over the whole network, from ISP connection to you…?

if you have freedom to do anything you want, it might be worth considering bridging your ISP modem/router to your CCR2004-1G-12S+2XS, and let that take care of all the routing, including the networks to use within virtual ESXI environment….

i guess it also depends on how you want your ESXI network to look like, firewall rules, routing, restrictions, etc…

if you create a virtual router within the ESXI, all it needs is a connection to your ISP gateway, one way or another, which will effectively be the WAN for your virtual router. all the guest OSes can be given their own VLAN / network accordingly.. their firewall rules will depend on 1. the OS rules you set and then 2. the router that allows/denies it, including routing between virtual networks…

it also comes down to how complex and scalable you want your virtual environment…. truth is, you can have a combination of router and network rules presented to ESXI, but still create virtual networks with that.. i think ultimately it comes down to what your guest OSes require and need…..
 
User avatar
Frederick88
newbie
Posts: 49
Joined: Thu Jun 24, 2021 12:34 pm

Re: Redoing Bridge VLAN Setup

Mon Apr 10, 2023 4:16 pm

i think instead of trying to understand all the possibilities and ways it can be done - what is your goal and requirements….

explain what you’re trying to achieve and why..

based on this. we can minimise your options and provide a clearly list of examples how it could be done, along with the pros and cons of it….
 
TopHatProductions115
newbie
Topic Author
Posts: 30
Joined: Fri Dec 10, 2021 2:44 am

Re: Redoing Bridge VLAN Setup

Tue Apr 11, 2023 1:15 am

i think instead of trying to understand all the possibilities and ways it can be done - what is your goal and requirements….

explain what you’re trying to achieve and why..

based on this. we can minimise your options and provide a clearly list of examples how it could be done, along with the pros and cons of it….
The ISP router is a living paradox in this case. I have administrative access to it and can change pretty much any setting I see fit. But it has so few supported (let alone, working) capabilities that I'd rather just be another client on said network. For example, the ISP router officially supports both port forwarding and DMZ. But only the DMZ function appears to work as it should. So, I ended up setting the current MikroTik router to CPE mode and connecting it as just another wireless client to the ISP router. I then threw the MikroTik into the DMZ and called it a day. I've had to manage firewall and everything else on the MikroTik side, which I don't really mind.

I want a bridge VLAN filtering setup for my ESXi server, so that I can isolate certain VMs/apps/services, like:
  • vSphere vMotion, vSAN, Replication, etc.
  • federated services (ie., PeerTube, Pleroma, etc.)
  • private/restricted services like iLO, vCenter, ESXi

I've also seen scenarios like these recently:
Where the ISP has their own priority/VLAN tag, and it needs to be stripped for successful DHCP lease. The tag can be stripped by a bridge (or at the interface). In preparation for when I move out (and inevitably get Internet service), I would like to have something ready for this scenario, among other possible hurdles.

My first thread on this was listed here, but didn't get very far:
I ended up dropping the idea of having vSphere manage VLANs. After creating the portgroups and VLANs on the Distributed Switch, some of my VMs were unable to migrate to the new VLANs. I faired better with having the VLANs managed by MikroTik equipment instead, and then having the VMs handle VLAN tagging. But, the current device hosting said VLANs appears to struggle with this task - especially compared to when it only hosted a single network.
 
LikeMyFloydPink
just joined
Posts: 14
Joined: Mon May 30, 2022 9:00 pm
Location: South Carolina

Re: Redoing Bridge VLAN Setup

Tue Apr 11, 2023 5:42 am

One thing to consider which may affect your setup is VMware Licensing. If you are using the vSphere/ESXi Evaluation, then you know you'll lose functionality and full feature/resource utilization after 60 days (I believe it's 60 days...) If that is the case, best to just scale back to using ESXi standard vSwitch (also just using the ESXi feature set) rather than the vSphere DS/features.

Below may be irrelevant:

You can create VLANs and assign them to one physical ether/SFP+ Router interface (i.e. - your uplink to ESXi port) which automatically changes the interface encapsulation to 802.1Q trunk. Then put IP addresses on the VLAN interfaces. Create a standard vSwitch and then add port groups for each VLAN you want to tag and then assign the PG to respective VM(s).
 
TopHatProductions115
newbie
Topic Author
Posts: 30
Joined: Fri Dec 10, 2021 2:44 am

Re: Redoing Bridge VLAN Setup

Wed Apr 12, 2023 8:22 am

One thing to consider which may affect your setup is VMware Licensing. If you are using the vSphere/ESXi Evaluation, then you know you'll lose functionality and full feature/resource utilization after 60 days (I believe it's 60 days...) If that is the case, best to just scale back to using ESXi standard vSwitch (also just using the ESXi feature set) rather than the vSphere DS/features.

Below may be irrelevant:

You can create VLANs and assign them to one physical ether/SFP+ Router interface (i.e. - your uplink to ESXi port) which automatically changes the interface encapsulation to 802.1Q trunk. Then put IP addresses on the VLAN interfaces. Create a standard vSwitch and then add port groups for each VLAN you want to tag and then assign the PG to respective VM(s).

I'm using an Enterprise Plus license, so I have indefinite access to Distributed Switch and all of its features. What I don't have is an NSX license. I currently have 8 VLANs to account for, and may add more in the future.

I'm not sure if I understood your last suggestion. Let me know if I missed the target (likely):
  • create a new standard vSwitch
  • assign one or more physical NICs (uplinks) to the vSwitch
  • create one portgroup for each VLAN, on said vSwitch
  • assign individual VLAN IDs to each portgroup
  • assign one or more vmkernel NICs to each portgroup
  • assign IPv4 addresses (within each VLAN's intended subnet) to each vmkernel NIC
  • migrate all VMs to the newly tagged portgroups

If the above process is correct, I've attempted it before. This relies on VST iirc (where the Virtual Switch handles VLAN tagging, in opposed to the VMs or another entity). The two main issues that I ran into with that plan were that not all of my VMs were able to migrate successfully, and any that did make it had no Internet connectivity after the migration - which destroyed their ability to download vital security patches. I'll ignore the first issue for the time being, since that would possibly require a separate trip to the VMware Community forums to resolve. Proper routing (outside of each VLAN) is one thing that may have to be done by a separate entity. There are three main options that I know of for this role (of VLAN router):
  • ESXi: dedicated router VM (RouterOS, VyOS, etc.)
  • NSX-T: Gateway/Logical Router
  • Opaque: Routing defined on dedicated hardware (MikroTik)

The first option requires yet another VM, which in turn needs compute resources from the hypervisor host and will be present on all portgroups/VLANs. If the hypervisor host ever comes under resource contention, I'd think network performance could take a hit. Less headroom/leeway if something goes wrong. The second option requires a paid subscription for NSX-T, which is unrealistic. The third option would offload the task to a physical router/switch (which I already own).

Here are a few questions that I have as a result of this:
  • What's handling DHCP when I define VLANs in vSphere? Is it the VLAN router I mentioned above?
  • Which will perform better in most scenarios - a physical router/switch or a vSwitch?
  • Which will perform more consistently if my ESXi host happens to run low on system resources?
  • What are the benefits of using a vSwitch (necessitates VST) in this scenario?
  • If I end up using the third option, then would I be better off defining the VLANs on the physical router/switch as well?

Sorry if my questions seem dumb or obvious. These are the things that came to mind when reviewing all of the data so far. I've been at this issue for at least one and a half months, and haven't had much luck on the vSphere VST side so far.
 
User avatar
Frederick88
newbie
Posts: 49
Joined: Thu Jun 24, 2021 12:34 pm

Re: Redoing Bridge VLAN Setup

Wed Apr 12, 2023 10:47 am

VST should work without too much hassle...

From existing router, trunk VLANs to ESXI host.
Configure VST accordingly, ensuring it includes the physical NIC that the trunk is connected too...
The vSwitch/vmkernel NIC should then hand off each VLAN's network tot he guestOS as untagged native traffic.
https://techexpert.tips/vmware/vmware-e ... iguration/
What's handling DHCP when I define VLANs in vSphere? Is it the VLAN router I mentioned above?
to keep it simple, you'd want the router of said VLAN network to also be DHCP server.
Which will perform better in most scenarios - a physical router/switch or a vSwitch?
depends how much routing you're doing between the guestOSes networks - virtualised router will generally be faster as data is all contained within ESXI host, but subject to the available ESXI resources.
If I end up using the third option, then would I be better off defining the VLANs on the physical router/switch as well?
even if you virtualise a router, you're still going to be configuring virtual networks and interfaces within ESXI... only difference is physical NIC vs virtual NIC, effectively.

Advantages of physical router:
  • You already have it in use - minimal setup compared to a new virtual router from scratch
  • Single place for all your routing and network rules
  • It's not subject to ESXI resources
Advantages of virtual router:
  • Potentially better performance (subject to ESXI resources)
  • WAN agnostic - will work as long as you have a WAN connection - can move ESXI host to another location without reconfiguring your host networks
Disadvantages:
  • another place to manage your networks in addition to existing router
  • GuestOSes will effectively be double NAT'ing to the internet
 
LikeMyFloydPink
just joined
Posts: 14
Joined: Mon May 30, 2022 9:00 pm
Location: South Carolina

Re: Redoing Bridge VLAN Setup

Fri Apr 14, 2023 6:11 am

Just now sitting back down with time to respond and without re-quoting the previous 2 posts - all I can offer (FWIW) is:

- @Frederick88 is giving the best method of attack

- I now see that I can't provide any insight to your use-case as I don't currently know how to circumvent the vendor recommendations allowing for wired connectivity/management other than seeking out someone with an air-gapped setup for consulting purposes, perhaps. It's a shame that apparently an ethernet cable to the living room seems out of the question.
 
TopHatProductions115
newbie
Topic Author
Posts: 30
Joined: Fri Dec 10, 2021 2:44 am

Re: Redoing Bridge VLAN Setup

Sun Apr 16, 2023 5:36 am

VST should work without too much hassle...

From existing router, trunk VLANs to ESXI host.
Configure VST accordingly, ensuring it includes the physical NIC that the trunk is connected too...
The vSwitch/vmkernel NIC should then hand off each VLAN's network tot he guestOS as untagged native traffic.
https://techexpert.tips/vmware/vmware-e ... iguration/
What's handling DHCP when I define VLANs in vSphere? Is it the VLAN router I mentioned above?
to keep it simple, you'd want the router of said VLAN network to also be DHCP server.
Which will perform better in most scenarios - a physical router/switch or a vSwitch?
depends how much routing you're doing between the guestOSes networks - virtualised router will generally be faster as data is all contained within ESXI host, but subject to the available ESXI resources.
If I end up using the third option, then would I be better off defining the VLANs on the physical router/switch as well?
even if you virtualise a router, you're still going to be configuring virtual networks and interfaces within ESXI... only difference is physical NIC vs virtual NIC, effectively.

Advantages of physical router:
  • You already have it in use - minimal setup compared to a new virtual router from scratch
  • Single place for all your routing and network rules
  • It's not subject to ESXI resources
Advantages of virtual router:
  • Potentially better performance (subject to ESXI resources)
  • WAN agnostic - will work as long as you have a WAN connection - can move ESXI host to another location without reconfiguring your host networks
Disadvantages:
  • another place to manage your networks in addition to existing router
  • GuestOSes will effectively be double NAT'ing to the internet

Thank you for addressing my questions. I will be sure to keep this in mind.
 
TopHatProductions115
newbie
Topic Author
Posts: 30
Joined: Fri Dec 10, 2021 2:44 am

Re: Redoing Bridge VLAN Setup

Sun Apr 16, 2023 5:38 am

Just now sitting back down with time to respond and without re-quoting the previous 2 posts - all I can offer (FWIW) is:

- @Frederick88 is giving the best method of attack

- I now see that I can't provide any insight to your use-case as I don't currently know how to circumvent the vendor recommendations allowing for wired connectivity/management other than seeking out someone with an air-gapped setup for consulting purposes, perhaps. It's a shame that apparently an ethernet cable to the living room seems out of the question.

The air-gap is definitely a pain point in the current setup. No matter how fast the internal networking for my server rack is, I will always run into a bottleneck once traffic leaves the rack environment :( To add insult to injury, the RB4011iGS+5HacQ2HnD-IN that's currently handling my VLAN setup doesn't seem to be handling it well. It runs hot, and network performance has been down. I think I need to move to a proper switch at this point.
 
TopHatProductions115
newbie
Topic Author
Posts: 30
Joined: Fri Dec 10, 2021 2:44 am

Re: Redoing Bridge VLAN Setup

Thu Apr 20, 2023 2:23 am

All discussion of upcoming changes to my setup are moving to a new thread. If you are interested in a challenge, stay tuned...

Who is online

Users browsing this forum: JDF, jookraw, m4rk3J, mhn6868 and 38 guests