HI,
There is a post explaining how to deal in details with vlans on mikrotik devices: http://forum.mikrotik.com/t/using-routeros-to-vlan-your-network/126489/1
You’ll find all you need in it to achieve what I’m suggesting below.
What you want to achieve is not clear for me, so I assume the following:
- You want both Proxmox servers having the same trunk with your 3 vlans
- When you write that the opnsense is “acting as a gateway” it means that the CRS is in charge of doing the routing
- The OpnSense gateway is in charge of the NAT (i.e. the Mikrotik router does no NAT)
I have the following setup (stripped down to focus on what you want to achieve, except for the DMZ part and ISP gateway, but these have no impact on the following explanation). There is no NAT between the internal router and Opnsense (in order to be able to see outgoing traffic source on OpnSense). Each link between boxes represents a single cable. I guess that it is similar to what you are aiming at.
Firewall
VPN servers
(no control) NAT trunk Vlans Internal router
┌───────────┐ ┌─────────────┐ "WAN", "VPN" ┌──────────────┐
Internet ─────┤ISP gateway├─────────┤ Opnsense ├──────────────┤ Mikrotik ├───────── Rest of the network
└───────────┘ └──────┬──────┘ └───────┬──────┘
│ │
│DMZ vlan │ Trunk Vlans 100,200,...
│ │
┌──────┴──────┐ ┌───────┴──────┐
│ Proxmox │ │ Proxmox │ 1..n hypervisors connected
└─────────────┘ └──────────────┘ to different ports
DMZ services LAN Services
If that’s more or less what you want to achieve, here is what you need to do:
- On OpnSense, create a VLAN on the interface which is connected to the Mikrotik router (keep vlan id, network address and mask for later usage)
- On Mikrotik router, create a new VLAN interface for your “WAN” VLAN, give it the above defined vlan id.
- On Mikrotik router, set the IP address of the VLAN interface + Set OpnSense appliance IP as gateway for WAN VLAN (they must be on the same network of course)
- On Mikrotik router, set the WAN interface list not to ether24 (connected to OpnSense appliance), but to your “WAN” VLAN interface
- On Mikrotik router, deactivate src-NAT for outgoing traffic (you’ll do NAT on the OpnSense appliance
- On Mikrotik router, in bridge port ether24, set pvid=1, set admit only vlan tagged (unless router IP address for WAN vlan is obtained through DHCP from Opnsense in which case set it to admit all) + activate ingress filtering
- On Mikrotik router, in bridge vlan, add an entry where bridge=your bridge, vlan=vlan id of WAN vlan, tagged=bridge AND ether24
- On Mikrotik router, make sure that your firewall rules allowing to access WAN are not using ether24 as outgoing interface, but WAN instead
Once done, this should allow you to access WAN from the inside of your network, and see the source IP of outgoing requests in OpnSense.
If as me you need to run vpn servers on the OpnSense appliance, add a VLAN for each VPN on the Opnsense and declare it on Mikrotik router side + add it to the trunk port (i.e. in bridge vlan add an entry as above, with vlan=vlanid of your VPN vlan, tagged=bridge and ether24)
Now the Hypervisor/Proxmox part:
- On Mikrotik router, create a new VLAN interface for your vlan id 20.
- On Mikrotik router, set the IP address of the created VLAN interface having vlan id 20
- Repeat previous two steps for vlanid 30 and 70
- On Mikrotik router, in bridge port ether2, set pvid=1, set admit only vlan tagged + activate ingress filtering
- On Mikrotik router, in bridge port ether3, set pvid=1, set admit only vlan tagged + activate ingress filtering
- On Mikrotik router, in bridge vlan, add an entry for your bridge, vlan=20, tagged= bridge, ether2, ether3
- On Mikrotik router, in bridge vlan, add an entry for your bridge, vlan=30, tagged= bridge, ether2, ether3
- On Mikrotik router, in bridge vlan, add an entry for your bridge, vlan=70, tagged= bridge, ether2, ether3
- On Mikrotik router, make sure that your firewall rules allowing to access your different vlans are not using ether2/3 as outgoing interface, but as outgoing interface list.
Now the two hypervisors are on the same vlans. You need then to declare them correctly on your virtual bridges in Proxmox configuration and assign the created vbrs to your VMs as needed.
Hope it helps