Community discussions

MikroTik App
 
adrianlewis
just joined
Topic Author
Posts: 11
Joined: Tue Oct 14, 2014 6:09 pm

Bridging VLAN interfaces on the same physical port

Tue Oct 14, 2014 6:46 pm

Hi,

First post here. I'm planning to implement a CCR to allow me to be able to connect multiple virtual firewalls (not Mikrotik) to a single internet IP subnet. The other vendor allows me to create the virtual firewalls with connections to the outside world via a single physical interface, but each virtual firewall HAS to have a different VLAN ID assigned to its internet interface. The virtual firewalls cannot share a single L2 network but they can be addressed with IPs on the same L3 subnet.

I have had partial success testing with a little RB951 by creating a bridge which includes multiple VLAN interfaces. If I create an IP address for the bridge interface which acts as a default gateway for the virtual firewalls, each of the virtual firewalls can access the bridge IP and route out to the internet. However, there is no traffic between the VLANs. If I use a different physical port for a VLAN interface and connect this to the same bridge, I do get VLAN-to-VLAN bridging but when the VLAN interfaces are on the same physical port, all VLAN-to-VLAN traffic is dropped.

Does anyone know if this is simply a default behaviour that can somehow be overridden or is this a bug of some sort?

I appreciate that switching and bridging works by flooding to all ports for broadcast frames and that the originating port is not included in this flooding but VLAN interfaces should be treated as separate interfaces in this mechanism even though they may be associated with the same physical port as the source of the traffic.

Basically, a single physical port with multiple VLAN sub interfaces all placed into a bridge behaves very similarly to a PVLAN trunk where the bridge interface is the promiscuous 'port' and the subinterfaces are the community/isolated 'ports'. I need all the 'ports' to be bridged.
 
User avatar
jacekes
Member Candidate
Member Candidate
Posts: 167
Joined: Tue Aug 30, 2011 9:34 am
Location: Poznan, Poland
Contact:

Re: Bridging VLAN interfaces on the same physical port

Wed Oct 15, 2014 9:23 am

I've had such a configuration (many VLAN interfaces on an ether interface, all added to a bridge) and it was working as you wish it would - layer 2 traffic was forwarded between the VLANs. I've used it on ROS v. 5.16, 5.19 and 6.10.

As for me, I didn't want the traffic to be forwarded and had to set up a bridge filter in forward chain. Check if your bridge filter has any entries.
I was certified a long time ago:
MTCNA# 1210NA193 MTCTCE# 1210TCE056 MTCWE# 1211WE010

ONE NETWORK DIAGRAM IS WORTH MORE THAN A THOUSAND WORDS!
 
adrianlewis
just joined
Topic Author
Posts: 11
Joined: Tue Oct 14, 2014 6:09 pm

Re: Bridging VLAN interfaces on the same physical port

Tue Oct 21, 2014 4:27 pm

Thanks for the info. I've just bought a pair of CCRs and will do some further testing. It's good to hear that it is at least possible (or should be) and this gives me some hope that the issue can be corrected. There were definitely no bridge filter rules but I'm going to test to see if there's any difference between a port on a switch chip vs a direct-to-cpu port. Fingers crossed I can resolve this on my own but if not, I'll try sending a support email once I've exhausted my testing efforts. I'm guessing that this is a fairly unusual configuration and so if a RouterOS change is the source of the issue, it may not have been noticed by anyone willing to inform Mikrotik about it.

If anyone else has done anything similar, I'd appreciate any info they have on this.
 
flipk12
newbie
Posts: 35
Joined: Mon Oct 06, 2014 5:49 pm
Location: Asturias/Spain

Re: Bridging VLAN interfaces on the same physical port

Tue Oct 21, 2014 5:57 pm

I've use fortigate vdoms and juniper vrf, and you can put the same subnet on diferents vlans of the same phisical interface because they belongs to a diferent routers. But same phisical interface means same mac address. Thinking on it, your provider's router will ask for the mac of the desired IP, arp protocol will answer the same mac for all the wan IP's like they where at the same L2 interface, so it will forward the L2 traffic to the mac, the traffic will arrive at the "main" interface with no tag on it, it will say my mac, my traffic, so it will lead the traffic to the l3 level but the ip that it will be trying to reach is on another vlan ..... and thinking on a bridge like a plain switch, with the same mac on many ports, where to forward the traffic, I think that rstp will say someting about it ..... but maybe I'm wrong.

Thinking on fortigate I'll do this job using one vdom to distribute traffic to the otrhers, using vdom links to join them, proxyarp and source nat to lead traffic in and out, or if the problem is that you have few interfaces on the firewall I will use the swith to split the traffic on the inside part of the firewall not in the outside one.

Nice problem, I've enjoyed thinking on it!!!
 
adrianlewis
just joined
Topic Author
Posts: 11
Joined: Tue Oct 14, 2014 6:09 pm

Re: Bridging VLAN interfaces on the same physical port

Tue Oct 21, 2014 6:42 pm

Oh crap (in my head I'm using far stronger words). Yes - I'm using FortiGate VDOMs and yes, all the VLAN interfaces on the same physical port use the same MAC address!

The problem with using VDOM links and a transparent VDOM to aggregate them is that you can't use these in an HA cluster with the vcluster option without creating yet another transparent VDOM for each physical member of the cluster. This also prevents you from easily migrating VDOMs between the cluster members.

This also doubles the session count on each cluster member and hence the memory utilisation as well. Might be time for a major rethink. MAC address translation?
 
flipk12
newbie
Posts: 35
Joined: Mon Oct 06, 2014 5:49 pm
Location: Asturias/Spain

Re: Bridging VLAN interfaces on the same physical port

Tue Oct 21, 2014 9:25 pm

Which fortigate? How many vdoms? And what do you use them for? If they are small ones ... do you know that you can split the switch into interfaces? If they are big ones .... do you think that CRS125 is powerfull enoug to deal with the fortigate throughput? I've got one at home and It can't deal with a 200/20Mb wan doing nat, I had bought a rb951g to do the routing job.

If you're not using them for ipsec I rather prefer another vdom at the fortigate to do nat than using mac translation.
 
adrianlewis
just joined
Topic Author
Posts: 11
Joined: Tue Oct 14, 2014 6:09 pm

Re: Bridging VLAN interfaces on the same physical port

Tue Oct 21, 2014 9:47 pm

At the moment, multiple FGT100D cluster pairs which have the physical interfaces available but it's the public IP addresses and physical switch ports that I'm more annoyed about. Instead of 4 switch ports per physical cluster pair (1 x LACP LAG on each cluster member), I'll have to use 8-10 times that to have all the WAN interfaces on a single IP subnet. Looks like I'll be forced to use physical interfaces instead of VLAN interfaces just to get unique MAC addresses for each VDOM interface.

The only other solution is to put each wan interface on a dedicated /29 and use 4 times the number of public IPs. Either way it gets very expensive to scale up. Going to have a shout at Fortinet but they're extremely unlikely to do anything about it. They don't care about IP address shortages.

Before anyone mentions this: You can't manually set a MAC address on a VLAN interface on a FortiGate - only the physical interface. There is a command of 'set substitute-dst-mac' but this is poorly documented and doesn't appear to do anything anyway. My only hope is that this could in fact be the answer but that it's broken at the moment and might be fixed at some point in the next year with a firmware update.
 
flipk12
newbie
Posts: 35
Joined: Mon Oct 06, 2014 5:49 pm
Location: Asturias/Spain

Re: Bridging VLAN interfaces on the same physical port

Tue Oct 21, 2014 11:49 pm

Ok, let's we do it without vdom links ....
If I were you I'll use a vdom to translate public ip's into private lans 172.16.1.1/29 172.16.2.1/ 172.16.3.1/29 .... 172.16.10.1/29 with NAT, I'll put each one in a separate VLAN (you'll need only 1 interface to do this using native VLAN fos public adresses and 10, 20, 30 ...... 100 for the others) and I'll use 172.16.1.2/29 ...... 172.16.10.2/29 at the out side of each customer vdom all of them in another interface. In the inner side you will need a switch to split all the vlans in ports. No matter how many vdoms you make (hardware limit is 10) it will only cost 3 ports of the fortigate, and a switch of course.
Ugly but .....
 
adrianlewis
just joined
Topic Author
Posts: 11
Joined: Tue Oct 14, 2014 6:09 pm

Re: Bridging VLAN interfaces on the same physical port

Thu Oct 23, 2014 3:38 pm

OK - Managed to get this working without using any NAT of IP or MAC addresses. Instead I'm using 'ip unnumbered' interfaces on the routerboard or what some call /32 routing in combination with proxy-arp.

Let's say we have 4.3.2.1/24 as a network where .1 is the RB IP and the gateway for the virtual firewalls. I assign that IP to a bridge interface (i.e. a Loopback interface) with the full /24 subnet - nothing fancy there. For each VLAN interface that I want to be on the same /24, I then assign the same IP (4.3.2.1) but using a /32 mask. The special bit however, is instead of leaving the network address as automatically calculated, I set this to be the IP of the virtual firewall's interface on that VLAN (e.g. 4.3.2.11, 4.3.2.21, 4.3.2.31 etc). This as I understand is much the same as using "ip unnumbered" in the Cisco world. The other special bit is to enable proxy arp on the RB's VLAN interface so that it replies to ARPs for other IPs on the same network that may be on different physical or VLAN interfaces. Even though the MAC addresses of the virtual firewall VLAN interfaces (FortiGate VDOMs) are the same and they're all on the same IP subnet, due to the fact that they're on different VLAN interfaces, this doesn't cause a problem in the MAC or ARP tables as they are no longer bridged at L2.

If I want to give a customer another IP to use, I simply put in a route to send traffic to that particular IP out of that customer's VLAN interface. It doesn't need to be an adjacent IP either.

This creates what I might call a layer 3 bridge of sorts. Traffic between interfaces is routed by the RB even though they are on the same subnet. While this might seem like unnecessary overhead for the RB, it does open up a number of useful options (in my use case at least) such as:

1. Preventing two customers from trying to use the same IP address - Unless an IP is specifically routed to that customer's VLAN interface, it will not work and more importantly, it will not affect traffic to and from the legitimate VLAN for that IP even though they're on the same IP subnet.

2. Full L3 queue functionality can be assigned per customer that also controls traffic between customers despite it appearing that they're all on the same network.

3. Reduction in broadcast domains - Each customer gets their own L2 broadcast domain on their VLAN. Only arp broadcasts are proxied and as these are also cached by the RB, there's a further reduction in broadcast traffic between customers.

Overall, I'm actually quite pleased that this issue arose as I now have a much better solution that the original one I had planned.

For a different use case of a similar configuration, check out Brian Horn's presentation at MUM US '14: https://www.youtube.com/watch?v=OTx1o8AajDY
 
flipk12
newbie
Posts: 35
Joined: Mon Oct 06, 2014 5:49 pm
Location: Asturias/Spain

Re: Bridging VLAN interfaces on the same physical port

Fri Oct 24, 2014 12:14 am

Pretty!

Who is online

Users browsing this forum: atakacs, skyfx, txfz, zark and 88 guests