MikroTik VLAN Bridging Help

runet · Tue Mar 27, 2007 12:50 am

Is there anyone that might have some knowledge of VLANs and bridging. I'm running into some issues and would like some advise.

Tue Mar 27, 2007 4:27 pm

I think you have to give more information about your issues with VLAN and bridging, if you wish to receive some advices.

runet · Tue Mar 27, 2007 5:49 pm

I will try to illustrate what I want to do:

In this experiment, there are 3 vlans that terminate on the access port of a switch on one side and on a mikrotik router on the other side.

(In real life I want to have about 100 vlans)

This is the vlan config:

add name="vlan1" mtu=1500 arp=enabled vlan-id=101 interface=ether1 comment="" disabled=no 
add name="vlan2" mtu=1500 arp=enabled vlan-id=102 interface=ether1 comment="" disabled=no 
add name="vlan3" mtu=1500 arp=enabled vlan-id=103 interface=ether1 comment="" disabled=no

Now, because I want to have all my vlans on the same subnet (so that I can hand out dynamic public IPs) I figured I would need to have a bridge, so...

/ interface bridge 
add name="sm" mtu=1500 arp=enabled stp=no priority=32768 ageing-time=5m forward-delay=15s \
    garbage-collection-interval=4s hello-time=2s max-message-age=20s comment="" disabled=no 
/ interface bridge port 
add interface=vlan1 bridge=sm priority=128 path-cost=10 comment="" disabled=no 
add interface=vlan2 bridge=sm priority=128 path-cost=10 comment="" disabled=no 
add interface=vlan3 bridge=sm priority=128 path-cost=10 comment="" disabled=no 
/ interface bridge broute 
add chain=brouting in-interface=vlan1 in-bridge=sm action=accept comment="" disabled=yes

The bridge (sm) has an IP number on the subnet I want to use for my clients; 192.168.5.0/24

/ ip address 
add address=192.168.1.221/24 network=192.168.1.0 broadcast=192.168.1.255 interface=ether2 comment="" \
    disabled=no 
add address=192.168.11.1/24 network=192.168.11.0 broadcast=192.168.11.255 interface=ether1 comment="" \
    disabled=no 
add address=192.168.5.1/24 network=192.168.5.0 broadcast=192.168.5.255 interface=sm comment="" disabled=no

I have tried this and the problem seems to be that not all clients can ping the gateway 192.168.5.1 not to mention pinging 192.168.1.221.

Is this the correct way to set this up?

runet · Tue Mar 27, 2007 7:11 pm

I have been playing around with a working experiment on the bench. And I am testing various functions.

- Clients can NOT ping each other. I would thin it should work. is this a bug?

- After a while the clients can't ping the bridge interface (sm). ->bug?

Any insight would be great.

changeip · Tue Mar 27, 2007 7:36 pm

If you create a bunch of vlans and then bridge them together you just threw out the security you get from creating a vlan ... I don't think you want to do that.

I am not really sure what to recommend to get what you need however, maybe someone else can chime in. Maybe pppoe ?

Sam

runet · Tue Mar 27, 2007 7:53 pm

Well, there is security now because I cant get the clients to see each other. Like I mentioned above I am not sure if this is a bug. My guess is that it would work like you said.

The purpose of having a bridge is so I can use one subnet for all the vlans (I have not found another way to accomplish this). in other words I could drop the bridge and have many 2 or 6 IP subnets. Who knows, dropping the bridge might fix the whole problem.

runet · Tue Mar 27, 2007 8:31 pm

I would like to emphasize here that the reason for the vlans is so that I can monitor/control bandwidth per customer. The reason for the bridge is so that I can have all my customers on the same subnet. I give my customers public IP addresses but would still like them to be assigned dynamically.

runet · Wed Mar 28, 2007 12:31 am

I am unable to get bridged vlans to work!

Connection is sporadic and unreliable!

Clients can not talk to each other despite the fact the the vlans ARE bridged!

Anyone care to comment? MT developers?

JJCinAZ · Thu Mar 29, 2007 7:51 am

Sure, I'll comment -- your design is broken. Use PPPoE or real subnetting.

runet · Fri Mar 30, 2007 4:59 am

Thank you for the comment.

The reason I am using vlans is because I want to implement QoS. I am also using Motorola Canopy.

I didn't want to subnet (but I will if I need to) because I want to hand out dynamic and public ip addresses.

PPPoE requires login, the vlan solution should be easier for the customer.

One of the reasons for coming up with the vlan design, is so I can monitor and limit bandwidth by interface -> if it comes in on vlan so and so, I know its from a certain customer. I don't need to know their mac address or anything else about them other then their VID.

And besides, why would bridging vlans be a problem?

changeip · Fri Mar 30, 2007 5:58 am

And besides, why would bridging vlans be a problem?

The whole point of vlans is to keep traffic separate. Once you bridge it then your back to no vlans pretty much. You might as well not use vlans - if you simply looking for a way to monitor / queue traffic then do it using IP addresses.

If your IP address, the gateway IP, is on vlan1, and you have customer A on vlan101, how do you expect them to talk ? 1 and 101 aren't on the same L2 network. What subnet are you planning on handing out, a /24 subnet mask? This puts everyone on the same subnet, but on different L2 networks ... doesn't work that way. The connection is sporadic because once their ARP entry times out then connectivity is gone. I'm surprised it even works at all. Do you see any other vendors doing this ?

runet · Fri Mar 30, 2007 5:26 pm

OK This is what things look like

[-----bridge$vlan1] --- [AP] ----- [Canopy SM1$]----- [customer1]
[MT-bridge$vlan2] --- [AP] ----- [Canopy SM2$]----- [customer2]
[-----bridge$vlan3] --- [AP] ----- [Canopy SM3$]----- [customer3]

The vlans terminate between the "$" signs. The advantage of using vlans is that if I am able to monitor traffic to a given customer (i.e. if it goes over vlan 101 I know it going to Mr. Smiths account). The disadvantage is that many subnets are required. If I want to provide my customers with public IPs it becomes wasteful, especially if I want my customers to receive dynamic IP assignments.

The only way I came up with so far, is to bridge the vlans and use one subnet for all vlans. Thus the bridge becomes the gateway interface.

This obviously did not work for me. and the reason is that when the vlans are bridged, the connection between the customer and the bridge comes and goes. That is why I thought there was a bug somewhere. I would think that it should ether work or not, but not both.

but on different L2 networks ... doesn't work that way. The connection is sporadic because once their ARP entry times out then connectivity is gone. I'm surprised it even works at all.

Dosn't "bridge" mean that ARP is rebroadcast over to all members of the bridge?

I have not yet found anyone else doing this. It would be nice speak with someone who has (if I can find one).

sten · Fri Mar 30, 2007 5:53 pm

Standing at that cross-road eh?
In the long run you would probably be better off with the waste of IP addresses. But why don't you at least try proxy arping and host routes instead of messing up your vlan structure? That way your vlans wouldn't leak.

EDIT:

You might actually not have to do proxy-arp since the kernel will reply arp for *any* ip on the router on *any* broadcast interface.

In 2.9.x you need to add a small unofficial address subnet for each vlan and then route the public ip to the local address on the router.

In 3.x you can route the host route out on the interface itself, forgoing that unofficial subnet. (At least that is what i read in the changelog) - a feature i can't wait to use for myself.

runet · Fri Mar 30, 2007 7:35 pm

You might actually not have to do proxy-arp since the kernel will reply arp for *any* ip on the router on *any* broadcast interface.

In 2.9.x you need to add a small unofficial address subnet for each vlan and then route the public ip to the local address on the router.

In 3.x you can route the host route out on the interface itself, forgoing that unofficial subnet. (At least that is what i read in the changelog) - a feature i can't wait to use for myself.

Thank you Sten,

Can you please explain this in more depth?

runet · Fri Mar 30, 2007 8:44 pm

Since you mentioned "proxy arp" I spent some time looking at the example at: http://www.mikrotik.com/testdocs/ros/2.9/ip/address.php

My questions are:

1. Does anyone know how to configure proxy-arp?

2. Will host C use 192.168.0.129 or 192.168.0.1 as a gateway

3. if host C uses 192.168.0.129 as a gateway, Wouldn't R1 get confused when trying to reach 192.168.0.1 for an outgoing connection? Would it broadcast on both interfaces ether1 and ether2?

JJCinAZ · Fri Mar 30, 2007 10:23 pm

Use the VLAN's per customer but don't give everyone a public IP unless they pay extra. If they just want basic Internet access, give them a private IP and NAT them out. If they want a public IP, subnet one to them. Yes, it costs you 4 IP addresses to sell 1, or 8 IP addresses to sell 5, but at $2 to $5 bucks a month per IP address, you can easily make money on this instead of giving it away.

If you want to put everyone into the same interface, you can use the SM and AP isolation features on the Canopy v8 software and you can still control bandwidth with the Mikrotik queues.

You have to ask yourself what scales best and after doing many things and buying up networks from people who tried many of the things you and other have mentioned, I can tell you that none of those methods scales well. Doing it right from the start, in my opinion, is to use a seperate VLAN per customer and use IP subnetting.

runet · Fri Mar 30, 2007 11:12 pm

Doing it right from the start, in my opinion, is to use a separate VLAN per customer and use IP subnetting.

I like the vlans too, for many reasons. I am not using v8 yet on the canopy equipment, but I will look into that too.

Might there be a way to use vlans which are separt L2 broadcast space, then echo ARP on all vlans. Then, assign addresses from the same /24 say to all devices connected to the other ends of the vlans. I thought this was called a bridge, but I am new to this.

If I have to, I can follow your advice and give out private addresses and assign public ones only to people that pay. It would just be nice to not have to put my customers through a double NAT.

JJCinAZ · Fri Mar 30, 2007 11:58 pm

We put customers through double-NAT all the time with no problems. If it really caused problems, it would be increasing our support costs and I'd get rid of it. I checked with our support people just now and, in fact, we have never had a single problem caused by double NAT, e.g. where the customer plugs in a Linksys/Dlink/Netgear router into the connection natt'ed by our Mikrotiks.

Yes, you should be able to bridge together a bunch of VLAN's and it should just look like a switch (which is simply a bridge in itself). There are details you have left out however, like how you connect to your upstream and whether or not you want to bridge the customers to your Internet feed or route them to your Internet feed.

changeip · Sat Mar 31, 2007 1:01 am

How about using rfc1918 space between you and the customer. Assign a /30 to each vlan, you get 1 IP they get 1 IP. You then setup a /32 route for their public IP and the next hop is their private IP out of the /30. This is basically the way cable networks are doing it, I believe. I'm wondering if you would need the interface routing in 3.0 to do this right though ...

Sam

JJCinAZ · Sun Apr 01, 2007 7:28 pm

How would that work? You would need to NAT the customers rfc1918 address to the public IP, else the upstream sees packets from a private address and should throw them out.

sten · Sun Apr 01, 2007 8:09 pm

imagine a clean router and your public ip subnet is 123.0.0.0/24

/ int bridge add name=loopback

/ ip address add address=123.0.0.1/32 interface=loopback

/ int vlan add name=vlan2 vlan-id=2 interface=ether2 arp=proxy-arp
/ int vlan add name=vlan3 vlan-id=3 interface=ether2 arp=proxy-arp
/ int vlan add name=vlan4 vlan-id=4 interface=ether2 arp=proxy-arp

/ ip address add address=10.0.2.1/30 interface=vlan2
/ ip address add address=10.0.3.1/30 interface=vlan3
/ ip address add address=10.0.4.1/30 interface=vlan4

/ ip route add dst-address=123.0.0.2/32 gateway=10.0.2.1
/ ip route add dst-address=123.0.0.3/32 gateway=10.0.3.1
/ ip route add dst-address=123.0.0.4/32 gateway=10.0.4.1

Now connect your computer on vlan2 and set ip address 123.0.0.2/24 with default gateway 123.0.0.1.
Connect additional test computer on vlan3 and set ip address 123.0.0.3/24 with default gateway 123.0.0.1

Does this work for you?

IntraLink · Sun Apr 01, 2007 8:32 pm

Wouldn't this be a simple solutions for your needs if you added a VLAN switch between the MT and the network, then tag/untagged the packets on the MT port?

So MT doesn't use VLAN, you just untag the ports coming from the VLAN side of the network.

So you could sort of "funnel" the VLANs to the MT port on the switch and it really shouldn't be replicating the broadcast traffic except to funnel to the MT port.

Does this make sense?

JJCinAZ · Mon Apr 02, 2007 3:14 am

As to sten's idea... The clean router has no IP address on an interface to the public facing Internet. If the upstream ISP as allocated a /24 to him, then his router must have an IP address in that range and I would expect the upstream ISP router to have the address 123.0.0.1/24, for example. Further, if a packet arrives at runet's router from a public address, e.g. 66.181.33.45 -> 123.0.0.2, then runet's routing table wants to route it to 10.0.2.1. When runet's router tries to arp out on vlan 2 for the host with that IP, none will be found.

Are you assuming something else here or am I just missing it?

sten · Mon Apr 02, 2007 3:38 am

of course proper configuration is absolutely necessary. i was saying "assume a clean router" to point out that this example shouldn't be superimposed on just any established configuration without some consideration.
basically i was giving a minimal demonstration of concept.

JJCinAZ · Mon Apr 02, 2007 6:05 am

I still don't see how the router will find a host with IP 10.0.2.1 in the example. The host connected on the end of VLAN2 will have address 123.0.0.2. Runet's router will not be able to pass on inbound packets for 123.0.0.2.

sten · Mon Apr 02, 2007 3:28 pm

My example shows how to use interfaces as gateways for routes.
Except in routeros 2.9 you cannot use an interface name as a "gateway" for the routes, you need to add an IP address and then use that IP address as the identifier for the interface.

In routeros 3.0 this whole setup would probably be much simpler as it would be something like this (not verified):


 / int bridge add name=loopback

/ ip address add address=123.0.0.1/32 interface=loopback

/ int vlan add name=vlan2 vlan-id=2 interface=ether2 arp=proxy-arp
/ int vlan add name=vlan3 vlan-id=3 interface=ether2 arp=proxy-arp
/ int vlan add name=vlan4 vlan-id=4 interface=ether2 arp=proxy-arp

/ ip route add dst-address=123.0.0.2/32 gateway=vlan2
/ ip route add dst-address=123.0.0.3/32 gateway=vlan3
/ ip route add dst-address=123.0.0.4/32 gateway=vlan4

The whole concept is usually also referred to as "unnumbered gateways" or "unnumbered routes".

runet · Mon Apr 02, 2007 4:47 pm

imagine a clean router and your public ip subnet is 123.0.0.0/24

/ int bridge add name=loopback

/ ip address add address=123.0.0.1/32 interface=loopback

/ int vlan add name=vlan2 vlan-id=2 interface=ether2 arp=proxy-arp
/ int vlan add name=vlan3 vlan-id=3 interface=ether2 arp=proxy-arp
/ int vlan add name=vlan4 vlan-id=4 interface=ether2 arp=proxy-arp

/ ip address add address=10.0.2.1/30 interface=vlan2
/ ip address add address=10.0.3.1/30 interface=vlan3
/ ip address add address=10.0.4.1/30 interface=vlan4

/ ip route add dst-address=123.0.0.2/32 gateway=10.0.2.1
/ ip route add dst-address=123.0.0.3/32 gateway=10.0.3.1
/ ip route add dst-address=123.0.0.4/32 gateway=10.0.4.1

Now connect your computer on vlan2 and set ip address 123.0.0.2/24 with default gateway 123.0.0.1.
Connect additional test computer on vlan3 and set ip address 123.0.0.3/24 with default gateway 123.0.0.1

Sten, did you actually try this and it worked?

Is it the proxy-arp that makes it work?

My question is, why does the loopback device have a mask of /32 and what would happen if is were to be /24?

sten · Mon Apr 02, 2007 6:22 pm

Sten, did you actually try this and it worked?

Slightly different setup.

Is it the proxy-arp that makes it work?

Not really.

My question is, why does the loopback device have a mask of /32 and what would happen if is were to be /24?

No particular difference in effect for what you are trying to achieve, but slightly different side effects.

runet · Mon Apr 02, 2007 7:48 pm

So if you were to change:

/ int vlan add name=vlan2 vlan-id=2 interface=ether2 arp=proxy-arp

To:

/ int vlan add name=vlan2 vlan-id=2 interface=ether2 arp=enabled

Would it still work?

sten · Mon Apr 02, 2007 8:05 pm

define "it".

surfe? sure
connect to your neighbor? not without host subnet mask change.

perhaps you'd want to hire someone with basic ipv4 knowledge to verify this for you?

runet · Mon Apr 02, 2007 8:53 pm

Thank you all for your comments!

JJCinAZ, I set up a system with a double NAT and it seems to work fine.

I like Sten's solution as well, I will try it on the work bench. By "it", I mean that traffic from the outside will forward to 123.0.0.2, and 123.0.0.2 will be able to see 123.0.0.3.

sten · Mon Apr 02, 2007 9:43 pm

you do not need proxy-arp to "surf the internet" but it might come handy to reach neighboring vlans. proxy-arp can be avoided if you use /32 net masks on the hosts.

MikroTik VLAN Bridging Help

MikroTik VLAN Bridging Help

bug?

Conclusion

Who is online