EoIP VLAN routing issues between 2 sites

[seansco]

EoIP VLAN routing issues between 2 sites

Our company is moving to a new building. I the process we want to be able to have both buildings linked on the same network (192.168.88.0/21) each subnet is on it’s own vlan (eg. 192.168.90.0/24 -vlan 1, 192.168.91.0/24 - vlan 2). All the vlans can talk to each other via routing on the main switch (the routerboards ARE NOT the main switch or main router at either location).

I have joined the 2 offices together with a routerboard and each location (RB1100AHx2) using an EoIP tunnel which is connected to the trunk ports of the main switch AT BOTH locations. The default gateway for the pcs is the main switch at their location. (eg. Location 1 – the default gateway for 192.168.90.0/24 is 192.168.90.1 / Location 2 the default gateway for the 192.168.90.0/24 is 192.168.90.254) and etc. for each vlan subnet.

The domain/DNS/AD servers are at location 1.

The link is up and works as expected for the most part.

Here is the problem I am having. From location 1 I cannot ping anything on a different subnet of the other location OTHER THAN the gateway of the other locations. (eg. Location 1 ip 192.168.90.73 can only ping 192.168.91.254 at location 2. Nothing else on that subnet. It can ping the other pcs at location 2 on the (same) 192.168.90.0/24 subnet. BUT if I change the gateway of a pc at location 2 to locations 1’s gateway then I can ping it from location 1.

I do not want the pcs at location 2 to have their gateway location 1’s gateway because I want location 2’s pcs to use Location 2’s internet connect for internet and if I have their gateway set to location 1’s gateway their internet traffic is routed through location 1’s internet and all the traffic between pcs at location 2 have to be routed through the eoip tunnel to location 1 which is saturating the tunnel and slowing things down.

Basically I want all location 1 and Location 2’s traffic to not have to run through the tunnel and to use their own internet connection but pc at different locations still be able to communicate like they are if they are at the same location.

It seems to me it has to be some sort of routing issues because like I said changing to gateway of a pc at location 2 to location 1’s gateway make it work.

I’ve cross posted this in a few other places.

[sindy]

What you want is not possible without a modification of routing somewhere. Hosts at each site can ping the gateway device of the remote site because it is in their own subnet, but to get to the subnet(s) behind that remote gateway device, they must either have a corresponding route configured on themselves, or the two gateway routers must send ICMP redirects saying “use IP x.x.x.x as a gateway for this subnet” and the PCs must follow the advice.

If the hosts get their addresses using DHCP, it may be possible to send them the routing table using the “classless routing” DHCP option (121 for normal devices and 249 for Microsoft devices, with a slightly different syntax). Some older devices don’t understand either, though.

But none of the above can be achieved using your Mikrotiks given their role and position in your network. They are just an “electronic extension cable” between the two “main switches” of the sites.

[seansco]

Thanks for the reply @sindy

But none of the above can be achieved using your Mikrotiks given their role and position in your network. They are just an “electronic extension cable” between the two “main switches” of the sites.

This is all I was expecting from the Mikrotiks; just “electronic extension cable between the two “main switches” of the sites”.

If the hosts get their addresses using DHCP, it may be possible to send them the routing table using the “classless routing” DHCP option (121 for normal devices and 249 for Microsoft devices, with a slightly different syntax). Some older devices don’t understand either, though.

Most of the hosts are getting their addresses via DHCP but I’m not sure how I can use this to solve my problem.

What you want is not possible without a modification of routing somewhere. Hosts at each site can ping the gateway device of the remote site because it is in their own subnet, but to get to the subnet(s) behind that remote gateway device, they must either have a corresponding route configured on themselves, or the two gateway routers must send ICMP redirects saying “use IP x.x.x.x as a gateway for this subnet” and the PCs must follow the advice.

At each location the gateway device for the pcs at the location (a cisco switch at location 1 & an Untangle Server at the location 2) are handling the routing between subnets for their location and that is working.

Since the mikrotiks are basically linking the 2 gateway devices how do the gateway devices tell each other what addresses are “on their side” of the extension cable? wouldn’t ARP be handling that? Do need to enable RIP or something else like that?

[sindy]

Most of the hosts are getting their addresses via DHCP but I’m not sure how I can use this to solve my problem.

Easily. At each site there are a few subnets (maybe just one per site?). So you have to tell devices at site O(ld) which subnets on site N(ew) to access via N’s “site switch” and vice versa; the other subnets on the same site are accessible via the own “site switch” whose address you already provide using DHCP as the “default gateway” (or Router or how is that field called).

To pack the routing table into the Option 121 or Option 247, it is best to read the RFC for Option 121, but basically the first byte contains the mask length in bits (so 255.254.0.0 is expressed as 15, 255.255.255.240 as 28 etc.), the next one to four bytes contain the beginning of the subnet address while only bytes with non-0 mask are stated (so for mask length 0-7 there is just one byte, for mask length 8 to 15 bytes there are two etc.), and the last four bytes contain the IP address of the gateway for that subnet. One such sequence is used for each subnet; there is no “list of subnets routed through the same gateway” syntax.

The difference between the standard Option 121 and Microsoft’s proptietary Option 249 is that if the former is used, the DHCP client must ignore the “default gateway” field in the basic body of the DHCP response (which effectively means that the default route must be provided within the Option 121 itself), whereas Option 249 is used in addition to the “default gateway” so it must not contain the default route. DHCP client devices which support one of these Options never use the other one; older devices may not ask for any of them.

So an example of Option 121 contents which says that the default route goes via 192.168.90.1 and an exceptional route to 192.168.91.0/24 goes via 192.168.90.254:

0,0,192,168,90,1,24,192,168,91,192,168,90,254

For the DHCP server at Cisco, you place the following into the ip dhcp pool section:
option 121 hex 0000.c0a8.5a01.18c0.a85b.c0a8.5afe
To add that also as Option 249, you just remove the default route part:
option 249 hex 18c0.a85b.c0a8.5afe

Since the mikrotiks are basically linking the 2 gateway devices how do the gateway devices tell each other what addresses are “on their side” of the extension cable? wouldn’t ARP be handling that? Do need to enable RIP or something else like that?

Don’t ask me, look how those two “site switches” are configured. There may be some dynamic routing protocol between them or there may be static routes, better to find it in their configuration than to sniff the traffic between them (but if you prefer that approach, you can use the Mikrotiks to do that). But what is more important is whether sending the ICMP redirects is permitted at both. A router normally sends an ICMP redirect if the gateway for a given packet is in the same subnet from which the packet has arrived, and it says “in this subnet, you have a direct gateway to the destination, so use it and don’t bother me”. If you can make this work on both the Cisco and the Untangle, you should not need to fiddle with the DHCP servers.

Maybe you can make them forward the packets the sub-optimal way rather than send the ICMP redirect, I’m not familiar with the possibilities of either of them.

ARP would only handle that if you would set the network masks short enough on the end hosts and activate arp-proxy functionality on both “site switches”. That way, the end devices would think they are in a single large subnet (like 192.168.0.0/16) and send ARP requests for all destinations inside that large subnet; the “site switches” would respond with their own MAC address to requests for all destinations in subnets connected to them, and thus trick the end hosts into sending the real packets to them, and forward them to the actual destination.

[seansco]

Easily. At each site there are a few subnets (maybe just one per site?). So you have to tell devices at site O(ld) which subnets on site N(ew) to access via N’s “site switch” and vice versa; the other subnets on the same site are accessible via the own “site switch” whose address you already provide using DHCP as the “default gateway” (or Router or how is that field called).

All the subnets are at both locations. What I am using to assign the correct gateway for devices at the “other location” is a DHCP reservation and then Option 003 “Router”. Maybe try Option 121 and have both with the priority being the gateway at the same location as the device?

[sindy]

All the subnets are at both locations.

This is a bit I have missed in your OP, and it invalidates the explanation I gave before. If the gateway at each site can route between all subnets, the problem that machines in subnet A on one site cannot get a response from machines in subnet B on the other site while they do get a response from machines in subnet B on the same site must be an issue of stateful firewall and non-symmetric routing.

On the picture below, you can see the paths of the request and response between Machine 1 in subnet A on site Cisco and Machine 2 in subnet B on site Untangle. The difference is that whereas it is the Cisco that routes the request packet from subnet A to subnet B, it is the Untangle that routes the response packet from subnet B to subnet A.
,-------------,
| machine 2 |
‘-------------’
^|v
,----------------, ^|v ,------------------,
| .>>|>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>‘|’>>|>>. |
| ^ |--------------- subnet B --------±–| v |
| Cisco ^ | | v Untangle |
| ^ | | v |
| ^ |—±---------- subnet A ------------| v |
| ‘<<|<<.|.<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<|<<’ |
‘----------------’ ^|v ‘------------------’
^|v
^|v
,-------------,
| machine 1 |
‘-------------’
This is not a problem if both devices act only as routers, but if one or both routers act also as stateful firewall even when forwarding between the internal subnets, non-symmetric routing prevents the responses from being let through because the device routing the response has not seen the corresponding request and therefore doesn’t permit the response to pass.

If you do need to control the traffic between the subnets by means of a firewall, I’m afraid I have no suggestion which would satisfy your need that traffic between machines in different subnets of same site would not have to be looped through the remote site and at the same time would not hit this issue when the two machines are in different subnets at different sites.