Dear Gentle Folk,

I have a central private NOC with about 30 servers in it, all with IP's on various subnets local to the NOC, but real IP addresses.

Periodically a server breaks and I take it home to work on it, but it has to stay up and running and it has to stay
on the same IP.

I use EOIP to share my home network IP space with the NOC IP space, so this works well. I plug in the server at home and everything works.

EOIP is however slow during night time rush hour, and starts to lose pings. It can get pretty bad even though
the traffic on the EOIP its self is minimal. Owners of the server notice this and complain during the time the server is in 'sick bay'.

Is there a better way to do this?

What is the standard way of sharing an IP subnet across two separated miks?

Running RB1100x with latest firmware.

Thanks in advance,

Homer W. Smith
CEO Lightlink Internet

You first need to investigate why "EOIP is slow".
Is your home internet connection slow? Is the ethernet segment heavily loaded with broadcast/multicast traffic? Are the routers overloaded with EOIP encap/decap?

Depending on what is the outcome you maybe can improve things by applying some filtering before the traffic enters the tunnel (bridge filters), or you could try using a PPP-based tunnel instead and not bridge the traffic but rather route the single IP address via the tunnel. That can be done using proxy-ARP on the NOC site and connecting the single address from home.

MTU could also be an issue here

How are you using EoIP? Is it just native EoIP or is it running inside another tunnel i.e. PPTP or L2TP?
If the latter, absolutely only use L2TP as thats the only UDP based VPN that MikroTik supports at this point in time

Either way i'd manually set the MTU to 1500 on the EoIP interface as that avoids issues with bridging ethernet interfaces to it (lowers the MTU of every interface in the bridge). If using another VPN you may need to set it lower but never less than half of total packet size to prevent unnecessary fragmentation, i.e. 1000
Maybe i'm wrong but I feel if you are going to fragment a 'slightly larger' packet i.e. 1530 bytes, it's better to split it more evenly and send i.e. a 1000 and a 530 byte packet (not counting further overhead) than a 1500 and a 30 byte packet

This can help packet re-ordering and retransmission issues. Which can be further compounded depending on the QoS scheme in the transit network. Very small packets are sometimes given higher priority

You first need to investigate why "EOIP is slow".
Is your home internet connection slow? Is the ethernet segment heavily loaded with broadcast/multicast traffic? Are the routers overloaded with EOIP encap/decap?

Depending on what is the outcome you maybe can improve things by applying some filtering before the traffic enters the tunnel (bridge filters), or you could try using a PPP-based tunnel instead and not bridge the traffic but rather route the single IP address via the tunnel. That can be done using proxy-ARP on the NOC site and connecting the single address from home.

The home connection is on the same physical wireless connection to our NOC as all our customers are on, by 3 bridged hops from home to NOC. over an ubnt AirOS 8 5.7G link.

Internet -> NOC core mik -> NOC mik2 -> Rocket AC -> Rocket AC -> Rocket AC -> home mik1 -> (bomded link)-> home mik2

Maybe 6 miles total, speeds at night are down in the 3 to 5 meg range between NOC and home over the EOIP, as the radio links are relatively full with other customers branching off left and right at each rocket.

There is very little traffic on the EOIP, even cross traffic is small at the NOC itself between all the servers there and the net which also travels over the EOIP on the local lan only to the core NOC mik -> Internet. Server -> NOC Core mik -> Internet.

I do not know how to measure encap/decap speeds, can you show me that?

Total EOIP traffic at night is 100kbps to 300lbps as shown by torch.

I have always wondered about routing single IP's. across tunnels. You mention PPP tunnels, is there a preference for which kind of tunnle i use? IPSEC, L2, PPTP etc? I am pretty ignorant of performance differences.

Thanks for your time.

MTU could also be an issue here.

How are you using EoIP? Is it just native EoIP or is it running inside another tunnel i.e. PPTP or L2TP?
If the latter, absolutely only use L2TP as thats the only UDP based VPN that MikroTik supports at this point in time

Either way i'd manually set the MTU to 1500 on the EoIP interface as that avoids issues with bridging ethernet interfaces to it (lowers the MTU of every interface in the bridge). If using another VPN you may need to set it lower but never less than half of total packet size to prevent unnecessary fragmentation, i.e. 1000
Maybe i'm wrong but I feel if you are going to fragment a 'slightly larger' packet i.e. 1530 bytes, it's better to split it more evenly and send i.e. a 1000 and a 530 byte packet (not counting further overhead) than a 1500 and a 30 byte packet

This can help packet re-ordering and retransmission issues. Which can be further compounded depending on the QoS scheme in the transit network. Very small packets are sometimes given higher priority

MTU is set manually to 1500 at both ends.

Native EOIP not running inside any other tunnel.

Thanks for your time

Homer

When you have a direct transparent L2 link, you should use a VLAN instead of EoIP!
Make sure the WiFi devices are configured for point-to-point.

If your home network is connected via private network and does not go over the internet. Then your best bet is VPLS
This takes a bit more to setup but its not too bad. All routers between you and the destination need to be running MPLS, and you need to make sure your L2MTU on every device (every radio/switch/router) is big enough. A bare minimum of 1508 on MikroTik but many other vendors include the ethernet frame, some also include FCS, so just assume 1526 bare minimum. You can always set it higher it does not cause any harm
Airfiber devices are 9600 I believe, those are fine. MikroTik is also big enough by default, just check any switches

Then enable LDP on all routers in between

Then create VPLS interface just on the end-points, and bridge the VPLS interface. It will function like EoIP but its much faster

When you have a direct transparent L2 link, you should use a VLAN instead of EoIP!
Make sure the WiFi devices are configured for point-to-point.

I personally hate extended a VLAN any further than 1 router. It gets very messy very quickly as you need to do it on every switch/router if its not just a trunk port. Hard to clean up later on as well, it makes things confusing. But for OP if its just this 1 instance it might be worthwhile as it is simpler than a VPLS setup

When you have a direct transparent L2 link, you should use a VLAN instead of EoIP!
Make sure the WiFi devices are configured for point-to-point.

Thank you for your answer, in our case there are both routers, and point to multipoint wifi between our NOC and my home network.

Homer W. Smith
CEO Lightlink Internet

MTU could also be an issue here.

How are you using EoIP? Is it just native EoIP or is it running inside another tunnel i.e. PPTP or L2TP?
If the latter, absolutely only use L2TP as thats the only UDP based VPN that MikroTik supports at this point in time

Either way i'd manually set the MTU to 1500 on the EoIP interface as that avoids issues with bridging ethernet interfaces to it (lowers the MTU of every interface in the bridge). If using another VPN you may need to set it lower but never less than half of total packet size to prevent unnecessary fragmentation, i.e. 1000
Maybe i'm wrong but I feel if you are going to fragment a 'slightly larger' packet i.e. 1530 bytes, it's better to split it more evenly and send i.e. a 1000 and a 530 byte packet (not counting further overhead) than a 1500 and a 30 byte packet

This can help packet re-ordering and retransmission issues. Which can be further compounded depending on the QoS scheme in the transit network. Very small packets are sometimes given higher priority

MTU is set manually to 1500 at both ends.

Native EOIP not running inside any other tunnel.

Thanks for your time

Homer

Wow, if you set manually the MTU to 1500 at both ends, it means that each router is fragmenting packets because of the headers, what I'd do is to reduce the MTU of the tunnel to 1450 and setup a mangle action to change the TCP-MSS to 1400 at least, and also be sure that the entire path between both ends of the EoIP is working great, you can try the real MTU available between ends by using the ping with the extra command: size=1500 do-not-fragment and you'll see if it works, if you want to use an MTU of 1500 for the tunnel you'll need a higher L3MTU at your internet connection.

EoIP is going to fragment anyway, it's a Layer2 bridging protocol, not Layer3
MTU (which is a L3 MTU) shouldn't even really be used. It will carry all L2 traffic at an MTU up to ~65535 or whatever its set to
So if you put it in a bridge or any standard Layer2 segment with switches etc you're going to have any packets larger than about ~1416 bytes be fragmented anyway, regardless of what you set the MTU to

But what does happen when you leave the MTU alone and put the EoIP interface into a bridge with other ethernet interfaces, it screws the L3MTU on all of them because the bridge interface uses the lowest MTU of all interfaces in it. So it also screws LAN connectivity

If it was a L3 VPN like SSTP/IPSec/OpenVPN/PPTP etc i'd agree, set it lower. But with EoIP if its ever going to be bridged with other interfaces it should be manually set to 1500
If its not bridged, fair enough if you are using it for L3 purposes, but then I question why you want EoIP in the first place. Typically you use it to carry things like PPPoE. If you want to avoid fragmentation entirely you're going to need some pretty low MTU on the PPPoE interfaces and sometimes things break. I've found many sites not happy with an MTU lower than about 1460

The do-not-fragment bit does not affect EoIP traffic at all. It only affects it when the L3 MTU is set and artificially limiting it. But you can set it to say 6000 and use it across the internet, then try send a 6000 byte ping with do-not-fragment ticked and it will work fine. In the background it is still fragmenting the packet, but it happens at a lower layer (at Layer2) it just doesn't fail

To answer your first question, an alternative of EoIP could be an L2TP Tunnel with BCP... Search for it on the wiki, plenty of info there... as well as MUM Presentations about that...
Now, since you added the EoIP inside the Bridge the Bridge of the MTU itself will be automatically go bellow 1500B and get the value of the lowest MTU inside the Bridge...
That can cause problems so you may take a look on that...