Community discussions

MikroTik App
 
jonasx
just joined
Topic Author
Posts: 11
Joined: Mon Nov 08, 2021 11:43 pm

Slow performance EOIP

Thu Dec 02, 2021 4:16 am

Edit -

After more investigation, I've found that its just the tunnel itself that is much slower than I would expect so the below is just a symptom.



I have 2 hex S routers with an EOIP bridge between them over the internet. Doesn't run on top of a VPN or anything.

Both are on full GB fiber connections and both test at around 1Gbps up/down with a speed test, same ISP and pings between them are 2ms. The bandwidth between the two LANs is close to ~500Mbps/TCP and ~950Mbps UPD which if I understand correctly is about as much as I'm going to get with IPSEC on the tunnel. Which, as an aside, I would love to get rid of, but I can't get the tunnel to work without that enabled.

The problem I'm having occurs when I try to use the foreign gateway for internet access. That is if I use LAN 2 gateway on a LAN1 device the speed tests at about 24Mbps down and 40Mbs up. Same result if I use LAN1 Gateway for a device on LAN2 - which even accounting for the IPSEC overhead is about 1/20th the speed i would expect.

MTUs on the EOIP are set at 1500

I'm not doing anything fancy, but I'm obviously doing something wrong. Any ideas on where to investigate?

I can provide configs and diagrams if needed.

Thanks!
Last edited by jonasx on Fri Dec 03, 2021 12:04 am, edited 2 times in total.
 
User avatar
nichky
Forum Guru
Forum Guru
Posts: 1275
Joined: Tue Jun 23, 2015 2:35 pm

Re: Slow Internet performance EIOP using 'sister' gateway for internet

Thu Dec 02, 2021 4:36 am

Diagram will do.
So much we can say here, bet let start with the MTU it self on the EoIP.
while its on 1500 can you sniff the traffic.
Once you done that then decrease to 1200 and start over.
And post everything here.
 
jonasx
just joined
Topic Author
Posts: 11
Joined: Mon Nov 08, 2021 11:43 pm

Re: Slow Internet performance EIOP using 'sister' gateway for internet

Thu Dec 02, 2021 4:56 am

Thanks for the quick reply...

I'm a network beginner..i think i Can put together a basic diagram of what I have but may need some help providing traffic logs, can you point me to some docs on how to capture and provide that?
 
User avatar
nichky
Forum Guru
Forum Guru
Posts: 1275
Joined: Tue Jun 23, 2015 2:35 pm

Re: Slow Internet performance EIOP using 'sister' gateway for internet

Thu Dec 02, 2021 5:23 am

if you begins just do this:
Decrease the MTU on the EoIP to 1200 and run the test , then compare with the test results that you have done before (mtu 1500)
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2985
Joined: Mon Apr 08, 2019 1:16 am

Re: Slow Internet performance EIOP using 'sister' gateway for internet

Thu Dec 02, 2021 11:36 am

FYI only.

Heavy EoIP overhead !

"VPLS tunnel is about 60% faster and less overhead than EoIP tunnel"

https://wiki.mikrotik.com/wiki/Transpar ... using_MPLS
 
jonasx
just joined
Topic Author
Posts: 11
Joined: Mon Nov 08, 2021 11:43 pm

Re: Slow Internet performance EIOP using 'sister' gateway for internet

Thu Dec 02, 2021 1:58 pm

Decrease the MTU on the EoIP to 1200 and run the test

That seems to have increased the speedtest quite a bit ~100Mbps Up/down now. However things still seem really sluggish on the client. Web pages load very slowly. Much slower than I would expect on a 100Mbps connection

Heavy EoIP overhead !
"VPLS tunnel is about 60% faster and less overhead than EoIP tunnel"
https://wiki.mikrotik.com/wiki/Transpar ... using_MPLS

Thanks I'll look into that a well.
 
tdw
Forum Guru
Forum Guru
Posts: 1843
Joined: Sat May 05, 2018 11:55 am

Re: Slow Internet performance EIOP using 'sister' gateway for internet

Thu Dec 02, 2021 2:11 pm

Running EoIP with an MTU less than 1500 often breaks things, e.g. slow of failed web page loading, as they expect it to be 1500 per the ethernet standards.

Do you really need layer 2 / ethernet connectivity? To avoid fragmentation of the encapsulated packets a common approach is to use layer 3 tunnels, e.g. IPIP or GRE, and MSS clamping to avoid fragmentation on TCP traffic.
 
jonasx
just joined
Topic Author
Posts: 11
Joined: Mon Nov 08, 2021 11:43 pm

Re: Slow Internet performance EIOP using 'sister' gateway for internet

Thu Dec 02, 2021 2:34 pm

I didn't mention that i was experiencing slow page loads as well when it was set to 1500.

I need broadcast to work between the 2 networks. (DHCP, some device discovery). I originally has an straight l2tp/ipsec based VPN between the 2 sites but was unable to get things to work and started getting in way over my head with tweaks , so I thought I'd give the EoIP thing a shot since it seemed pretty easy to set up.

I don't have a preference on how the 2 networks get connected, I just need to have the devices on both to behave as if they are on the same 'switch', and I need the option to route internet bound traffic from one segment out the gateway of the other. All with only a very basic networking knowledge that I have.

Thanks
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2985
Joined: Mon Apr 08, 2019 1:16 am

Re: Slow Internet performance EIOP using 'sister' gateway for internet

Thu Dec 02, 2021 2:57 pm

https://en.wikipedia.org/wiki/Maximum_transmission_unit

Where is the time that TCP/IP automatically reduced the packet size to 576bytes when a routing step was involved?

1500 as MTU is for ethernet. Most other connections are smaller (even due to protocol overhead if encapsulation is used). To avoid fragmentation (big part and a small overflow part) TCP does MTU path discover and sets the max MSS size as to avoid fragmentation. Most routers/firewall/gateway do help that process by overwriting the max MSS size in the packets of the TCP session handshake with a fixed value set in the config. UDP has no discover mechanism. (Kerberos via UDP will fail over satellite with a long OU membership list in Windows)

When using encapsulated channels (IPsec, IPsec NATtraversal, PPPoE, PPPoA, xDSL line, satellite links, PPP, SLIP, ....) you learn to tweak and tune the MSS and/or MTU size.
On a local link (ethernet) the effect of a smaller MTU size should not be noticable. (Somewhat more packets, but no fragmentation)
However always be carefull with the TCP Congestion avoidance protocols. Double congestion control or retransmit systems can make it all turtle slow. (TCP in TCP, TCP in wifi, ...)

PS:
I'm old enough in this to once have had to reduce the MTU size of a Unix server ethernet adaptor to 1400, to avoid TCP ACK delays on uneven packets. Speed gain by this was 100 fold. (Was a 4Kbyte block protocol issue!)
 
jonasx
just joined
Topic Author
Posts: 11
Joined: Mon Nov 08, 2021 11:43 pm

Re: Slow Internet performance EIOP using 'sister' gateway for internet

Thu Dec 02, 2021 11:55 pm

Was able to do a bit more testing with the set up and found that the whole tunnel is just slow, not just using the gateway at the remote site.

Previously I had tested the speed router to router with the built in tool and got speeds close to what I would expect. Just doing a straight file copy for a rough test yielded about 35-40Mbps.

I did mess with the MTUs quite a bit and experimented but was never really able to get anywhere near a tenth of the 'wire speed'.

IS Eoip just that slow or more likely do I have something mis-configured?

I watched several howto videos and believe I have things set up correctly, summary of the config as follows, (if there's a better way to dumps this info please let me know and I will do so) -


Site A -

ETH 1 - Public IP 1.2.3.4
ETH 2 - Private LAN 192.168.10.1 - Connected to local switch
eoip-tunnel to Site B
eoip-bridge containing ETH2 and EoIP Tunnel Interface

Site B -

ETH 1 - Public IP 5.6.7.8
ETH 2 - Private LAN 192.168.10.5 - Connected to local switch
eoip-tunnel to Site A
eoip-bridge containing ETH2 and EoIP Tunnel Interface
 
tdw
Forum Guru
Forum Guru
Posts: 1843
Joined: Sat May 05, 2018 11:55 am

Re: Slow performance EOIP

Fri Dec 03, 2021 12:52 am

Post the configs from /export hide-sensitive in a terminal window after redacting any public IPs, etc. in a code block (the [] icon in the row above the text box when posting on the forum)
 
jonasx
just joined
Topic Author
Posts: 11
Joined: Mon Nov 08, 2021 11:43 pm

Re: Slow performance EOIP

Fri Dec 03, 2021 1:22 am

Site 1 - Home
# dec/02/2021 17:10:40 by RouterOS 6.49.1
# software id = SG5Z-256W
#
# model = RB760iGS
# serial number = #############
/interface bridge
add name=eoip-bridge
/interface eoip
add allow-fast-path=no local-address=1.2.3.4 mac-address=\
    MA:CM:AC:MA:CM:AC mtu=1472 name=eoip-tunnel-to-cabin remote-address=\
    5.6.7.8 tunnel-id=69
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool0 ranges=192.168.10.200-192.168.10.254
/ip dhcp-server
add address-pool=dhcp_pool0 disabled=no interface=eoip-bridge name=dhcp1
/interface bridge port
add bridge=eoip-bridge interface=ether2
add bridge=eoip-bridge interface=eoip-tunnel-to-cabin
/interface detect-internet
set detect-interface-list=dynamic
/interface list member
add interface=ether1 list=WAN
add list=LAN
/ip address
add address=192.168.10.1/24 interface=ether2 network=192.168.10.0
/ip dhcp-client
add disabled=no interface=ether1
/ip dhcp-server network
add address=192.168.10.0/24 dns-server=1.1.1.1 gateway=192.168.10.1
/ip firewall nat
add action=masquerade chain=srcnat
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=America/Chicago

Site 2 - Remote
# dec/02/2021 17:05:34 by RouterOS 6.49.1
# software id = YU90-TMNZ
#
# model = RB760iGS
# serial number = #############
/interface bridge
add name=eoip-bridge
/interface eoip
add allow-fast-path=no local-address=5.6.7.8 mac-address=\
    MA:CM:AC:MA:CM:AC mtu=1472 name=eoip-tunnel-to-Home remote-address=\
    1.2.3.4 tunnel-id=69
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/interface bridge port
add bridge=eoip-bridge interface=ether2
add bridge=eoip-bridge interface=eoip-tunnel-to-Home
/ip neighbor discovery-settings
set discover-interface-list=all
/interface l2tp-server server
set enabled=yes use-ipsec=yes
/interface list member
add interface=ether1 list=WAN
add list=LAN
add interface=eoip-bridge list=LAN
/ip address
add address=192.168.10.5/24 interface=eoip-bridge network=192.168.10.0
/ip dhcp-client
add disabled=no interface=ether1
/ip firewall nat
add action=masquerade chain=srcnat
/ip service
set telnet address=1.2.3.4/32
set ftp disabled=yes
set www address=1.2.3.4/32
set ssh disabled=yes
set www-ssl address=1.2.3.4/32 disabled=no
set api disabled=yes
set winbox address=1.2.3.4/32
set api-ssl disabled=yes
/system clock
set time-zone-name=America/Chicago

The MTUs are still at 1472 due to some earlier experimenting. Thanks for taking a look
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2985
Joined: Mon Apr 08, 2019 1:16 am

Re: Slow performance EOIP

Fri Dec 03, 2021 2:35 am

OK now we see the config. Looks OK.

But still consider what might be happening. For the clients on the ethernet sides the MTU is 1500, and for the EoIP it is 1472. This 1472 will avoid unwanted packet split or drop because of the tunnel real MTU size, only if the sender is aware of this limitation (MTU size discover). You need sniffer data to see what really happens. But here it is not excluded that the client device is still convinced it can send 1500byte sized packets, which will be fragmented by the MT before it enters the EoIP tunnel. The MT itself does not make that mistake with B-test as it knowns the MTU size of the EoIP tunnel. Sniffer will tell if you have packet split or not. Wireshark can analyze the content of the session handshake. IF you have packet split, then either the MSS mangle rule should be added, or the MTU size of the ethernet should be in line with EoIP.

Now about your usage/test setup protocol used. What are you using? TCP? UDP? What application protocol?
For TCP what are your settings? The "congestion avoidance algoritms" are a whole world on their own, but they have a major impact on the performance and certainly through a tunnel.
https://en.wikipedia.org/wiki/TCP_congestion_control

For experimenting and tuning I only have found some documents for Windows. Linux is advanced in this.

https://www.ontimetech.com/blog/is-wind ... computers/
https://www.thewindowsclub.com/window-a ... windows-10
https://docs.microsoft.com/en-us/window ... uning-nics
 
jonasx
just joined
Topic Author
Posts: 11
Joined: Mon Nov 08, 2021 11:43 pm

Re: Slow performance EOIP

Fri Dec 03, 2021 3:42 am

Are you suggesting that I should change every interface in the whole chain to the same MTU? Which I guess at this point is an unknown good value?

Oh and sorry to answer more of your questions -

Any protocol, i really just want a seamless joining of two LANs over the internet as if they were plugged into the same switch, which is what i understood Eoip would provide. If you can provide me a way to check the settings you asked for I will be happy to provide.

For testing I am just using speedtest.net...here's a comparison for the local gateway, during off peak hours i get round 900/850, this time of day this result is pretty typical

Image

vs the remote gateway over the eoip tunnel.

Image

Those results would be similar if I ran them from the remote location as well.

the LAN to LAN test I did earlier was just an SMB file copy form a NAS device on my home LAN to the Cabin LAN, results are about about the same as the above screen shots...maybe a bit faster
 
User avatar
bpwl
Forum Guru
Forum Guru
Posts: 2985
Joined: Mon Apr 08, 2019 1:16 am

Re: Slow performance EOIP

Fri Dec 03, 2021 10:58 am

Well, I think I would start with the Sniffer. Looking at what is transmitted, in what packets, and also looking closely to the timing. With 50 times slower there should be some visible delay. Is it the ACK? Is it a burst and then wait? What was negotiated in the TCP session setup?

Maybe setting "Clamp TCP MSS , Allow Fastpath, Don't Fragment" are the first things to check on influence.
Also keep an eye on the CPU. EoIP is intense (IPSec?) on CPU.

Just an extra question: you really need full L2 connectivity? (Passing all broadcasts and multicasts).
SSTP is also slow, PPTP is faster. Don't know about L2TP. IPIP, OVPN and other.
 
jonasx
just joined
Topic Author
Posts: 11
Joined: Mon Nov 08, 2021 11:43 pm

Re: Slow performance EOIP

Fri Dec 03, 2021 3:34 pm

Thanks. I'd really rather not up load a full wireshark capture to a public forum, any filters that would be useful for me to apply or anything in particular I should focus on? I'm a wire shark novice, but just comparing the 2 captures i did (one with local gateway and one with remote ), i don't see anything that stands out. Of course maybe I'm not looking at the right places.

Clamp TCP MSS is enabled , disabling has no affect on the performance, I've tried don't fragment - set to "no" and "inherit", no difference that I can tell.

I would like to try Allow Fast Path but I can't enable that with ipsec applied. ipsec isn't a requirement for me but I've not been able to get the tunnel to connect without it turned on for some reason.

I would like L2 for DHCP, i can work around that by setting statics on the remote side or running an independent server there. However I do have a device on the Home network that I believe relies on broadcast to be discovered. If there's a relatively simple way to work around that with another tunneling method I'm certainly not married to the EoIP solution.
 
felix84
just joined
Posts: 9
Joined: Thu Feb 09, 2017 4:13 pm

Re: Slow performance EOIP

Sat Jan 01, 2022 7:35 pm

Hi, do you have any progress? We have the same issue with EOIP over 1Gb wan link. But in our case the latency is 35ms. Also tried everything, but no luck. Iperf show 37Mbit/sec. Real speed even slower (depending on the protocol)
 
gotsprings
Forum Guru
Forum Guru
Posts: 2103
Joined: Mon May 14, 2012 9:30 pm

Re: Slow performance EOIP

Sun Jan 02, 2022 2:55 pm

When I suddenly couldn't go to the office...

I set up an EoIP tunnel from my house to the office.
I had to make a few changes from default and write a script to update my LOCAL IP address in EoIP Everytime I got a new DHCP lease from each provider. (For the IPSec encryption)

Office is 1000/25
Home is 1000/1000

My speed tests are pretty consistent at 21/13 when I am on the EoIP network.

At the office the EoIP tunnel is bridged to the main bridge.
At home the tunnel is bridged to a VLAN only.
Added VLAN to my switch and wireless.

Connect to SSID of Office. I am l2 there.
Connect to SSID home. Everything stays local.
 
jonasx
just joined
Topic Author
Posts: 11
Joined: Mon Nov 08, 2021 11:43 pm

Re: Slow performance EOIP

Tue Jan 11, 2022 9:33 pm

I've not really made any meaningful progress. I ended up doing a site to site pptp tunnel then creating a eiop tunnel over that connection. The speed tests a little faster and it just seems more 'stable' that the eoip tunnel alone.

It's still about 1/20th of the wire speed, which is frustrating and not fast enough for me to stream my movie library. I'll continue to mess with it but don't want to break something that sort of works.
 
User avatar
JohnTRIVOLTA
Member
Member
Posts: 345
Joined: Sun Dec 25, 2016 2:05 pm
Location: BG/Sofia

Re: Slow performance EOIP

Tue Jan 11, 2022 11:33 pm

.....
Office is 1000/25
Home is 1000/1000
....
Your problem is in the bold number or 25mb/ps up - the office link!
 
gotsprings
Forum Guru
Forum Guru
Posts: 2103
Joined: Mon May 14, 2012 9:30 pm

Re: Slow performance EOIP

Wed Jan 12, 2022 12:18 am

.....
Office is 1000/25
Home is 1000/1000
....
Your problem is in the bold number or 25mb/ps up - the office link!
I know that.

Who is online

Users browsing this forum: Google [Bot], RogerWilco and 14 guests