Community discussions

MikroTik App
 
User avatar
halacs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 88
Joined: Thu Jul 06, 2017 5:45 pm
Location: Hungary

Debugging EoIP tunnel

Sun Jun 13, 2021 9:17 am

Hi,

I have an EoIP tunnel between a Mikrotik CCR1009-7G-1C-1S+PC (site A) and a RB4011iGS+RM (site B) router.
I have run out of the ideas what to check, why I experience low bandwidth: cca 28 Mbps instead of 1 Gbps.
Is there any idea what can be wrong? If needed I can attach config of both side.

About the config in a nutshell:

- SiteA has 1Gbps symmetric fiber connection. SiteB has 1Gbps / 25 Mbps coaxial. I want to see near to 1 Gbps file copy from site A to site B.
- I set EoIP tunnel as a VLAN trunk.
- IPSec passwrod is set on the EoIP tunnel.
- There is an additional NAT bridge to deal with dynamic IP addresses at both side.
- SiteA has a PPPOE Internet conenction with 1480 MTU (automatically set).
- MTU of EoIP tunnel set manually to 1300. If I let it set automatically, both end have different automatic MTU value which is definitely strange.

What I have checked so far:

- Per core CPU utilization on both site: while I'm copying a file from a samba server at site A to another Linux machine at site B, one CPU core reach 90-100% for 1-2 seconds then go back under 10%. I see this repeating continuously. Overall CPU utilization is always under 15%, however, I also have a user manager (radius server) on my site A router. I think this is fully okay, right?

- Set lower MTU on the hosts where I copy from and to (on the samba server and on the samba client Linux hosts): EoIP has 1300 MTU with 1250 MSS clamp so I have set 1250 MTU then I restarted smbd on the server host. Nothing changed to 1500 MTU was set back on both hosts. This means for me that not the MTU is the problem, or if it is even the problem then I can't see why this huge bandwidth loss just because of the MTU (how to calculate this loss based on the MTU value?).

- if I try samba copy from the same samba server but inside Site A NOT via EoIP tunnel, it is as expected: 640-720Mbps (80-90 MBps) which is quite near to 1 Gbps. This says for me that NOT the samba server itself is slow.

- I checked hardware documentations on mikrotik.com. In case of CCR1009-7G-1C-1S+PC, docs say that 133.5 Mbps (26,6 MBps) speed with the smallest 64 byte packet (?) length and AES-256-CBC + SHA256 on IPSec. 97.4 Mbps (12.175 MBps) was given for the same for the other router (RB4011iGS+RM). In reality, I experience 24-32Mbps (3-4MBps) speed which is anyway much slower (cca 70% drop), especially because I feel I should check speeds in the 512 byte column of the docs which is even faster.

- I have checked switch stats too. In case of CCR1009-7G-1C-1S+PC, "/interface ethernet switch print stats" has empty output. I guess it is because there is no dedicated switch ship in it. Right? In case of the other router, which is a RB4011iGS+RM router, I see something like this (not so meaningful for me):
[admin@siteB] > /interface ethernet switch print stats 
                name:         switch1        switch2
      driver-rx-byte:  14 521 336 835 23 698 153 721
    driver-rx-packet:      19 510 463     22 653 094
      driver-tx-byte:  26 127 861 597 13 200 576 097
    driver-tx-packet:      23 026 151     19 287 656
            rx-bytes:  14 404 274 057 23 562 247 761
           rx-packet:      19 510 463     22 653 368
        rx-too-short:               0              0
               rx-64:               0            274
           rx-65-127:       1 033 305      3 608 906
          rx-128-255:       8 540 998      1 088 897
          rx-256-511:         544 965        225 193
         rx-512-1023:         441 236        112 681
        rx-1024-1518:       8 234 017     17 423 493
         rx-1519-max:         715 942        193 924
         rx-too-long:               0              0
        rx-broadcast:             139         66 641
            rx-pause:               0            274
        rx-multicast:          91 742         56 665
        rx-fcs-error:               0              0
      rx-align-error:               0              0
         rx-fragment:               0              0
     rx-length-error:               0              0
           rx-jabber:               0              0
             rx-drop:               0              0
            tx-bytes:  25 805 507 167 12 930 548 913
           tx-packet:      23 026 405     19 287 656
        tx-broadcast:             112        309 172
            tx-pause:             254              0
        tx-multicast:           1 111        218 176

- I checked what is network throughput if I copy a file via apache2 on top of HTTPS with NO EoIP tunnel (so via the public IP). I experienced this way 160-240Mbps (20-30 MBps) throughput. It is still under the expected 1 Gbps, but I can accept this as this was even much higher sometimes (close to the the 1Gbps with 2 threads). I feel this is also okay.

- Then, finally I have checked bandwidth with the Mikrotik own bandwidth tool. Bandwidth server was run on the SiteA router, client on the SiteB router. If I set remote-udp-tx-size and local-udp-tx-size to 1250, I got a correct speed, but otherwise.
[admin@SiteB] > /tool bandwidth-test address=192.168.0.254 dire
ction=receive remote-udp-tx-size=1250 local-tx-speed=1250 protocol=udp
                status: running
              duration: 15s
            rx-current: 449.0Mbps
  rx-10-second-average: 445.8Mbps
      rx-total-average: 399.6Mbps
          lost-packets: 4908
           random-data: no
             direction: receive
               rx-size: 1250
      connection-count: 20
        local-cpu-load: 26%
       remote-cpu-load: 48%


[admin@SiteB] > /tool bandwidth-test address=192.168.0.254 dire
ction=receive protocol=udp                                            
                status: running
              duration: 6s
            rx-current: 0bps
  rx-10-second-average: 0bps
      rx-total-average: 0bps
          lost-packets: 0
           random-data: no
             direction: receive
               rx-size: 1500
      connection-count: 20
        local-cpu-load: 0%
       remote-cpu-load: 2%


[admin@SiteB] > /tool bandwidth-test address=192.168.0.254 dire
ction=receive remote-udp-tx-size=1250 local-tx-speed=1250 protocol=tcp
                status: running
              duration: 13s
            rx-current: 0bps
  rx-10-second-average: 0bps
      rx-total-average: 48bps
           random-data: no
             direction: receive
      connection-count: 20
        local-cpu-load: 0%
- I have checked ping if there is packet loss or bad round trip time, but there is no such an error when I ping samba server at Site A from samba client at Site B.
$ ping 192.168.0.6
PING 192.168.0.6 (192.168.0.6) 56(84) bytes of data.
64 bytes from 192.168.0.6: icmp_seq=1 ttl=64 time=17.8 ms
64 bytes from 192.168.0.6: icmp_seq=2 ttl=64 time=20.1 ms
64 bytes from 192.168.0.6: icmp_seq=3 ttl=64 time=25.6 ms
64 bytes from 192.168.0.6: icmp_seq=4 ttl=64 time=14.3 ms
64 bytes from 192.168.0.6: icmp_seq=5 ttl=64 time=17.3 ms
^C
--- 192.168.0.6 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 14.339/19.033/25.573/3.756 ms
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Debugging EoIP tunnel

Mon Jun 14, 2021 6:10 pm

All this world of word and no one line of config.

If you put all this effort into doing the complete export of both devices and putting them on the site, you were done sooner.
 
User avatar
halacs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 88
Joined: Thu Jul 06, 2017 5:45 pm
Location: Hungary

Re: Debugging EoIP tunnel

Mon Jun 14, 2021 7:36 pm

Config of Site A: https://gist.github.com/halacs/9d5ec999 ... 25f03bbea0

Config of Site B: https://gist.github.com/halacs/9de29479 ... 4c0f9fe6c5

Both after removing sensitive data.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: Debugging EoIP tunnel

Tue Jun 15, 2021 1:09 am

Both after removing sensitive data.
Have you tried EoIP without encryption, and/or IPsec by itself without EoIP, for comparison purposes?
 
User avatar
halacs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 88
Joined: Thu Jul 06, 2017 5:45 pm
Location: Hungary

Re: Debugging EoIP tunnel

Tue Jun 15, 2021 8:11 am

Have you tried EoIP without encryption, and/or IPsec by itself without EoIP, for comparison purposes?
I tried to remove IPsec secret from the EoIP interface (both side, of course) but the EoIP tunnel got disconnected and remained disconnected after 1-2 minutes too. I think I should not change my firewall rules just because of this change because GRE is also needed for IPsec.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: Debugging EoIP tunnel

Wed Jun 16, 2021 10:09 pm

I tried to remove IPsec secret from the EoIP interface (both side, of course) but the EoIP tunnel got disconnected and remained disconnected after 1-2 minutes too. I think I should not change my firewall rules just because of this change because GRE is also needed for IPsec.
You don't have to change the existing rules. You could make a new rule, input chain, protocol gre, source ip address = public IP of the far side, action allow. Then, move that rule to the top. Do that on both sides, and it should come up properly without IPsec. After you are finished with the testing you can disable or delete that rule.
 
User avatar
halacs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 88
Joined: Thu Jul 06, 2017 5:45 pm
Location: Hungary

Re: Debugging EoIP tunnel

Wed Jun 16, 2021 11:16 pm

I could try it without IPsec. The key was not the firewall rules because GRE was allowed originally but the source IP: previously I got the advice to add an extra NAT and define a private IP as source at both side to deal with dynamic IP addresses. When I removed both the IPsec secret and the source IP EoIP became ready with perfect performance: 1192 Mbps (149 MBps). But why that private IP could cause this? Why IPsec+private IP could work at all and how these can affect the performance?
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 11968
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:

Re: Debugging EoIP tunnel

Wed Jun 16, 2021 11:59 pm

Who suggest you to use NAT for EoIP???
adding naTranslation add work to CPU and the naTranslation are not hardware accellerated, IPsec can be if hardware support this
Last edited by rextended on Tue Jun 22, 2021 3:33 pm, edited 1 time in total.
 
User avatar
halacs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 88
Joined: Thu Jul 06, 2017 5:45 pm
Location: Hungary

Re: Debugging EoIP tunnel

Thu Jun 17, 2021 8:02 am

Someone in this forum :) but now I understand then why my tunnel is so slow.

What to write into the local address of the EoIP tunnel? I have dynamic IP on both side. Can I leave them empty with IPsec?
 
User avatar
halacs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 88
Joined: Thu Jul 06, 2017 5:45 pm
Location: Hungary

Re: Debugging EoIP tunnel

Thu Jun 17, 2021 8:32 am

Hmm, maybe the bandwidth it still not as expected: I noticed that even if my Ubuntu says cca 1Gbps speed, my Mikrotik routers (on EoIP interface) says only about 200-300 Mbps. Anyway this is much better then before but now I have no IPsec.

I disabled the bridge I created months/years ago just before the extra NAT for the tunnel to handle dynamic WAN addresses.

I tried to leave local IP field empty and set IPsec password at the same time, but this way tunnel doesn't get ready.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: Debugging EoIP tunnel

Tue Jun 22, 2021 12:03 am

I tried to leave local IP field empty and set IPsec password at the same time, but this way tunnel doesn't get ready.
You can probably write a script to handle the changes for the tunnel automatically in event of an IP change.
 
User avatar
halacs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 88
Joined: Thu Jul 06, 2017 5:45 pm
Location: Hungary

Re: Debugging EoIP tunnel

Tue Jun 22, 2021 7:09 pm

I am wondering why UDP can 4 times quicket (cca 400 Mbps) then TCP (90-100Mbps).

In case of UDP, packet size must be set to 1300 which is the MTU of the EoIP tunnel. For TCP, I have a TCP clamp rule in firewall: MSS is set to 1250.

As I read btest is resource heavy I use two additional Mikrotik router for btest client and server as normal host on my network.
 
User avatar
halacs
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 88
Joined: Thu Jul 06, 2017 5:45 pm
Location: Hungary

Re: Debugging EoIP tunnel

Tue Jun 22, 2021 7:10 pm

I tried to leave local IP field empty and set IPsec password at the same time, but this way tunnel doesn't get ready.
You can probably write a script to handle the changes for the tunnel automatically in event of an IP change.
So far it looks like empty source field is fine: changed WAN IP is not a problem. I don't remember however what fixed the issue I mentioned above with empty source field. I hope this wont change :) If it will do so I will write a script. Thanks for the advice!

Who is online

Users browsing this forum: FranMercedesG and 23 guests