Community discussions

MikroTik App
 
Alupis
just joined
Topic Author
Posts: 23
Joined: Wed Feb 29, 2012 6:30 pm

Very Slow output for traffic passing through CHR

Fri Apr 22, 2022 2:41 am

UPDATE IN BELOW POST - No longer think this is a GRE Tunnel issue

Basic Setup:

CHR P10 licensed & activated, hosted on Vultr cloud hosting provider running 7.2 Stable, 1000Mbps/1000Mbps
CCR hardware running 6.48.6 Long Term, 1000Mbps / 45Mbps

Topology:

Office Workstation (Private: 10.0.0.5/27)
|
V
CCR (Public: 1.1.1.1/29, Private: 10.0.0.1/27, Tunnel: 172.16.1.1/30)
|
V
TUNNEL (GRE, EOIP, or IPIP with IPsec)
|
V
CHR (DHCP assigned fake-static Public: 2.2.2.2/23, Private: 10.1.0.1/27, Tunnel: 172.16.1.2/30)
|
V
Virtual Server (Private: 10.1.0.5)

I initially created a GRE tunnel with IPsec Secret between the two endpoints. The Tunnel connects, and IPsec Active Peers show up as expected. After adding Routes, I can pass traffic as expected. Everything looked good until I attempted to pass actual workloads through the tunnel... and discovered they are extremely slow, less than 1Mbps and often in the Kbps range.

I disabled IPsec Secret and tried just a plain tunnel, same thing.

I tried the same configuration, but used EOIP and IPIP instead. Same thing with both.

If I do a BTest between both mikrotiks, I can achieve reasonable throughput (300-700Mbps+ down, depending if IPsec is enabled or not).

If I bypass the CHR and talk directly to a public IP on the cloud Virtual Server, I can achieve maximum throughput as well, which would indicate it's not the virtual server, perhaps.

The problem only occurs when passing traffic through the CHR either direction.

Sniffing the connection, I see a lot of retransmissions and duplicate packets - unsure if that is the problem or standard for a tunnel like this. Seems suspicious though.

I've attached cleaned/trimmed exports from each side. I tried to remove things that were obviously unrelated, such as our "road warrior" IKEv2 configs. If something doesn't make sense, let me know and I'll check the config.

CHR:
# apr/21/2022 17:25:19 by RouterOS 7.2
# software id = 
#
/interface ethernet
set [ find default-name=ether1 ] disable-running-check=no
set [ find default-name=ether2 ] comment=\
    "MTU 1450 per Vultr VPC Documentation" disable-running-check=no mtu=1450
/interface gre
add allow-fast-path=no mtu=1426 name=gre-tunnel2 remote-address=1.1.1.1
/disk
set sata1 disabled=no
/interface list
add name=WAN
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec profile
set [ find default=yes ] dh-group=modp1024 enc-algorithm=aes-128
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc lifetime=8h pfs-group=\
    modp4096
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=LAN
add interface=gre-tunnel2 list=LAN
add list=LAN
/interface ovpn-server server
set auth=sha1,md5
/ip address
add address=10.1.0.1/27 interface=ether2 network=10.1.0.0
add address=172.16.1.2/30 interface=gre-tunnel2 network=172.16.1.0
/ip dhcp-client
add interface=ether1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip dns static
add address=10.1.0.4 name=db.example.net
add address=10.1.0.5 name=cloud.example.net
/ip firewall address-list
add address=1.1.1.1/29 list=office
add address=10.0.0.1/27 list=office
add address=172.16.1.0/30 list=gre_tunnel
/ip firewall filter
add action=accept chain=input comment="Accept all GRE Tunnel Traffic" \
    protocol=gre src-address-list=office
add action=accept chain=input comment="Accept all GRE Tunnel Traffic" \
    dst-address-list=gre_tunnel protocol=gre
add action=accept chain=input comment=\
    "VPN IPSec Encapsulating Security Payload (ESP)" protocol=ipsec-esp
add action=accept chain=input comment=\
    "VPN IPSec Encapsulating Security Payload (AH)" protocol=ipsec-ah
add action=accept chain=input comment="Allow ICMP" protocol=icmp
add action=accept chain=input comment=\
    "Allow LAN to access Services (DNS, etc)" in-interface-list=LAN
add action=accept chain=input comment="Allow Established & Related" \
    connection-state=established,related
add action=accept chain=input comment="Allow All Traffic from Office" \
    in-interface-list=WAN src-address-list=office
add action=drop chain=input comment="Drop Everything Else" log=yes \
    log-prefix=DROP
add action=accept chain=forward comment="Allow All Traffic from Office" \
    out-interface-list=LAN src-address-list=office
add action=accept chain=forward comment="Allow All Traffic to Office" \
    dst-address-list=office out-interface-list=WAN
add action=accept chain=forward comment="Allow Established & Related" \
    connection-state=established,related
add action=accept chain=forward comment="Allow LAN to Talk to Each Other" \
    connection-state=new in-interface-list=LAN
add action=drop chain=forward comment="Drop Everything Else" log=yes \
    log-prefix=DROP
/ip firewall nat
add action=masquerade chain=srcnat comment="Default Masquerade" \
    out-interface-list=WAN
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
/ip ipsec policy
set 0 disabled=yes
/ip route
add disabled=no distance=1 dst-address=10.0.0.1/27 gateway=gre-tunnel2 \
    pref-src=0.0.0.0 routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=America/Los_Angeles
/system ntp client
set enabled=yes
/system ntp client servers
add address=0.us.pool.ntp.org
add address=1.us.pool.ntp.org
add address=2.us.pool.ntp.org
/tool sniffer
set file-limit=5000KiB file-name=CAPTURE2.cap memory-limit=2000KiB
CCR:
# apr/21/2022 17:26:04 by RouterOS 6.48.6
# software id = 
#
# model = CCR1009-8G-1S-1S+
# serial number = 
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1588 speed=100Mbps
set [ find default-name=ether2 ] disabled=yes l2mtu=1588 speed=100Mbps
set [ find default-name=ether3 ] disabled=yes l2mtu=1588 speed=100Mbps
set [ find default-name=ether4 ] disabled=yes l2mtu=1588 speed=100Mbps
set [ find default-name=ether5 ] disabled=yes l2mtu=1590 speed=100Mbps
set [ find default-name=ether6 ] disabled=yes l2mtu=1590 speed=100Mbps
set [ find default-name=ether7 ] disabled=yes l2mtu=1590 speed=100Mbps
set [ find default-name=ether8 ] l2mtu=1590 speed=100Mbps
set [ find default-name=sfp-sfpplus1 ] advertise=\
    10M-full,100M-full,1000M-full l2mtu=1590
set [ find default-name=sfp1 ] advertise=10M-full,100M-full,1000M-full \
    disabled=yes l2mtu=1590
/interface gre
add allow-fast-path=no mtu=1426 name=gre-tunnel2 remote-address=\
    2.2.2.2
/interface list
add name=LAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=CCR1
/ip ipsec profile
set [ find default=yes ] dh-group=modp1024 enc-algorithm=aes-128
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc lifetime=8h pfs-group=\
    modp4096
/ip pool
add name=dhcp_pool1 ranges=10.0.0.2-10.0.0.30
/ip dhcp-server
add address-pool=dhcp_pool1 authoritative=after-2sec-delay interface=sfp-sfpplus1 \
    name=dhcp1
/interface list member
add interface=sfp-sfpplus1 list=LAN
/ip address
add address=1.1.1.1/29 interface=ether1 network=1.1.1.0
add address=10.0.0.1/27 interface=sfp-sfpplus1 network=10.0.0.0
add address=172.16.1.1/30 interface=gre-tunnel2 network=172.16.1.0
/ip dhcp-server network
add address=10.0.0.0/24 dns-server=10.0.0.1 gateway=\
    10.0.0.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip firewall address-list
add address=10.0.0.0/24 comment="Our LAN Subnet" list=LAN
add address=2.2.2.2 list=CLOUD-SERVERS
add address=10.1.0.0/27 list=CLOUD-SERVERS
add address=172.16.1.0/30 list=GRE-TUNNEL
/ip firewall filter
add action=accept chain=input comment="Accept all GRE Tunnel Traffic" \
    protocol=gre src-address-list=CLOUD-SERVERS
add action=accept chain=input comment="Accept all GRE Tunnel Traffic" \
    dst-address-list=GRE-TUNNEL protocol=gre
add action=accept chain=input comment="Established & Related Connections" \
    connection-state=established,related
add action=accept chain=input comment="Allow External Ping" protocol=icmp
add action=drop chain=input comment="Drop Invalid Connections" \
    connection-state=invalid log=yes log-prefix=INVALID
add action=accept chain=input comment=\
    "VPN IPSec Encapsulating Security Payload (ESP)" log-prefix=IPSEC- \
    protocol=ipsec-esp
add action=accept chain=input comment=\
    "VPN IPSec Encapsulating Security Payload (AH)" log-prefix=IPSEC- \
    protocol=ipsec-ah
add action=accept chain=input comment="VPN IPSec Internet Key Exchange (IKE)" \
    dst-port=500 log-prefix=IPSEC- protocol=udp
add action=accept chain=input comment="VPN IPSec NAT Traversal (NAT-T)" \
    dst-port=4500 log-prefix=IPSEC- protocol=udp
add action=accept chain=input comment="Allow LAN => Router" src-address-list=\
    LAN
add action=drop chain=input comment="Drop Everything Else" log=yes \
    log-prefix=DROP
add action=accept chain=forward comment="Established & Related Connections" \
    connection-state=established,related
add action=drop chain=forward comment="Drop Invalid Connections" \
    connection-state=invalid log=yes log-prefix=INVALID
/ip firewall nat
add action=masquerade chain=srcnat comment=\
    "Default NAT Masquerade: LAN --> WAN" out-interface=ether1
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip ipsec policy
set 0 disabled=yes
/ip route
add distance=1 gateway=1.1.1.0
add distance=1 dst-address=10.1.0.0/27 gateway=gre-tunnel2
Last edited by Alupis on Sat Apr 23, 2022 12:53 am, edited 1 time in total.
 
Alupis
just joined
Topic Author
Posts: 23
Joined: Wed Feb 29, 2012 6:30 pm

Re: Very Slow CHR -> CCR GRE/IPIP/EOIP Tunnel

Sat Apr 23, 2022 12:39 am

I think I've narrowed the problem down to traffic going from any of the cloud servers through the CHR. So output from the CHR's perspective.

A MTR in both directions shows zero packet loss:

Image

Cloud Server to Cloud Server throughput is great, measured with iperf:

Image

And Office upload to Cloud Server is fine, however... Cloud Server upload to Office is terrible:

Image

I tried another Cloud Server in the Virtual Private Network (VPC in Vultr terms) and got the same results:

Image

I even tried running Speedtest.net from the Cloud Server and got similar results:

Image

A BTest from the CHR to the CCR looks reasonable for being a IPsec inside a tunnel, which means it's traffic passing through the CHR that is the issue, not traffic originating from the CHR:

Image

What is going on here? I can't seem to think of any configs in the CHR that would throttle things like this. The servers are all Rocky Linux 8.x, with a mostly vanilla setup. If it was a networking issue on the servers I would expect to see the same problems with all traffic, not just traffic going outbound through the CHR.

This makes me think this is not a GRE tunnel issue... I'm not sure what to make of this.
 
mada3k
Long time Member
Long time Member
Posts: 682
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: Very Slow output for traffic passing through CHR

Sat Apr 23, 2022 8:17 pm

Hmm... MTU or fragmentation issue?

Just start count the overhead backwards from the MTU used by your ISP
 
Alupis
just joined
Topic Author
Posts: 23
Joined: Wed Feb 29, 2012 6:30 pm

Re: Very Slow output for traffic passing through CHR

Tue Apr 26, 2022 1:31 am

Hmm... MTU or fragmentation issue?

Just start count the overhead backwards from the MTU used by your ISP
Thanks. I had initially suspected a possible MTU issue.

According to Vultr, all interfaces on the VPC network must bet set to 1450 MTU.

https://www.vultr.com/docs/how-to-creat ... cloud-vpc/

When initially creating the VM, if you attach it to a VPC network is auto-sets the MTU on the correct interface.

So this means,

CHR:
ether1 WAN - MTU 1500
ether2 LAN - MTU 1450
GRE-Tunnel - MTU 1426 (guessed here, matches CCR side as well)

VM:
ether1 DISABLED
ether2 LAN - MTU 1450

For traffic that is VM -> CHR -> Public, it should go MTU1450 -> ( MTU1450 -> BUFFER -> MTU1500 ) -> Public, where everything between the parentheses are inside the CHR.

From one of the cloud VM's (interface MTU 1450), doing a
ping 4.2.2.2 -M do -s 1422
returns fine, but using a packet size of 1423 errors out and needs to fragment. This leaves overhead of about 28 bytes for a VM -> Public packet.

I also tried re-enabling one of the cloud VM's public interfaces (not behind the CHR, attached directly to the public) and a speedtest shows just shy of 5Gbps in both directions:

Image

So it seems to entirely be related to traffic going through the CHR. VM to VM traffic is good (about 2Gbps). VM to public traffic is good (about 5Gbps). VM traffic through CHR is bad (less than 1Mbps).

Not sure where this leaves me...
 
darksoul
just joined
Posts: 2
Joined: Fri Dec 25, 2015 5:42 am

Re: Very Slow output for traffic passing through CHR

Sun Jul 24, 2022 4:16 am

Hi,

I ran into the same issue but with slow upload speeds from my VPC nodes to the outside via mikrotik.

I had reached out their tech support, we ran some tests and everything had point out to the router.

I am sure it's not 100% mikrotik issue, but it seems to me that it's a combination of the way how VPC is organized.

Had anyone had success on this problem?

Thanks.
 
MikeToTheP
just joined
Posts: 1
Joined: Fri Sep 30, 2022 5:51 pm

Re: Very Slow output for traffic passing through CHR

Fri Sep 30, 2022 5:53 pm

Did you ever resolve this?

I'm having similar issues and looking for answers.
 
Underdogs
just joined
Posts: 1
Joined: Thu Dec 01, 2022 3:00 am

Re: Very Slow output for traffic passing through CHR

Thu Dec 01, 2022 3:07 am

I'm having a similar problem in a Vultr environment. The virtual Mikrotik CHR (running 7.6) can upload/download over 500mb/s on the WAN port. The cloud computers connected to it on the virtual LAN can download over 500mb/s, but only upload 7-10mb/s. And even uploading to the Mikrotik directly from the cloud computers is constrained by this speed.

I will note that the problem was initially much worse, with uploads in the 250kb/s range. What helped was reinstalling the cloud Mikrotik CHR on an Intel platform vs. AMD. But the upload of 7-10mb/s is still awful. Tried many MTU settings with no change. Currently have 1450 set on the Mikrotik LAN, as recommended. On the cloud computers (running Windows) I've got the MTU set to 1422 which is optimal to avoid fragmentation.

I have another Vultr account with a nearly identical setup and on that one the uploads work fine. Anyone else run into this?
 
luddite
just joined
Posts: 22
Joined: Fri Apr 06, 2012 12:09 am

Re: Very Slow output for traffic passing through CHR

Thu Dec 29, 2022 10:03 am

Same same, just found this post, after umpteen MTU adjustments and iperf tests between Windows and Linux hosts over the Vultr VPC I can only get 300-600Mbps - Mbps!
(tried lots of mtu values from 1300 up to 1450)

Like poster above said if enabling NIC connecting directly to internet get 4Gbps from Speedtest.net.

Going to give up on Vultr I think.

Who is online

Users browsing this forum: almdandi, GoogleOther [Bot], johnson73, miravic and 82 guests