Significantly higher latency between GRE tunnels

RouterOS General
1

Hello everyone.

I have GRE tunnel between hAP ax² and 951Ui-2HnD routers without any encryption. It’s really simple setup with /30 on GRE interfaces and static routes pointing to each other’s LAN subnets. The problem is, when I ping routers’ WAN interface public IP addresses (from one router to another) I get average 5-10ms, but when I ping IP address assigned to neighbor’s GRE tunnel, then I get something around 80-120ms…

I can’t understand what’s wrong here. Both routers are running on fiber connections, both have marginal CPU utilization (somewhere around 2-5%) and I’ve also set MTU to 1400 on both tunnels (after problem was reported) to avoid any sort of fragmentation. RouterOS upgrades didn’t help either.

What could be causing this?

2

What does /tool sniffer quick ip-protocol=icmp,gre ip-address=public.ip.of.remote,gre.ip.of.remote show on both routers while pinging with default size (hence small) packets?

3

Honestly I don’t understand the output. Both time and numbers columns increase by 1 integer on each polling:

(Public IP addresses replaced by bogon)

INTERFACE       TIME    NUM  DIR  SRC-MAC            DST-MAC            SRC-ADDRESS      DST-ADDRESS      PROTOCOL  SIZE  CPU
GRE_TO_BRANCH   33.694  336  ->                                         192.168.255.17  192.168.255.18    ip:icmp    56    0
ether1_UPLINK   33.694  337  ->   48:A9:8A:B5:1D:42  DC:38:E1:A1:9B:41  203.0.113.10    203.0.113.11      ip:gre     94    0
ether1_UPLINK   33.71   338  <-   DC:38:E1:A1:9B:41  48:A9:8A:B5:1D:42  203.0.113.11    203.0.113.10      ip:gre     94    0
GRE_TO_BRANCH   33.71   339  <-                                         192.168.255.18  192.168.255.17    ip:icmp    56    0
ether1_UPLINK   33.732  340  ->   48:A9:8A:B5:1D:42  DC:38:E1:A1:9B:41  203.0.113.10    203.0.113.11      ip:icmp    70    0
ether1_UPLINK   33.743  341  <-   DC:38:E1:A1:9B:41  48:A9:8A:B5:1D:42  203.0.113.11    203.0.113.10      ip:icmp    70    0
ether1_UPLINK   33.816  342  <-   DC:38:E1:A1:9B:41  48:A9:8A:B5:1D:42  203.0.113.11    203.0.113.10      ip:gre     94    0
GRE_TO_BRANCH   33.816  343  <-                                         192.168.255.18  192.168.255.17    ip:icmp    56    0
GRE_TO_BRANCH   33.816  344  ->                                         192.168.255.17  192.168.255.18    ip:icmp    56    0
ether1_UPLINK   33.816  345  ->   48:A9:8A:B5:1D:42  DC:38:E1:A1:9B:41  203.0.113.10    203.0.113.11      ip:gre     94    0
GRE_TO_BRANCH   34.695  346  ->                                         192.168.255.17  192.168.255.18    ip:icmp    56    0
ether1_UPLINK   34.695  347  ->   48:A9:8A:B5:1D:42  DC:38:E1:A1:9B:41  203.0.113.10    203.0.113.11      ip:gre     94    0
ether1_UPLINK   34.711  348  <-   DC:38:E1:A1:9B:41  48:A9:8A:B5:1D:42  203.0.113.11    203.0.113.10      ip:gre     94    0
GRE_TO_BRANCH   34.711  349  <-                                         192.168.255.18  192.168.255.17    ip:icmp    56    0
ether1_UPLINK   34.732  350  ->   48:A9:8A:B5:1D:42  DC:38:E1:A1:9B:41  203.0.113.10    203.0.113.11      ip:icmp    70    0
ether1_UPLINK   34.742  351  <-   DC:38:E1:A1:9B:41  48:A9:8A:B5:1D:42  203.0.113.11    203.0.113.10      ip:icmp    70    0
ether1_UPLINK   34.824  352  <-   DC:38:E1:A1:9B:41  48:A9:8A:B5:1D:42  203.0.113.11    203.0.113.10      ip:gre     94    0
GRE_TO_BRANCH   34.824  353  <-                                         192.168.255.18  192.168.255.17    ip:icmp    56    0
GRE_TO_BRANCH   34.824  354  ->                                         192.168.255.17  192.168.255.18    ip:icmp    56    0
ether1_UPLINK   34.824  355  ->   48:A9:8A:B5:1D:42  DC:38:E1:A1:9B:41  203.0.113.10    203.0.113.11      ip:gre     94    0
4

I have picked only the interesting groups of packets and removed the MAC address columns for easier reading.

INTERFACE       TIME    NUM  DIR  SRC-ADDRESS      DST-ADDRESS      PROTOCOL  SIZE  CPU
GRE_TO_BRANCH   33.694  336  ->   192.168.255.17  192.168.255.18    ip:icmp    56    0
ether1_UPLINK   33.694  337  ->   203.0.113.10    203.0.113.11      ip:gre     94    0
ether1_UPLINK   33.71   338  <-   203.0.113.11    203.0.113.10      ip:gre     94    0
GRE_TO_BRANCH   33.71   339  <-   192.168.255.18  192.168.255.17    ip:icmp    56    0

GRE_TO_BRANCH   34.695  346  ->   192.168.255.17  192.168.255.18    ip:icmp    56    0
ether1_UPLINK   34.695  347  ->   203.0.113.10    203.0.113.11      ip:gre     94    0
ether1_UPLINK   34.711  348  <-   203.0.113.11    203.0.113.10      ip:gre     94    0
GRE_TO_BRANCH   34.711  349  <-   192.168.255.18  192.168.255.17    ip:icmp    56    0

I wanted to see/show you which stage of processing causes the delay. The TIME column shows the number of seconds from starting the sniff, with a 1 ms resolution. So you can see that the time the sending router needs to encapsulate the ICMP packet into the transport GRE one is less than 1 ms, the response arrives back in 16 ms, and decapsulation again takes less than 1 ms. So what was the ping 192.168.255.18 showing when you were doing this test?

5

Ah I see, so it’s delta from absolute time. Right now it’s dead midnight and no activity in either sides, from what I can see, there are no delays during encapsulation/decapsulation of ICMP in GRE on either router on any direction. It seems like GRE arrives late than unencapsulated ICMP sent over WAN link. Right now it’s not so pronounced, but comparing pings side by side shows that there are still spikes when pinging inner IP…


Thanks for the help. This tool is the handy one I didn’t know about :slight_smile:

6

It cannot be excluded that GRE takes another path between the public addresses than ICMP, ISPs sometimes have funny ideas on their own, and if some government requirements get added to the mix, weird things may happen.

Out of curiosity, is the behavior about the same if you use IPIP instead of GRE? Mikrotik doesn’t use any of the advanced features of GRE (except EoIP where it does use them but in a very strange manner) so there is no advantage of GRE as compared to IPIP.

7

I forgot to mention I’ve tried IPIP as well to exclude problems that could be specific for GRE and the result was really the same unfortunately. What baffles me is that I know our ISPs do not implement any sort of sophisticated traffic manipulation except generic CBWFQ policing and MPLS TE.