Page 1 of 1

MPLS latency.

Posted: Sun Feb 03, 2013 1:15 am
by victornunes
Hello everybody.

I realized that with the use of mpls, latency with a simple ping mikrotik default of 56 bytes jiiter adds a very high combined with a high latency.

see for example, results in a sequence of three consecutive traceroute mpls with and one without mpls along a sequence of pings 20 for each case

with mpls:

1 14ms 6ms 138ms <MPLS:L=709,E=0>
2 151ms 155ms 140ms

1 110ms 143ms 147ms <MPLS:L=709,E=0>
2 148ms 153ms 150ms

1 4ms 8ms 10ms <MPLS:L=709,E=0>
2 11ms 141ms 147ms

The pings mpls:
sent=20 received=20 packet-loss=0% min-rtt=26ms avg-rtt=96ms max-rtt=162ms

Now the test sequence without mpls:

1 2ms 1ms 1ms
2 15ms 17ms 7ms

1 2ms 1ms 2ms
2 7ms 11ms 19ms

1 2ms 1ms 1ms
2 13ms 7ms 7ms

Now without the ping mpls:

sent=21 received=21 packet-loss=0% min-rtt=4ms avg-rtt=19ms max-rtt=93ms

see the stark difference.
Both routers "source" and "destination" are RB450, with RouterOS 5.22.

I will limit myself only provide this information for now.
If someone finds the data scarce, feel free to ask for more information.

Re: MPLS latency.

Posted: Sun Feb 03, 2013 4:18 am
by IPANetEngineer
MPLS is usually faster and more efficient because the router only has to read an 8 byte tag to forward instead of a 20 byte IP header.

What kind of links are connecting the routers?

Re: MPLS latency.

Posted: Sun Feb 03, 2013 5:09 pm
by victornunes

Describe the types of links between the router that I did ping to the first hop,, why the high latency jpa begins on the first jump, which means that if I find the solution, I can extend to other links

rb4500 - ubiquiti(nanobridge m5) - ubiquiti(rocket m5) - rb1100(

rb540 - RouterOS 5.22
nanobridge m5 - mtu 1518
rocket m5 - mtu 1518
rb1100 - RouterOS 5.21

Re: MPLS latency.

Posted: Sun Feb 03, 2013 6:34 pm
by adairw
Don't quote me but I think you need a larger mtu. I use 1544 for all radio links and router interfaces. If I remember correctly 1524 or 1526 is minimum for mpls/vpls?

Re: MPLS latency.

Posted: Sun Feb 03, 2013 6:46 pm
by victornunes
With a mtu of 1518, I can guarantee delivery of 1500 bytes without fragmentation ip.

I did this test with a bit ping with "Do not Fragment".

Remember that the above are my tests with packets of 56 bytes, ie, no need to enter (yet) the merits of mtu.

Re: MPLS latency.

Posted: Sun Feb 03, 2013 8:00 pm
by IPANetEngineer
If you want to pass a 1500 byte packet without fragmentation over mpls:

1526 is the minimum mpls mtu for untagged vpls

1530 is the minimum mpls mtu for tagged vpls

All L2 mtus in the path must be equal to or higher than the mpls mtu

Re: MPLS latency.

Posted: Sun Feb 03, 2013 8:54 pm
by victornunes

But remembering, excessive latency occurs with packets of 56 bytes.

Re: MPLS latency.

Posted: Mon Feb 04, 2013 12:15 pm
by mrz
Check that links are stable, ospf is not flapping etc. Such latency increase is typically due to link flapping and frequent rerouting.

Re: MPLS latency.

Posted: Sat Feb 16, 2013 12:42 pm
by SwissWISP
This sounds a bit like the very annoying GRE/EoIP problem discussed on the Ubnt forum.
Maybe they are related.

Unforunately Ubnt doesn't care. I looks like they think it's a Mikrotik problem...

- Mat

Re: MPLS latency.

Posted: Mon Oct 28, 2013 6:02 pm
by phendry
I see a similar thing also with v5.22 but in our setup every RF link is RouterOS running NStreme with CCQ's ~95%. Disable LDP traceroute shows delay increasing slightly the further away you get as you would expect but with LDP enabled and labels distributed we see first hop increases dramatically but all remaining hops responding around the same. I believe this is because when the TTL in the trace expires at each hop, the packet still needs to continue to the end of the LSP before a response can be returned to the originator. This is a well documented issue with traceroute in an MPLS network. Is this what you are seeing?

Re: MPLS latency.

Posted: Fri Apr 25, 2014 5:21 pm
by solelunauno
There is a key to try some kind of better understanding on tunnel latency problem we are discussing about in this thread. And I found it.

Suppose a network composed by 3 wireless hops and 1 bridge @ 1st hp and 1 router @ 2nd hop (RouterA-AP->CPE-bridge-AP->CPE-router-AP->CPE-RouterB).

You ping RouterB from RouterA directly or inside an Eoip tunnel between A and B.

If you ping with less than 1500 bytes packet size, you notice some big difference between inside and outside Eoip (avg 4ms out, avg 40ms in).

if you ping with 1510 bytes packet size (something little more than 1500), you can see very similar latency inside and outside Eoip tunnel (avg 50-60ms).

if you ping with 10000 bytes backet size (something big), you can see even better result inside Eoip (90-100ms) than outside it (110-120ms).

So, the problem, according to me, is that if you use ICMP and wireless gear can see it is ICMP simple traffic (less than 1500 bytes size), they doesn't make anything; they only forward packets without lose time in making decisions.

If you ping inside tunnel (and so wireless device see it as normal tcp traffic, not icmp), or if you send packets so big to require ethernet fragmentation (with headers to indicate fragments), it seems that wireless devices lose time checking packets and making some kind of decision before forwarding it.

And this time lost is for every kind of traffic different form simple ICMP little unfragmented packets!

I think this is the behavior of every kind of tunnel, included VPLS. Every time packets are something different from simple little ICMPs, wireless multiple hops create high latency.

Re: MPLS latency.

Posted: Mon May 05, 2014 2:43 pm
by victornunes
Let me update the situation
In tests conducted with the goal of finding exactly the point that the latency was increased , I discovered something very interesting.

The test had consisted solely using the " ping " command and wireshark software .
I ran wireshark on two ubiquiti radios , and the test consisted in this .

Router1 - > radio1 - > Radio2 - > Router - > radio3 - > radio4 - > Router2

With wireshark , I can check the timestamp of package " echo reply " , thus discovering how long it takes for the package " atraversar " link radio .
Results of my tests :
I realized that increased latency occurs in the back of the package , when package has no mpls marking due process " penultimate hop" mpls .
Ie , the echo reply packet exits the router 2 and goes to router1 , when arriving at Radio2 , the package is with a latency of 2/3 ms , but when arriving at radio1 , the echo reply packet is with a latency of 20 / 30/40 ms .

Anyway , I believe I have been clear .
Hope this helps.