Community discussions

MUM Europe 2020
 
User avatar
horse1bun
just joined
Topic Author
Posts: 13
Joined: Wed Jul 29, 2015 1:05 am

MPLS/VPLS Packet Loss and Slow Speeds

Thu Nov 30, 2017 9:32 pm

I am troubleshooting packet loss and slower than expected throughput in a VPLS tunnel. Hopefully somebody here can point me in the right direction?

Here is the setup:

VPLS on Bridge Interface, CCR1016-12S-1S+ <--trango backahaul--> LDP Interface, CCR1036-12G-4S <--trango backahaul--> VPLS on Bridge Interface, CCR1016-12S-1S+

We setup a user with a VPLS circuit across three hops. They are using RFC2544 testing between two Ciena appliance at the locations. They are unable to reach a 100 Mbps throughput and seeing packet loss at all levels:
+--------------------------------------------- BENCHMARK Test Results and Statistics ----------------------------------------------+
| Test instance CO2544TEST using profile CO2544TEST dest MAC c4:83:6f:0a:0b:0c and vtag-stack 4004 |
|----------------------+-----------------------------------------------------------------------------------------------------------|
| | Results for green traffic based on requested bandwidth of 100 Mbps |
| |-----+------------------+----------+----------+----------+---------------------------+---------------------+
| | | Frame | | | | KPI | |
| | PCP | Size | Min | Avg | Max | % (Mbps) Result | Iter |
| |-----+------------------+----------+----------+----------+------------------+--------+---------------------+
| Throughput (Mbps) | 0 | 1518 | 7.71 | 57.83 | 98.70 | -- ( -- ) | -- | 6 |
| |-----------------------------------------------------------------------------------------------------------+
| | Results for green traffic |
| |-----+------------------+-----------+-----------+-----------+------------+--------+------------------------+
| | | Frame | | | | KPI | Samples |
| | PCP | Size | Min | Avg | Max | usec Result | Cycles Packets |
| |-----+------------------+-----------+-----------+-----------+------------+--------+--------+---------------+
| Latency (usec) | 0 | 1518 | 12067 | 14317 | 130939 | -- | -- | 10 | 80399 |
| |-----------------------------------------------------------------------------------------------------------+
| | Results for green traffic based on the maximum throughput found |
| |-----+------------------+-------------------+------------+------------+------------------------------------+
| | | Frame | Bandwidth | | | KPI |
| | PCP | Size | % (Mbps) | % 1st Test | % 2nd Test | % Result |
| |-----+------------------+-------------------+------------+------------+------------+-----------------------+
| Frame Loss | 0 | 1518 | 100 ( 100) | 0.31 | -- | -- | -- |
| Frame Loss | 0 | 1518 | 90 ( 90) | 0.37 | -- | | |
| Frame Loss | 0 | 1518 | 80 ( 80) | 0.74 | -- | | |
| Frame Loss | 0 | 1518 | 70 ( 70) | 8.18 | -- | | |
| Frame Loss | 0 | 1518 | 60 ( 60) | 2.74 | -- | | |
| Frame Loss | 0 | 1518 | 50 ( 50) | 0.20 | -- | | |
| Frame Loss | 0 | 1518 | 40 ( 40) | 0.37 | -- | | |
| Frame Loss | 0 | 1518 | 30 ( 30) | 3.43 | -- | | |
| Frame Loss | 0 | 1518 | 20 ( 20) | 2.39 | -- | | |
| Frame Loss | 0 | 1518 | 10 ( 10) | 0.37 | -- | | |
| |-----------------------------------------------------------------------------------------------------------+
| Statistics | |
| Matching Test | |
| RX pkts | 758928 |
| IPv4 all | 758928 |
| IPv4 UDP | 758928 |
| IPv6 all | 0 |
| IPv6 UDP | 0 |
| non IP | 0 |
| unrecognized | 0 |
| RX seq num stats | 3075 |
| Dupl seq | 0 |
| Out-of-order seq | 1078 |
| Disc in seq | 1997 |
| TX pkts | 770857 |
+----------------------+-----------------------------------------------------------------------------------------------------------+
We raised L2 MTU on all interfaces between endpoints. I can ping a 1600 KB packet between the VPLS endpoints, and traceroute shows only 1 hop.
full-size-packets.png
I can run a Bandwidth Test and get about 200 and the bandwidth test shows no packet loss.
vpls-btest.png
Traffic generator does show loss between endpoints at anything over 10 Mbps or so.
traffic-gen-vpls.png
Does anyone know what specifically might be the cause of this? Or worked with RFC2544 testing and know a way I can simulate the same using Mikrotik tools?

What can I possibly do to fix this?
You do not have the required permissions to view the files attached to this post.
 
User avatar
vasilevkirill
Trainer
Trainer
Posts: 48
Joined: Tue May 22, 2012 7:38 am
Location: Russian, Saint-Petersburg
Contact:

Re: MPLS/VPLS Packet Loss and Slow Speeds

Thu Nov 30, 2017 10:05 pm

hi
disable mpls
add EoIP tunnel instead VPLS and test again
and change MTU and L2MTU as well as VPLS tunnel
Vasilev Kirill
( MTCNA, MTCRE, MTCWE, MTCTCE, MTCUME, MTCINE, MTCIPv6E, MTCSE )
MikroTik Certified Trainer & Consultant
Trainer Сertificate Number TR0417
https://www.mikrotik.me
Cell:+7 (905) 207-35-78
 
User avatar
horse1bun
just joined
Topic Author
Posts: 13
Joined: Wed Jul 29, 2015 1:05 am

Re: MPLS/VPLS Packet Loss and Slow Speeds

Thu Nov 30, 2017 11:31 pm

I have tested the backhaul capacity and it seems we have about 350 Mbps / 500 Mbps capacity overhead available.

This user is requiring MPLS for this connection so I can't provide an EoIP tunnel and still satisfy them. Do you mean for testing purposes? And what specifically do you recommend to change with MTU/L2MTU?
 
User avatar
vasilevkirill
Trainer
Trainer
Posts: 48
Joined: Tue May 22, 2012 7:38 am
Location: Russian, Saint-Petersburg
Contact:

Re: MPLS/VPLS Packet Loss and Slow Speeds

Fri Dec 01, 2017 12:29 am

Yes for testing.
change in the EoIP MTU and L2MTU such as on the VPLS tunnel
Vasilev Kirill
( MTCNA, MTCRE, MTCWE, MTCTCE, MTCUME, MTCINE, MTCIPv6E, MTCSE )
MikroTik Certified Trainer & Consultant
Trainer Сertificate Number TR0417
https://www.mikrotik.me
Cell:+7 (905) 207-35-78
 
andreint
just joined
Posts: 1
Joined: Tue Dec 05, 2017 1:17 am

Re: MPLS/VPLS Packet Loss and Slow Speeds

Tue Dec 05, 2017 1:25 am

I have the same problem. If you downgrade the middle MPLS CCR to version 6.36.4, the problem does not occur. This problem only occurs in versions greater than 6.36.4. I already tested it with 6.39.3 (bugfix only) and it continues.
 
User avatar
IPANetEngineer
Trainer
Trainer
Posts: 1053
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: MPLS/VPLS Packet Loss and Slow Speeds

Tue Dec 05, 2017 6:15 pm

Two questions:

1) Have you run an RFC2544 test on the base RF link without MPLS to see if it passes?

2) What are your MTU settings for: L2MTU, L3 MTU and MPLS MTU on the MikroTik routers?

This is a presentation I did for WISPs that want to use MPLS at the 2016 US MUM

https://mum.mikrotik.com//presentations ... 279781.pdf
Global - MikroTik Support & Consulting - English | Francais | Español | Portuguese +1 855-645-7684
https://iparchitechs.com/services/mikro ... l-support/ mikrotiksupport@iparchitechs.com
 
User avatar
horse1bun
just joined
Topic Author
Posts: 13
Joined: Wed Jul 29, 2015 1:05 am

Re: MPLS/VPLS Packet Loss and Slow Speeds

Wed Jan 03, 2018 2:19 pm

After much troubleshooting, I wanted to give an update.

We plugged the RFC2544 appliance directly into the Trango backhaul. The test failed there. Speaking with their engineer, it sounded like a flow control issue. Theory was that when raw data is sent to the interface the traffic can be "bursty" - reaching more than the 375 x2 Mbps radio link capacity and overwhelming the buffer on the radio(s) causing the packet loss.

We turned on flow control on the radios and on the Mikrotik interfaces to see if this would help. Still did not pass the test. The buffer on the radios is only 128 Kb is what I recall the engineer saying.

We planned on upgrading to a Giga Lynx but they are on back order so we installed a SAF Tehnika Integra WS.

At this point we were still seeing packet loss between two directly connected routers behind the wireless backhaul (connected via fiber) and still not passing the test over the wireless backhaul.

Due to this and some other odd issues happening with the CCR-1016 we have had in service at this site for several years, we decided to upgrade to a CCR-1036.

I am happy to report that these steps resolved the issue and our user is satisfied with their testing so far. I have attached my traffic generator results :)
You do not have the required permissions to view the files attached to this post.

Who is online

Users browsing this forum: No registered users and 6 guests