Community discussions

 
ncats
just joined
Topic Author
Posts: 8
Joined: Mon Apr 23, 2018 9:37 pm

Bottleneck on CCR (possible queue related)

Mon Apr 23, 2018 9:57 pm

I was running a CCR-1036(v6.41.1). I used a lag to deliver 4Gbps. I have 2,300 simple queues. Each simple queue use the "default" queue type and the default queue type was modified to have a 100 packet queue. Each interface queue on the CCR-1026 was changed from Hardware to "ethernet-default" with a packet queue of 100.

This setup served us well for quite a while. at some point performance problems became noticeable. During the day while total traffic was roughly 1,200Mbps, customers with simple queues limited at 20Mbps could easily pull 20Mbps. However, in the evening hours when traffic rose over 2,000Mbps, these same customers could only pull 8-10Mbps. This would be very consistent, like the router was specifically limiting them at a lower speed through the evening hours.

A week ago, we replace the CCR-1036 with a CCR-1072. We left all the queue types default and have found that performance has suffered. Customers are getting less than 1Mbps in the evening under the same conditions, yet in the daytime, they are still seeing their full speeds. I have verified that my Internet connection is working and capable of much greater speeds during the evening hours. The bottleneck is at the CCR-1072.

Can anyone help me understand the optimal configuration to handle multi-gigabit throughput while maintaining bandwidth limiting queues for individual IPs?

Thanks!
 
ncats
just joined
Topic Author
Posts: 8
Joined: Mon Apr 23, 2018 9:37 pm

Re: Bottleneck on CCR (possible queue related)

Tue Apr 24, 2018 3:18 pm

I changed all of my interface queues to "multi-queue-ethernet-default" and increased the queue size to 500.
my simple queues are assigned the default queue type and I increased the queue size on that to 150.

Last night, regular customer connections did not appear to be bottlenecked anymore. The highest plan we sell is 25Mbps and that speed was achievable.

The router still does not perform perfectly however. In testing using speedtest.net, an uncapped test client gets less than half the speed inside the router as it does outside the router. I see 150-200Mbps down inside and 500Mbps+ outside.

I would love to know a formula that would help me both understand and appropriately calculate an ideal queue type and settings.

Thanks!
 
ncats
just joined
Topic Author
Posts: 8
Joined: Mon Apr 23, 2018 9:37 pm

Re: Bottleneck on CCR (possible queue related)

Tue Apr 24, 2018 11:20 pm

One last update: A client with the potential for 100Mbps had the following results:

Total Router Traffic / Client Speed Test
1600Mbps: 90Mbps
2000Mbps: 60Mbps
2500Mbps: 30Mbps
1800Mbps: 60Mbps
1200Mbps: 90Mbps

Again, the speed tests on the outside of the router remain flawless. The CCR is still a bottleneck, but increasing the queue size has helped substantially. I've yet to try queue sizes higher than 500. Is there a whitepaper regarding queues that would help calculate an ideal queue configuration dependent on packet load?
 
R1CH
Forum Veteran
Forum Veteran
Posts: 846
Joined: Sun Oct 01, 2006 11:44 pm

Re: Bottleneck on CCR (possible queue related)

Tue Apr 24, 2018 11:50 pm

Are you sure this isn't caused by your LAG? Depending how you are distributing packets you may be saturating one of the ports with too much traffic. Any chance to test with a 10G uplink?
 
ncats
just joined
Topic Author
Posts: 8
Joined: Mon Apr 23, 2018 9:37 pm

Re: Bottleneck on CCR (possible queue related)

Wed Apr 25, 2018 10:14 pm

R1ch,

The LAG was configured on the CCR1036 and ran perfectly balanced according to the status and graphs. This no longer matters, because the CCR1036 was the old router. It was replaced by the CCR1072. There is no longer a LAG since the CCR1072 has 10G.

As stated, there is a drastic difference to be made in the queue sizes. I do not know the role of interface queue vs simple queue and how they interact. As a result, I do not know how to calculate an optimal queue size for the type of traffic we are processing.

Is a 500 packet queue a lot? Is it too little? I could use some helping understanding.
 
ncats
just joined
Topic Author
Posts: 8
Joined: Mon Apr 23, 2018 9:37 pm

Re: Bottleneck on CCR (possible queue related)

Wed May 01, 2019 7:51 pm

We recently hit this same wall again. Our interface queues were set to multi-queue-ethernet-default with a queue size of 1000. This served us up to a peak throughput of 4Gbps. I've had to increase the queue size to 1500 to get our peak throughput above that.

We do not need interface queues. We have 10Gbps capacity. However, running only-hardware-queue creates a bottleneck.

This is a CCR1072-1G-8S+ on v6.43.7

What is the recommended use of interface queues when you have excess Internet capacity?
 
madman22
just joined
Posts: 5
Joined: Fri Mar 23, 2018 4:28 pm

Re: Bottleneck on CCR (possible queue related)

Sat May 18, 2019 1:36 am

I'm pretty sure we are running into this issue as well on a CCR1072. Bursting up to 3.4gbps using the default mq-ethernet-default settings. Customers are getting really randomly fluctuating speedtests. Going to try increasing the buffer size during the next maintenance window. All of our traffic is coming in on vpls tunnels and getting a dhcp queue applied then all going out the uplink port. My problem could also be the default settings that are used when a dhcp lease applies a queue.

I just upgraded to 6.44 though and we are seeing other problems that were not on 6.43, cpu usage is about 5% higher per core for the same bandwidth.

Wish there was some sort of visual representation of what queues/processes are holding up packets/processing.
 
User avatar
amt
Long time Member
Long time Member
Posts: 514
Joined: Fri Jan 16, 2015 2:05 pm

Re: Bottleneck on CCR (possible queue related)

Sat May 18, 2019 3:53 pm

We recently hit this same wall again. Our interface queues were set to multi-queue-ethernet-default with a queue size of 1000. This served us up to a peak throughput of 4Gbps. I've had to increase the queue size to 1500 to get our peak throughput above that.

We do not need interface queues. We have 10Gbps capacity. However, running only-hardware-queue creates a bottleneck.

This is a CCR1072-1G-8S+ on v6.43.7

What is the recommended use of interface queues when you have excess Internet capacity?
any news about this ? setting queue size to 1500 solve your problem ? and do you faced any bad side while increasing que size ? also what do u choose for kind mq pfifo ?

Who is online

Users browsing this forum: No registered users and 83 guests