Community discussions

 
dbeyzade
just joined
Topic Author
Posts: 10
Joined: Sat Jan 06, 2018 11:11 am

RB1100AHx4 latency spikes

Mon Feb 26, 2018 5:27 pm

Hi there,

I have two completely independent internet connections fed into my RB1100AHx4. One is a 250mbps up down "gigaclear", the other is a 40mbps download 5mbps upload "BT" connection. In everyday use only the gigaclear connection is in use, the other is an on standby backup
2837f5183808f9eb8ee665cbb5ff189d5e37208d.png
61f272096790f9f96c618b16efbbcb66b758d3da.png
I have both connections monitored via each connection to show ping response and latency (see attachments). You will notice there there some ping spikes which are lining up identically. Therefore I have come to the conclusion that the RB1100AHx4 is causing this - any ideas why? Indeed during web browsing there are occasional hangs - hence the investigation.

Thank you for any ideas / help
You do not have the required permissions to view the files attached to this post.
 
lambert
Long time Member
Long time Member
Posts: 526
Joined: Fri Jul 23, 2010 1:09 am

Re: RB1100AHx4 latency spikes

Mon Feb 26, 2018 7:38 pm

Which ports are you using for the connections? Are they both on the same switch chip?

While I have no personal experience with the x4, the older RB1100AH and AHx2 units had recommendations for using multiple independent ports rather than the switched port groups. Perhaps the x4 also requires care in selecting the ports for uplink?
 
Paternot
Long time Member
Long time Member
Posts: 572
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: RB1100AHx4 latency spikes

Mon Feb 26, 2018 9:07 pm

Have You tested the connection with another router? I have seen graphics like this related to the Intel Puma7 chipset, in cable modems. Could be this case?
 
dbeyzade
just joined
Topic Author
Posts: 10
Joined: Sat Jan 06, 2018 11:11 am

Re: RB1100AHx4 latency spikes

Mon Feb 26, 2018 10:59 pm

Thanks for the help guys,

The 1st ISP Gigaclear is connected to port 1 and the Second ISP AAISP / BT is connected to port 2. However both connections are not used at the same time, the second connection is only a backup so it's only used if the primary fails so I don't think its a switching chip problem.

The gigaclear has a custom FTTP router.
The BT has an Openreach VDSL modem

So I think it rules out both providers having PUMA chipsets, and surely they wouldn't exhibit the latency spike at exactly the same time as shown in the graphs?
 
dbeyzade
just joined
Topic Author
Posts: 10
Joined: Sat Jan 06, 2018 11:11 am

Re: RB1100AHx4 latency spikes

Mon Feb 26, 2018 11:02 pm

I do have around 30 firewall rules, plus NAT's to differnt PUBLIC IP addresses' for different subnets, and also connection tracking. Do you think all this could cause the problem? CPU usage is only showing 1 - 2 percent when I have inspected it
 
Paternot
Long time Member
Long time Member
Posts: 572
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: RB1100AHx4 latency spikes

Mon Feb 26, 2018 11:47 pm


So I think it rules out both providers having PUMA chipsets, and surely they wouldn't exhibit the latency spike at exactly the same time as shown in the graphs?
True. I didn't notice they were at the same time - only looked at the pattern. Well, it rules out the PUMA chipset. How do You choose the default route? Could be some sporadic problem with this? Maybe a DHCP renew, forcing the router to recalculate the routes? Do you have a CPU spike as well? I remember people complaining about a NATed PPPoE disconnection being too heavy on the CPU. Although it was going on on a CCR, with 2k or 3k clients...
 
lambert
Long time Member
Long time Member
Posts: 526
Joined: Fri Jul 23, 2010 1:09 am

Re: RB1100AHx4 latency spikes

Mon Feb 26, 2018 11:55 pm

Move the big connection to port 13. See if latency remains and if they still line up. It's a pain, but 1 and 2 are on the same switch chip. Depending on what is in ports 3, 4, and 5, that 1Gbps connection to the CPU may be overloaded.
 
Paternot
Long time Member
Long time Member
Posts: 572
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: RB1100AHx4 latency spikes

Tue Feb 27, 2018 1:48 am

Move the big connection to port 13. See if latency remains and if they still line up. It's a pain, but 1 and 2 are on the same switch chip. Depending on what is in ports 3, 4, and 5, that 1Gbps connection to the CPU may be overloaded.
His 1100 is the x4. Its internal connections are 2,5Gb/s
 
lambert
Long time Member
Long time Member
Posts: 526
Joined: Fri Jul 23, 2010 1:09 am

Re: RB1100AHx4 latency spikes

Tue Feb 27, 2018 3:11 am

That's cool. It's still a single switch. Splitting is still a possible win.
 
dbeyzade
just joined
Topic Author
Posts: 10
Joined: Sat Jan 06, 2018 11:11 am

Re: RB1100AHx4 latency spikes

Tue Feb 27, 2018 11:38 am

The default route is set to Gigaclear. I then have a script running that checks connectivity to this ISP and if it has failed marks this route higher that the 2nd ISP.

There is other stuff connected to ports 3,4,5 so I think moving the Gigaclear connection over to port 13 is definitely worth trying.

However if you look at the graph the latency spikes continue throughout the night, when there is virtually zero traffic running through the system.

The DHCP renew is an interesting idea, but i have no idea how I would go about checking this out.

What is the best way for checking for a CPU spike - a graph?
 
User avatar
sebastia
Forum Guru
Forum Guru
Posts: 1578
Joined: Tue Oct 12, 2010 3:23 am
Location: Antwerp, BE

Re: RB1100AHx4 latency spikes

Tue Feb 27, 2018 2:25 pm

With regards to dhcp renew, does the ip actually change? If not it means that a process will request extension, and process response, a pattern like so many running over the router. That in itself will not result in increased latency.
 
Paternot
Long time Member
Long time Member
Posts: 572
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: RB1100AHx4 latency spikes

Tue Feb 27, 2018 3:08 pm

The default route is set to Gigaclear. I then have a script running that checks connectivity to this ISP and if it has failed marks this route higher that the 2nd ISP.

What is the best way for checking for a CPU spike - a graph?
You can enable graphics to the CPU, on the router itself.

tools -> graphing
 
dbeyzade
just joined
Topic Author
Posts: 10
Joined: Sat Jan 06, 2018 11:11 am

Re: RB1100AHx4 latency spikes

Tue Feb 27, 2018 3:18 pm

With regards to dhcp renew, does the ip actually change? If not it means that a process will request extension, and process response, a pattern like so many running over the router. That in itself will not result in increased latency.
No the addresses provided from each ISP never change - they are in effects static
 
dbeyzade
just joined
Topic Author
Posts: 10
Joined: Sat Jan 06, 2018 11:11 am

Re: RB1100AHx4 latency spikes

Tue Feb 27, 2018 3:19 pm

The default route is set to Gigaclear. I then have a script running that checks connectivity to this ISP and if it has failed marks this route higher that the 2nd ISP.

What is the best way for checking for a CPU spike - a graph?
You can enable graphics to the CPU, on the router itself.

tools -> graphing
The graphs for CPU are empty - do I need to enable it somewhere?
Capture.PNG
You do not have the required permissions to view the files attached to this post.
 
Paternot
Long time Member
Long time Member
Posts: 572
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: RB1100AHx4 latency spikes

Tue Feb 27, 2018 5:49 pm


The graphs for CPU are empty - do I need to enable it somewhere?
Capture.PNG
Yes. In the Tools -> Graphing section.

At the command line:
/tool graphing resource add

Will add CPU, memory and disk graphs. This will allow access to the graphs from any address. If you want to limit this, add the range of allowed addresses, or limit it in the firewall.
 
dbeyzade
just joined
Topic Author
Posts: 10
Joined: Sat Jan 06, 2018 11:11 am

Re: RB1100AHx4 latency spikes

Fri Mar 02, 2018 10:41 am

Here is the graph - nothing odd I think
graph.PNG
You do not have the required permissions to view the files attached to this post.
 
Paternot
Long time Member
Long time Member
Posts: 572
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: RB1100AHx4 latency spikes

Fri Mar 02, 2018 3:16 pm

Have you checked the network cables? Look at the ethernet interface status page. See if there are reconnection, lost ethernet packages, loss of traffic...

I had a problem with a Ubiquiti POE, wich was fast ethernet/full duplex - but only got synchronized at fast ethernet/half duplex. And the packet loss was about 2%. Forced it to 100/full duplex, and all is well.

All this to say that your problem may be one of compatibility at the ethernet layer. Wouldn't be the first one with this kind of problem here...
 
dbeyzade
just joined
Topic Author
Posts: 10
Joined: Sat Jan 06, 2018 11:11 am

Re: RB1100AHx4 latency spikes

Fri Mar 02, 2018 3:20 pm

Ports 1 - 6 are in use all at 1gbps full duplex - with the exception of port 2 which is at 100mbps full duplex. No errors, no drops
 
User avatar
Cha0s
Forum Veteran
Forum Veteran
Posts: 874
Joined: Tue Oct 11, 2005 4:53 pm

Re: RB1100AHx4 latency spikes

Fri Mar 02, 2018 4:48 pm

How does your monitor system (I can't tell which is it from the screenshot - never seen it before) connect to the router and subsequently reach the uplinks? Is there a switch in between?

Do you monitor any other local devices (and the router itself)? Do you see the same latency spikes on local devices as well?

What I am trying to say, is, is it possible that the lag is caused by the monitoring box itself rather than the router?
It has happened to me before with Smokeping, where the whole box was lagging (don't remember why) causing spikes on all graphs at the same time.

Also, I've seen changes in the latency when traffic flows from another cpu core than before (ie during a reboot of the router).
I've seen this behavior in RB3011 but the deviation is very small (0.1-0.3ms) and it's not as brief as shown in your graphs but can last for hours (until for some reason the traffic shifts back to the other cpu core)

Who is online

Users browsing this forum: No registered users and 15 guests