Help with Latency problem needed

anders2 · Thu Feb 13, 2025 11:42 pm

Hello,

I have been running a Hap AC3 for about two years now without noticing any problems. Then I wanted to play Path of Exile 2 - and things went bad.

When playing the game, I get a latency spike every 15.5 seconds. Latency goes from 15-17ms to above 300ms for about half a second to a second.

My setup is an ISP provided router <-> (ether1) HAP AC3 (ether3) <-> PC. Connecting my PC directly to the ISP provided router makes the problem go away completely - played for about an hour, no sign of the problem.

I have tried the following without seeing any changes to this behaviour:

I have tried upgrading the HAP firmware, it now runs 7.17.2
I have tried replacing cables and using other ports on both ISP router and HAP - except for changing ether1, that is a last resort as it would be a pain to reconfigure (at least to my knowledge)
I have tried disabling IP -> Settings -> Allow Fast Path
I have tried disabling Fast Forward on the bridge
I have tried disabling the filter rule for fasttrack (default configuration)
I have tried setting the queue for ether1 (wan) to "multi-queue-ethernet"
I have tried creating a queue tree for ether1
....and I have tried rebooting the HAP

What I have observed:

It happens exactly every 15.5 seconds. Some spikes are smaller other are larger (max 1 second)
Using the torch on ether1, ether3 and the bridge, I can see a decline in packages at the exact same time as the latency spike in the game. It drops to about 50% of the packages for a brief second
There are no packets lost in any of the metrics
The CPU load is around 1-2%
I have around 175mb free memory of 256mb
I have 102mb free HDD space
The game seems to be using TCP instead of UDP (my queue reflects this)
Most packets are fasttracked (unless i disabled it as described above)

I have truly exhausted my own capacity in debugging this problem and would really appreciate any input on what to try next.

Although I have been using the HAP for a couple of years, I am by no means an expert in neither Mikrotik devices nor networks in general, so bare over with me if I missed something obvious.

Thanks in advance!

Moba · Fri Feb 14, 2025 3:25 am

Things to try:

1. Disable CAKE, if used
2. From a CMD prompt: ping 1.1.1.1 -t (or any local DNS server) and check for spikes
3. Make sure you updated both the firmware and the software packages
4. Test other games to see if you can replicate the behaviour
5. Backup your config and reset the device to defaults
6. Netinstall
7. Make sure your network drivers are updated
8. Revert to a prior ROS version
9. Revert to v6, if possible (firmware can't be older than factory)

anders2 · Fri Feb 14, 2025 12:34 pm

Thank you for the suggestions. Here are my results:

I don't use CAKE
Let me get back to this
Everything is updated on all devices - firmware 7.17.2, software 7.17.2 (stable)
Tried with Diablo 4 and Path of Exile 1 - the latency is not noticable ingame, most likely due to better ingame handling of unstable networking, but when monitoring the packets using torch, it shows the exact same problem
Tried making a backup, resetting and reapplying the backup, did not resolve the problem
Tried completely wiping it with NetInstall and reapplying my backup, did not resolve the problem
Everything is up to date on the PC used for the testing
When I began looking into this a few days back, I was on an older version (7.12.x if I remember correctly) and that version had the issue as well. If you have suggestions for a specific version to try, please let me know, but it does not seem like something that was introduced in 7.17
Not possible, factory firmware is 7.2.3

As I said, I would get back to the ping testing - that showed some weird results.

I used to following command for all tests (on linux)

ping -c 1000 -i 0.1 1.1.1.1 | ts "%b %d %H:%M:%.S"

I am using the -i option to send every 0.1 second instead of every second to make sure I would hit the spikes that I am seeing.

When connected directly to the ISP router, i got these results:

Feb 14 11:02:34.026676 1000 packets transmitted, 1000 received, 0% packet loss, time 100305ms

When connected through the HAP, I got these results:

Feb 14 10:58:54.805102 1000 packets transmitted, 969 received, +7 errors, 3.1% packet loss, time 100396ms

Definitely a problem with packet loss. When looking through the results for when having the HAP in the loop, I noticed that the ping fails approximately every 15.5 seconds - which matches the spikes I see for the games. I have attached the output. At least this tells me that it is not game-related.

Ca6ko · Fri Feb 14, 2025 1:10 pm

try pinging 192.168.8.1.

anders2 · Fri Feb 14, 2025 1:34 pm

As suggested, I tried doing the ping test towards the LAN interface (192.168.8.1 ether3) on the HAP and got no errors.

ping -c 1000 -i 0.1 192.168.8.1 | ts "%b %d %H:%M:%.S"
Feb 14 12:18:23.761413 1000 packets transmitted, 1000 received, 0% packet loss, time 103846ms

It seems like the connection to ether3 on the HAP is fine.

I then tried doing the same test against the ISP router 192.168.7.1 and still did not get any package loss - which seems inconsistent to me.

ping -c 1000 -i 0.1 192.168.7.1 | ts "%b %d %H:%M:%.S"
Feb 14 12:21:04.977789 1000 packets transmitted, 1000 received, 0% packet loss, time 103889ms

I then tried again pinging 1.1.1.1 and got the expected results of 7 failed packages - fails every 15.5 seconds.

ping -c 1000 -i 0.1 1.1.1.1 | ts "%b %d %H:%M:%.S"
Feb 14 12:23:33.448161 1000 packets transmitted, 972 received, +7 errors, 2.8% packet loss, time 100387ms

I did a traceroute to 1.1.1.1 and tried the ping test towards the first item on the list that is not in my house (ISP gateway). That test fails as well:

Feb 14 12:31:12.319223 1000 packets transmitted, 966 received, +6 errors, 3.4% packet loss, time 100525ms

What seems inconsistent to me, is that when I connect my PC directly to the ISPs provided router, there are no problems, but when it goes through the HAP, the problems occur.

lurker888 · Fri Feb 14, 2025 2:29 pm

Hi there!

Getting "net unreachable" (that fast) means that there is no route to the network present. This can happen because of some sort of DHCP confusion or port flapping.

Port flapping is the common one. Can you confirm that you don't have link up/down notifications (in the log) and the relevant counter (for the interface) isn't really high?

This sort of thing is really common with some ISP supplied equipment, the most notorious being the FritzBox.

What baffles me, is that the same should be present when pinging 192.168.7.1 through the ac3. I know it's bad form to ask, but are you *absolutely* sure that you did this test correctly? Is it fully repeatable?

Could you please redo the test with 8.8.8.8? Not that there's anything wrong with 1.1.1.1, but sometimes these hosts (officially DNS servers) limit the number of pings returned to the same address for abuse reasons. I'm quite sure they do it differently, so if both show the same reading, then we can rule this out.

anders2 · Fri Feb 14, 2025 3:19 pm

Thank you for the response.

Can you confirm that you don't have link up/down notifications (in the log) and the relevant counter (for the interface) isn't really high?

I tried poking around. There are no logs in my ISP provided router. It is weird, but looked at the user guide, and it doesn't mention logs at all. It is an Icotera i4882-00. I also looked at the logs through WinBox - the only mention of link up or down was when I rebooted the HAP earlier today.

When I look at both ether1 (wan) and ether3 (lan) interfaces, they both show "last link up time" as the same time i rebooted earlier today and the "last link down time" is blank.
I am not sure if there are other ways of checking this - as I stated in my original post, I am no network expert, so if you have any other ways of checking, please let me know.

I re-executed the tests and this time I created scripts for doing a run to make sure it is the same. It is a very fair question to ask if they are repeatable - I am baffled as well. I did the tests again, with the exact same results.

pingtest-hap.zip contains the script and the results when the connection was:
PC -> (ether3) HAP (ether1) -> ISP Router

pingtest-nohap.zip contains the script and the results when the connection was:
PC -> ISP Router

Both tests were done with WIFI turned off on both the ISP router (allways off) and the HAP and my PC being the only thing connected to the network.

When going straight to ISP router:
No errors or dropped packets when doing 1000 pings to either cloudflare, google or the ISP router

When going through the HAP:
Errors when pinging cloudflare and google - no errors when pinging ISP supplied router or HAP.

The fact that I do not get the errors when pinging the ISP supplied router really surprises me - but it is consistent with earlier results.

lurker888 · Fri Feb 14, 2025 4:08 pm

I meant logs on the Mikrotik. Have a look. They should contain details.

I am baffled by your results.

Otherwise please post full config export of your ac3.

anders2 · Fri Feb 14, 2025 5:01 pm

I rebooted the device, ran the tests and looked at the log. Nothing out of the ordinary - have attached a screenshot. Only thing that peaks my interest is that I have noticed that every time I reboot, it has to adjust the cloud time. Is that normal behaviour?

I did an export file=xxxx and have attached the file as well. I believe that should be the configuration.

lurker888 · Fri Feb 14, 2025 5:32 pm

Yep. Those logs are normal.

The "cloud time" thing is also normal. These devices don't have a proper (external, battery-backed) RTC, so they lose a few tens of seconds at each reboot.

I'll take a look at your config - at some indefinite point in time in the future

lurker888 · Fri Feb 14, 2025 5:46 pm

That point in time was now.

You have a misconfiguration. On ether 1 you can either:
* use a DHCP client to acquire an address and a default route
* set an address manually and configure a default route by hand

You have attempted to do both, the second one only halfway. Your test results are actually consistent with this (mis)configuration.

I suggest the second approach. To achieve this:
1. disable / remove the dhcp client on ether1
2. add a default route: /ip/route/add dst-address=0.0.0.0/0 gateway=192.168.7.1

Let me know how it works out!

anders2 · Fri Feb 14, 2025 5:59 pm

It works!

Feb 14 16:55:54.997435 --- 1.1.1.1 ping statistics ---
Feb 14 16:55:54.997469 1000 packets transmitted, 1000 received, 0% packet loss, time 100292ms

Lurker888, you are truly amazing!

Thank you so much for assisting

Now it is time for some gaming!

lurker888 · Fri Feb 14, 2025 6:02 pm

Some more notes:
* You'll also have to set up the DNS server(s) manually (now you're getting them via dhcp) google is 8.8.8.8 and 8.8.4.4, cloudflare: 1.1.1.1 (IP->DNS)
* You are currently giving out (via your dhcp server -> networks configuration) 192.168.8.11 to your clients to usefor dns. Is this correct? Is this a typo? (I ask because Mikrotiks automatically propagate the DNS servers received via dhcp to its clients automatically, so if this is typo, you might not have noticed until we stop the dhcp client. If you really have a dns server on 192.168.8.11, that's totally fine - but nothing indicates this to be the case to me...)

lurker888 · Fri Feb 14, 2025 6:09 pm

Glad to hear it worked out!

anders2 · Fri Feb 14, 2025 6:17 pm

I run AdGuard on 192.168.8.11 to support my network and to filter content, its fallback is 1.1.1.1 - so that is correct

lurker888 · Fri Feb 14, 2025 6:31 pm

That's totally fine.

You will still have to do step 1 of what I wrote - in order for the router itself to be able to do "automatic" updates, access the IP->Cloud things (DDNS, time service...) You can set its DNS server to the Adguard, but personally I wouldn't. No use in having ad blocking for a router, and it just introduces another possible point of (hard to diagnose) failure.

Help with Latency problem needed [SOLVED]

Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed [SOLVED]

Re: Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed

Re: Help with Latency problem needed