Throughput issue; to/fm rb fine, passthrough=crap, ideas please!

RouterOS General
1

Explanation:

Laptop connect by cable (via power shot) to CPE antenna (Sextant, SXT or QRT).
CPE connects to AP wireless (802.11an + NV2 encrypted) and via AP to several backhauls and nodes to reach internet.
Speed limit for client only in the last router that connects to the internet provider’s network with plenty of capacity. In the internal network client has all speed that physically is available to its disposal (but shared…).

CPE antenna is router.
LAN side is wired to laptop has dhcp-server and hands out IP to laptop.
All LAN IP’s get masqueraded IP to leave CPE on wlan interface with IP from the CPE.
Wlan is radio-station mode and dhcp-client getting IP from AP. (So this is the client’s IP)
AP is just router and routes 0.0.0./0 ‘up’ towards next router towards border router etc.
Border router takes care of src-nat and issues public IP to client.
From border router a route is made back to the client hosting AP.
Whole chain of routers (ALL MT network) is ROS v6.27 with updated firmware..

Usually the speed a client can get into his PC, laptop or any device is slightly less than what the client’s simple-queue in the border router limit it to. (And off course shared in case of several devices)
If the simple queue says 10Mb than client will see 9,5 - 9,8Mbps max. (downloads, upload similar story but less)

Since some months some complaining clients; “no speed”.

Standard procedure is to log into the CPE and perform a MT-bandwidth test (tcp) to border router first, than to a big router in front of it (so no queues effect ) and ultimately to the IP of the AP or router attached to AP.

First test would reveal if the speed is limited by the queue.
Second test would show if the route from CPE towards border router is having enough capacity to serve more than demand.
Third test is to see if the channel capacity of the client-AP part is sufficient.

Usually these all show good and works fine for 95% of clients.
But this client still complaints. So we pay the visit and hook up our laptop directly to the to the CPE-LAN of client (bypass his wifi router this way).
first we log into CPE and perform earlier tests again, but now from local. All good.
Then we perform MT windows based bandwidth test towards same routers; no speed at all. Some kbps… sometimes 1 or 2 Mb where the client has right to 6-7 or more… what’s happening?

So maybe the cable is crap?
Perform MT windows test from laptop towards CPE; 90Mbps download, 60-80Mpbs typical upload.

[Questions:
Should this be closer to 100Mbps if this is normal ethernet and cat5e cable?
Should this be close to 100Mbps in BOTH directions at the same time? Never seen that…
If this is only 60 down (towards laptop) and for instance 30 up, means there is so many losses that if client is assigned 10Mb several nodes away (and they all have some losses) that 10Mb will never make it due the combined losses?]


Download tcp test initiated from laptop from AP (download only) = < 2Mbps, from AP attached router, same…
Download tcp test initiated from CPE from AP (download only) = 18-24Mbps.
udp test are almost similar but initiated at CPE udp download from AP to CPE shows 40-60Mbps depending on other client’s traffic. (Basically the channel capacity is sufficient)

There are no firewall rules in CPE
There is only masq rule in nat.
There are no mangle rules in place.


I have several other similar routers in similar setup that don’t have same problem.
So, WHAT IS HAPPENING? I am left without any clues. Clients unhappy, I am unhappy and so far can’t solve the issue.

We changed cables, no help.
We changed CPE antenna, only in some instance it helped. Same antenna at other client had no issue.
We default config’d and rebuild config of CPE, no help.
We changed AP for that client, in some instances it helped. (? But why? Why should the throughput “over” an antenna be influenced by the AP that it is connect to?)

Please vent your ideas or advises. I need to tackle this…

2

Don’t get tricked by the CPU limitations on these devices to flood a connection with btest.
Watch the CPU on the AP when doing a btest and if it hits 100% during the test, you’re actually pegging the CPU and not the network.

If you have a powerful enough dedicated btest server at each tower, then you could test to those and not slam the CPU of the AP.

An even more clever trick would be to create an anycasted address on these “tower” test points - so that you can tell techs / customers to test to the anycast address, which will automatically be the nearest test point. You can use the unicast address if unsure which test point they’re hitting, or to specify a certain one.

If you’re sure it’s not the CPU issue on the AP/routers you’re testing to, then I would start looking for telltale signs of interference / poor link quality. Look for rapidly changing tx/rx rates, rates that dip to basic rate, etc.

Test from actual ethernet port at tower site (if possible) towards the Internet to make sure the tower’s backhaul is not the problem, etc.

3

I know that. I always look at the cpu while performing a test. But even if the cpu shows 100% while I test a download from AP towards CPE that comes in with 20+Mbps where same test performed from a laptop sitting behind that CPE only gets 1-2Mbps from same AP (and now cpu reads only 20% or so) it means imho something goes wrong at the CPE?

An even more clever trick would be to create an anycasted address on these “tower” test points - so that you can tell techs / customers to test to the anycast address, which will automatically be the nearest test point. You can use the unicast address if unsure which test point they’re hitting, or to specify a certain one.

Can you give me an example how to do that. “anycasted address”?? I never understood the many types of “cast” anyway. So help me out because it could give me tool to handout to customers. www.speedtest.com becomes more and more unreliable by the day it seems. Probably over usage too.

If you’re sure it’s not the CPU issue on the AP/routers you’re testing to, then I would start looking for telltale signs of interference / poor link quality. Look for rapidly changing tx/rx rates, rates that dip to basic rate, etc.

Test from actual ethernet port at tower site (if possible) towards the Internet to make sure the tower’s backhaul is not the problem, etc.

Well, I know basically all my links have more or less issues with interferences. There is not so much I can do about. We have in a small (15km circle) region probably around 60 to 70 AP’s and/or backhaul links in the 5Ghz band so no matter where I look there is almost always nearby usage in the channel of choice.
It’s a big hassle to find over and over again relative free frequencies for my AP-client networks and the backhauls feeding them.
Most AP’s have at least 3 backhauls needed to reach my central and it very rare all backhauls towards central have above 90 or close to 100% CCQ all the time.
AP-Client networks seldom reach these levels. I am happy to see 60-80% CCQ ranges.

And yes indeed, measured over the whole link from client towards my central I am wondering if the combined losses of each wireless segment prevents customers to get lets say 6-10mb download where within my network speeds are unlimited and backhaul links themselves usually have over 150Mbps conn. rates.

But in this particular issue I have client that can have their 6-10mb downloads without problems where other clients in the same AP-network can have same speed towards their CPE but NOT towards their LAN…

I now even have some clients saying this issue developed over the last weeks where I am sure nothing changed…

4

That’s a logical conclusion. What does a test to the CPE from this laptop reveal?
(sorry, lots of details in this thread, so if I missed this detail, then I apologize)

Well, I find that speedtest.net gets very different results based on which test site. I tend to use Atlanta GA because that’s where my Internet link comes from. The script auto-detects an endpoint in my own city, and it’s actually farther away from me in cyberspace. Most end users don’t know how to know which site is really the best one to test with.

As for anycast, basically, you choose an anycast address and put it on a loopback interface on several Mikrotiks (e.g. 10.10.10.10/32) then you connect them to your network in various locations, but with a local IP at each POP. Then you have OSPF on every one of these mikrotiks advertise the 10.10.10.10/32.

This way, ospf will route to the nearest one.

5

Download test performed on CPE from AP (or router higher up) gives 10,15 or higher speeds.
Same download test performed from laptop connected to the ethernet of the CPE gives 1, 2 sometimes 4 or 5 and at times even higher results.
MT windows download test performed from the laptop but traffic from CPE itself reveals high speeds. Download around 80 or 90 Mb and upload some 50-70Mb (I think now cpu of laptop is limiting factor)
Anyway, it seems the cable connection seems to be fine. (But than again, should I not measure 95-100 both up and down? After all the ethernet status shows conn. rate of 100Mbps with full duplex?)

Could it be that if I measure only 70-80Mb download taken from CPE over the ethernet cable towards laptop this means 20-30% losses?
If now a tcp download is performed through that CPE over a wireless link that also shows some 20-40% losses it means these combined losses mean the poor performance?
Ethernet status shows an ethernet link that is good. But what is good? Just the electrical connections are ok (which means with 100Mb only the 2 pairs are connected right plus the others deliver the poe) and the static signal is ok. But the moment traffic flows for all kind of reasons the actually data has dropping packets downgrading the link fast…
This can be poor connector too. We have 3. 1 in the CPE, 1 to connecto to the powershot and the powershot itself into the wifi router (or laptop).
What is only one of the tiny connectors of a ethernet plug or socket has a slightly corroded or poor connected pin?

Otherwise I can’t explain the issue.
We tried other cable, but not a real test case. The test results are all over the place at times anyway. The failing links at times suddenly seem to work perfectly too…

6

Your troubleshooting seems pretty sound to me.
I agree that if a direct connection is only getting 70-80% of expected capacity, then that’s going to get really amplified over connections to the Internet in general. (it wasn’t a cpu bottleneck when it pegged out at 80, right?)

And just to be sure, there’s not some crazy-wild set of queues and mangle rules that apply to through traffic but not in/out traffic, right?

7

Just a short question: what speed the interfaces have?
I mean, both CPE and 'puter is fast ethernet or both gigabit? or mixed?

Have a try with same kind! NV2 has issues with ethernet sync if CPE is capable of GE and 'puter is FE or vice versa.

8

I watch the cpu closely and if it might be a factor I try an attached router to by pass. But when the AP cpu stays below 50% while running a download toward the laptop and only 1-3Mb reach the laptop where the cpe downloads with 15-30Mb’s and the cpu of AP hits 100% the issue is not the cpu in the download towards laptop.

I erased all firewall filter, nat and mangle rules and checked there are no queues in place. In the CPE the only rule set is the masq. for lan traffic leaving the wlan interface. And that is standard on all my 500+ CPE’s…

9

OK, this sounds interesting I need to explore. Most CPE in my network are still having fast ethernet (100M) interfaces but since some time we have the gigabit interface equipped SXTs, SEXTANTs and QRTs entering our network. But indeed all the wifi routers we use in 99% of the client installations only have fast ethernet equipped ports…
So this is worthwhile investigating…
I think I have to set the ports ‘manual’ and only 100M connection. See if that will solve the issue. And if it is I will make a report to MT.
I’ll let you know how this goes on…

10

On fast ethernet you should be getting 94Mbps with an iperf TCP test. I’ve been able to hit that with really old (2003-era) PC hardware so I doubt your laptop is the bottleneck unless the speed test is inefficient (Flash or Javascript testers would fall into this category) or you’re punishing your field techs by making them use prehistoric laptops :stuck_out_tongue:

11

“iperfTCP” test? What is that exactly? What do you mean?
I use the MT bandwidth test tool. Its not very good but it has its purpose.
Hmm, the laptop this tech uses is an old panasonic toughbook, with winXP so yeah, not the latests. You might call it “prehistoric”
He sort of ‘eats’ a laptop per half a year so I stopped giving hem a new one… But heho, I can give him mine for the test.
I think we should also developpe a proper 20-30mtr ethernet cable that is proven to be compliant with the giga standard. This way we can use this as a thresh mark cable to compare towards his on spot made utp/ftp cables.
After reading some articles online I thing there is more into cable connections to go wrong than just make sure the 8 wires are properly connected…

But this week we have to setup a test for the previously mentioned ‘sync’ issue as well. It sounds very plausible to me that MT boards that have gigabit ports don’t work well with fast ethernet equipped devices.
Or maybe even the interface queues come in the picture now? We set since half a year all boards both wireless and ethernet interfaces by default to “only hardware queue” but maybe this give now an issue with gigabit ports connected to fast ethernet devices?
Actually I have seen more issues with the ports lately so who knows… what is the best practise anyway?

12

Fixing the rate do nothing, however i don’t know the root of this problem. Maybe backpressure or timing issue.
This is not exist with nstreme, only with NV2.
Easily reproducable within bridging network.

13

I’d presume you mean when the failing port is part of the bridge? My network is mainly routed, but here and there I do make use of bridge.
In this specific case the CPE is router, LAN and wlan are part of different network with src-nat = masq rule.

The problem on our side is also that the issue is not always there. So how to reproduce…?

14

in my experience:

OK:
Internet uplink-(GE)-PC Router-(GE)-Switch-(GE)-433GL → (wireless) → 433GL-433GL->(wireless)->SXT->CustomerPC

Also OK:
Internet uplink-(GE)-PC Router-(GE)-Switch-(GE)-433GL → (wireless) → 433-433->(wireless)->SXT->CustomerPC

NOT OK:
Internet uplink-(GE)-PC Router-(GE)-Switch-(GE)-433 → (wireless) → 433GL-433GL->(wireless)->SXT->CustomerPC
NOT OK:
Internet uplink-(GE)-PC Router-(GE)-Switch-(FE)-433 → (wireless) → 433GL-433GL->(wireless)->SXT->CustomerPC
NOT OK:
Internet uplink-(GE)-PC Router-(GE)-Switch-(GE)-433GL → (wireless) → 433-433GL->(wireless)->SXT->CustomerPC

the issue exist only with NV2, and not with NStreme.

What is strange, the uplink(customer->internet) is higher than downlink (internet->customer)