Hello all!
So I’m not sure where to go with this issue I’ve been having. In some ways it seems like a BH issue, but surely it must be something else. A short summary of our network: We use /30 networks between all hops and OSPF for routing. All BH’s in this situation are licensed and can do 200+ mbps full duplex. The routers involved with the situation below are as follows:
site A CCR1036-12G-4S
site B CCR1016-12G
site C rb1100AHx2
site D rb1100AH
“other site” (connected directly to site B) 11 rb1100
I can move about 250mbps of tcp traffic over the link between site A and site B. I can do ~250mbps tcp speedtest from site A to site B. I can also do this from Site C (all the way to A) which connects through Site B. However if I do this test from D to site A (which goes through site B and C) I only get about 90mbps. Even though I can do about 210mbps from site D to site B.
Now, there is another tower that also connects directly to site B, I can do about 200+mbps tcp test from this other site to site B just fine, but just like From site D in the other example, I cannot get more than about 80mbps through site B to site A. So the common denominator seems to be something at site B.
I’m all ears on this and perhaps I’ve overlooked something. But I have double checked CPU loads on all routers, all physical interfaces between all sites for errors, duplex mismatches, and routing errors. But I have not been able to identify an issue. During peak times we don’t move more than 220mbps over the site B to A link. And as a note, when calculating bandwidth tests I was watching the physical interface to see all traffic, not just data generated by the routerOS bandwidth test.
Any help will be greatly appreciated, and hopefully I simply over looked something. Thanks in advance!
Omally01…
What are the latency figures like on the various hops? How do those figures look under the loads described? Are you seeing packet drops under load?
Thanks for the reply, below I have a traceroute from my computer which is one hop from Site A. The data below is from running the download test from site C in order to maximize load on the network, since doing the test from D would only add about 65mb to the load for a total of about 90mb . I also ran these tests from D, but decided not to post them since they latency figures did not look different from the test below.
Traceroute without load
1 192.168.20.1 (192.168.20.1) 3.535 ms 3.700 ms 4.481 ms
2 site A 5.391 ms 5.708 ms 5.865 ms
3 site B 7.060 ms 7.270 ms 7.634 ms
4 site C 9.000 ms 9.074 ms 9.519 ms
5 site D 10.968 ms 11.274 ms 11.290 ms
traceroute with max load
1 192.168.20.1 (192.168.20.1) 1.041 ms 1.184 ms 2.230 ms
2 site A 2.911 ms 2.984 ms 5.079 ms
3 site B 5.470 ms 7.199 ms 7.401 ms
4 site C 8.684 ms 8.873 ms 8.955 ms
5 site D 13.882 ms 13.915 ms 13.975 ms
During the test there was no packets dropped based on the dropped packets stat on the mikrotik bandwidth test. I also confirmed that there was no dropped packets on the physical interfaces, ethernet and wireless. Based on the results above it would appear everything should be fine, not sure what to think. Thanks again for the help.
If possible, try to do the tests from both a point beyond A and a point beyond D. Involving the routers in generating or sinking TCP traffic affects the accuracy of the results.
Also do a comparison with UDP. Since UDP does not use windowing it should not be affected by latency etc.
Sorry for the delay in getting back, and thanks for the help!
ok, so I did a test between routerboards that were one further “out” in both directions.
Doing a test from Site E (of course is connected to site D) to a routerboard on the “other side” of site A
This was not a full duplex test, each direction was tested on it’s own, and the result listed is actual data over the wire, not the bandwidth test chart data.
RX TCP ~65mbps
RX UDP ~220mbps
TX TCP ~85mbps
TX UDP ~262mbps
I’m glad you had me do this, I never bothered to try a UDP test before, though still not sure what do make of the results.
Let me know what you think, thanks!
Omally01…