I am noticing slow download speeds between one CCR1009 and CCR1072. Uploads are fine.
This is how they are connected.
CCR1009(6.44.3) --MPLS(Vodafone)–CCR1072(6.44.3) (at datacenter)–Internet(two uplinks)+peering
The clients are behind CCR1009.
Interfaces between CCR1009 and CCR1072 and towards the internet are SFP.
Clients connected with LAN. Static public IP routes between 1009 and 1072.
The download speeds on speedtest.net and fast.com are horrible, uploads are fine. Browsing is fine too. Downloading large files like Linux updates or package installation gets throttled.
Here is what I have done so far.
FastTrack
disabled peering
Used either of the uplinks
Removing NAT at CCR1009
None of the above helped.
Firewall rules are very basic. Dropping some port 25 and 53 incoming packets.
Hope someone might have encountered the same issue. So asking a question here.
I’d first check whether the throttling is not a matter of hardware errors or ill-configired MPLS tunnel at Vodafone side. So I’d use packet sniffing at both ends to see whether all data received at 1072’s uplink make it to the interface to which the MPLS link is connected, and whether what leaves the 1072 makes it to the 1009’s uplink interface and further to the client. This should tell you where the throttling happens. If you can see some packets sent by the 1072 to be missing at the 1009’s uplink interface, it can be a bad cable on either end or a wrong bandwidth limit configured at Vodafone side; if you can see Rx errors in the statistics of 1009’s uplink interface, it’s a bad cable there, but you cannot distinguish a bad cable between the 1072 and Vodafone’s MPLS box from misconfiguration at Vodafone.
We suspected Vodafone MPLS so got another MPLS link from a separate provider. The issue is same with second MPLS provider too.
Not losing any packets at either ends. No tx or rx error on the interfaces either.
So if you sniff at both interfaces at each machine into a file, what difference between timestamps of the packet received on uplink and the packet sent on downlink you can see in Wireshark, and what difference can you see between a download packet and it’s ACK packet in upload direction? If you don’t lose packets on the way, they must be queued, and thus delayed, somewhere.
I’m afraid that without packet sniffing it is impossible to identify the issue, as you need to analyze how the various speed tests algorithms differ from each other. There may be some MTU/MSS issue, or some TCP window size adaptation algorithms, or sending packet rate variations.
I’ve compared one run of speedtest.net with one run of nperf.com, on a quick look there is a visible difference during the upload test, where nperf.com keeps sending with high packet rate even though it means a lot of lost packets and retransmissions due to bandwidth limitation whereas speedtest.net accommodates to the channel bandwidth and lowers packet rate to avoid retransmissions. So I suppose the remote server behaves the same way in the download direction.
Other than that, nperf.com uses four TCP streams per direction whereas speedtest.net uses eight (I’m pretty sure it was using just four some months ago but by then my uplink bandwidth was lower).
But it was just a single run of each tool and I didn’t dive deep into the analysis. Wireshark is your best friend here.
At our CCR1072 end we have untagged VLAN given to us by the MPLS provider(Multiple Untagged VLAN for multiple remote locations).
We notice some packet loss on these links.
Here is the packet capture of the 2 packets (one where the reply was received and one where the reply was not received).