Have you seen a CDN overloading a customer? Help me gather information on the issue.

What CDN?
What have you identified the traffic to be?
What is the access network?
Where is the rate limiting done?
How is the rate limiting done (policing vs. queueing, SFQ, PFIFO, etc,, etc.)?
What is doing the rate limiting?
What is the rate-limit set to?
Upstream of the rate-limiter, what are you seeing for inbound traffic?
One connection or many?
How much traffic?
How does other traffic behave when exceeding the rate limit?
Where is NAT performed?
What is doing NAT?
Shared NAT or isolated to that customer?
Have you done a packet capture before and after the rate limiter? The NAT device?
Would you be willing to send a filtered packet capture (only the frames that relate to this CDN) to the CDN if they want it?



There have been reports of CDNs sending more traffic than the customer can handle and ignores TCP convention to slow down. Trying to investigate this thoroughly so we can get the CDN to fix their system. Multiple CDNs have been shown to do this.

A CDN is just a web host that distributes a site or large files over a wide area. They only send out what is requested. If your throttling isn't working, it is because you have it configured wrong. The CDN will send out as fast as the available bandwidth. They are never expected to slow down - as that is by definition what they are supposed to avoid.

A CDN is just a web host that distributes a site or large files over a wide area. They only send out what is requested. If your throttling isn't working, it is because you have it configured wrong. The CDN will send out as fast as the available bandwidth. They are never expected to slow down - as that is by definition what they are supposed to avoid.

That isn't entirely correct. I know what I'm doing. Please just stick to what I asked.

Many are starting to implement FastTCP and some of them are eager to find out if theirs are at issue and to work on a remedy.

Where is the question?

https://goo.gl/forms/LvgFRsMdNdI8E9HF3

I have made this into a Google Form to make it easier to track compared to randomly formatted responses on multiple mailing lists, Facebook Groups, etc.

https://docs.google.com/spreadsheets/d/ ... sp=sharing

I have made the anonymized answers public. This will obviously have some bias to it given that I mostly know fixed wireless operators, but I'm hoping this gets some good distribution to catch more platforms.

I am also confused by the terminology you use. I agree with IntrusDave. CDN is a cloud system that hosts files. Clients request files as needed. CDN can't overload nobody.
Example CDN https://www.maxcdn.com

If you mean something else, please clarify what you are talking about.

I mean exactly what I am saying. I saw it once on my network, but it's been worse and on-going for others. CDNs have been modifying TCP to be more aggressive through methods like Fast TCP. In my situation, Microsoft's CDN had 190 connections open sustaining 2x my customer's rate limit upstream of the queue. This crowded out all other traffic. In other cases people have seen them start with a high window size and maintain a high window size for the duration of the transfer, regardless of what the receiver returned. Some people have seen over 30 megabit/s coming into a 1.5 megabit/s queue sustained for several hours.

In my opinion, you have the whole situation backwards.

Your 190 or something customers are simply all browsing the web. Nearly all modern websites include Jquery or similar JS files in their head, so everyone queries the same CDN address at the same time. This explains why you have so many requests to (not from) this server.

No, single customer, 190 connections to the same Microsoft IP.

There is also a thread on Whirlpool (Australian tech site) of people with packet captures, though they may only be on the client side and not the service provider side.


Normis, I know what I'm talking about.

I have handled way too many calls from customers at the WISP I work at where Microsofts Windows 10 update was eating up their bandwidth and overloading their rate limit by a misbehaving CDN (usually limelight networks). The CDNs are starting to abuse TCP to get data to their customers as quickly as possible. Just try a firewall filter for any TCP ACK packet with bytes 65-1500 -- whois that IP and it will almost certainly belong to a CDN. My guess is that if you shape downstream traffic at the core this is less noticeable -- however a lot of WISPs will shape at the CPE which means the AP and Backhaul are receiving far more throughput than the client is limited at.

I have handled way too many calls from customers at the WISP I work at where Microsofts Windows 10 update was eating up their bandwidth and overloading their rate limit by a misbehaving CDN (usually limelight networks). The CDNs are starting to abuse TCP to get data to their customers as quickly as possible. Just try a firewall filter for any TCP ACK packet with bytes 65-1500 -- whois that IP and it will almost certainly belong to a CDN. My guess is that if you shape downstream traffic at the core this is less noticeable -- however a lot of WISPs will shape at the CPE which means the AP and Backhaul are receiving far more throughput than the client is limited at.

I am in conversations with Akamai, Level 3 and soon Microsoft. I'll likely approach LimeLight as well. The ones I've talked to so far are anxious to fix it on their end. Wherever you shape it, it just moves where the problem is. That's why I'm working with the CDNs to fix the problem. Please fill out the form I've linked to if you haven't.

So you are saying that Windows update opens 190 connections to the CDN. So CDN is not the problem, Windows update is.

*sigh*

The CDNs are implementing a variety of modified TCP stacks that are not working out so well for many operators. The impetus is on them to fix it.