Hi!
After a lot of troubleshooting, I managed to identify that the root cause for my IPTV issues look to be the drops on my final switch before the client.
Additionally, the drops are happening on the CRS304 path, but I haven’t been able to capture those yet, except visually. They last half a second and are almost impossible to catch.
Traffic from the router to the first switch is clean. It starts dropping on the CRS518. If I turn packet capture on that very switch, the packet loss is instantly shown, regardless of what I do and reports the following:
IPv4 total length exceeds packet length (1453 bytes)]. MTU is 1500 across the board.
The only actual issues I visually experience are with realtime traffic (IPTV).
Network topology attached.
All switches are on the latest firmware. HW offloading does not seem to make a difference. default configuration with no other changes besides HW offloading test, static IP and negotiation where applicable.
Exports from the switches on the path attached.
I can provide a wireshark capture if you think it would help.
Any assistance appreciated.
admin@CRS504.txt (5.01 KB)
[admin@swSoba100G] export.txt (2.52 KB)
Am I seeing UDP encapsulated in TCP? If yes, why?
Traditional IPTV is seven MPEG-TS packets at 188 octets each per L2 frame, thus well under 1500 even after UDP framing overhead. (Inconveniently, 8×188=1504)
I get these errors without IPTV on as well. The only thing I need to do to reproduce is turn on packet sniffing on the interface.
How are you running a packet capture from the CRS518, and what does that switch’s CPU usage look like while you are doing it?
You have to send all traffic of interest through the CPU in order to make it visible to “/tool/sniffer”, in which case your 100G switch becomes a ~400M router.
Your test is likely to cause more problems than is paid for by fresh enlightenment.
Far better to run the sniffers on the endpoint PCs.
I’m running the capture like this:
I have discovered that all of my switches except the 518 have TX drops. It seems that bufferbloat may be in play here.
It’s not bufferbloat. That’s a routing phenomenon.
The closest switching equivalent — which is in fact a likely explanation here — is not setting up Ethernet flow control appropriately on the boundaries between one network speed and another.
Within a given L2 speed domain, buffering is unhelpful, backing my prior statement. Between speed domains, you still don’t want buffering, but you must do something to cope with the fact that some senders are faster than some receivers. You see this with IPTV because it’s UDP, which lacks its own flow control mechanisms.
That said, single IPTV streams shouldn’t be approaching even 1G, much less the 25 and 40G link speeds you show.