Hi Forum. I’m having issues with Windows 10/11 clients that use Mellanox 3 and Intel 10G SPF+ Cards going to a CRS326-24S+2Q+RM - this problem started randomly or maybe as I’ve deployed more clients or introduced something bad into the mix.
Sorry - I WILL POST CONFIG When I Get to the CRS326 - I am writing this from home
Network Topology
- CRS326-24S+2Q+RM switch as CORE Switch Running Router OS (will provide version when onsite)
- 2x 10G LACP DAC Links to a Cisco 48 Port PoE (Services Phones and Low Speed 1Gbps End User Clients)
- 2x 10G LACP DAC Links to a Ubiquity Enterprise 8 port PoE for Wifi Dishes
- 2x 1G LACP DAC Links to Ubiquity 24 port PoE (Services Phones and Low Speed 1Gbps End User Clients)
- 2x 10G LACP DAC Links to Windows File Server A (VLAN10)
- 2x 10G LACP DAC Links to Windows File Server B (VLAN10)
- 7x SFP+ Single Mode Fiber 10km runs to end user workstations at 10Gbps each (VLAN10)
- 2x DAC LACP Links to Unify Dream Machine+ (used for Wifi Network Controller and Door Controller)
There are 5 subnets / vlans - none of them are routed by the CRS326 (It’s being used, as a L2 switch with LACP features) - at least I hope I configured it this way in RouterOS
The CRS is only managed via the Management LAN interface, Management is not available to Userland.
VLAN 10 ( Userland )
VLAN 20 ( IP Phones )
VLAN 30 ( IVR / Cameras )
VLAN 50 ( Utility VLAN, Used Primarily to Trick Dream Machine into reaching Internet )
All the Gateways on said VLANS are on a EdgeRouter Pro, with also does internet NAT are are received untagged - by various ports, that are not on CRS326
Symptoms are as follows:
When a 10Gbps Windows Client is connected, internet browsing is impossible as it’s making requests - so it takes longer to load advertising banners on speedtest, then to run it.
Some Windows 11 Clients experience random freezing in the OS (with Intel (with Offloading) and Melanox Cards)
ICMP - Ping from Windows 10/11 10Gbps SPF+ Client to LACP DAC Windows Server A or B - Which are all on CRS directly shows random timeouts or ping times from 3ms to 600ms to unreach.
ICMP - Ping to Edgerouter Default GW “through” the switch to next - does same random long and short numbers 4ms - 1000ms
All 1Gbps Clients speaking to LACP DAC 10GB Windows Servers work just fine, same for EdgeRouter(internet load times) - All Wifi6 Clients work just fine.
So the only time things start getting wonky is when I directly wire a 10Gbps SFP+ Windows Desktop into the CRS, - ICMP Gets Weird, Machines Jitter with loading websites etc…
I tried Swapping Transceivers and Fiber runs on both ends, since each of those gets 2 pairs of fiber, Looked into multiple posts of people comparing booting Linux vs Windows on the desktop client,
and having these issues with other types of hardware or cards, purchased Intel Intel XL710-BM2 and it’s transceivers to eliminate Mellanox Single SFP+ card or drivers as the issue.
Funny thing is when I upgraded the core to CRS 10Gbps a year ago, everything worked fine, including clients deployed of Fiber, it is as if Microsoft or Unifi shipped some update that wrecked the NIC Drivers or similar or someone or something Introduced some type of Issue into the LAN/LANS that is prohibiting from it working properly. Or the 10km transcievers from FS/Mikrotik lazered themseves, or something happened to the CRS over time … No Idea. - Thus this forum post.
So at this moment I’m kind-of-stuck. I’m not onsite to download the config, but will be and will add it here - I believe I did everything right with having a single bridge0 on the CRS,
I’m not CLI savvy so it was all configured via MGMT interface using Winbox - Any ideas welcome, as I’m sitting and counting PCIe lanes between GPUs and Network Cards, some Clients are Dual Xeon workstations that clearly have enough lanes for ICMP not to do this, I’m certain that there is someone out in the universe that had this problem with 10Gbps Windows 10 or 11 desktop users connected directly to the CRS. Thanks in advance for any input.