Hello,
I would like to share something we have been experiencing recently on our network;
we are a small WISP; in one of our networks we are currently doing traffic shaping only centrally in our core (server room), and not at cpe level; usually it works fine but since a couple of days we are noticing that some customers have probably compromised CPEs (mostly UBNT) and/or PCs and are flooding us with some strange traffic; yesterday I torched a customer and found she was constantly flooding us with 10Mbps traffic;
as shaper we are using a CCR1036-8G-2S+ with ROS 6.15
(after the traffic is processed by its queue tree, it is passed to another CCR1036-8G-2S+ with 6.15 that is doing only NAT and therefore the uplink packets get a public IPv4, and are further passed to the next CCR which is our BGP router (CCR1016-12S-1S+ with 6.23) facing our peers, so summarizing we are using three CCRs in our core, and it is actually working fine; we currently have 600Mbps to the internet, but the bottleneck is our internal network, which we started building in 2007 and that has some old backhauls and legacy 802.11a APs (20Mbps available on each ath)
both the shaping ccr and the natting ccr are so using just 2 interfaces (the SFP+), in and out, at any moment RX bandwidth from IN should roughly be the same as TX bandwidth to OUT interface
BUT, when some customers are attacking us with garbage, it gets just to our shaping router and then stops and goes nowhere, therefore we notice the bandwidth usage is not symmetrical there, for example right now RX from customers is 18Mbps higher than TX to upper-level CCR
THE QUESTIONS ARE:
How to find the IPs of the customers flooding us?
what is this garbage traffic looking like?
which kind of application / botnet / attacker is generating it? we think there is a chance it could be intentional (from one of our competitors…)
When we find the IPs, how could we block that traffic, other than disconnecting the user at AP level?
If we can’t completely protect us from such kind of attacks, how could we mitigate their potential effects by implementing some fw rules, blacklisting connections etc.?
at the moment our nat-ccr has no fw rule at all, our shaper-ccr has a dozen of simple fw rules, on both of them average cpu usage is 1 / 2% (on our bgp average is 1%, except on startup when loading the full bgp tables); current uptime of everything is 119d, so this architecture seems quite stable and I would avoid any ROS update unless necessary, I know for DoS protection upgrading to 6.21+ would be better
in future I would like to upgrade all three CCRs to some upcoming 1036/1072 variants having dual removable PSUs, for peace of mind; seeing the average cpu level the perfect future devices for me would be actually just 16 cores, in order not to generate unnecessary heat, a perfect all-round device would be a a CCR1016 with 3 SFP+, 3 SFP, 2 copper ethernet, no display, dual hot swappable CPU (outside of our datacenter we would need the same device in a passive cooled variation, with on the rear TWO wide-range DC inputs for external PSUs, ideally one should be a terminal block for easier integration
our internal network is flat, bridged, so network control is done only centrally, at cpe level we have only masquerading from their internal subnet to a single private transport IP; we plan about segmenting the network using vlans, and probably implementing shaping at cpe level too, but it becomes more complex to manage and mantain
The problem in this situation is not that anybody is really generating too high traffic to the Internet, since the traffic is effectively blocked from our queueing ccr (so a customer with a 7/1Mbps plan can send only 1Mbps and not 10Mbps!), but:
- the customer is not aware of the problem in his internal lan or cpe and just feels that usable uplink speed for him is getting near to zero;
and even more important: - wasting 18Mbps throughput on an old 802.11a AP (and/or on the backhaul connecting it to our core) means there is a near-DoS situation there for other users connected on the same AP, they start noticing low bandwidth, high latency, packet loss
The screenshots are showing such low bandwidth levels because I have taken them tonight around 6am; anyway the 18Mbps delta was roughly the same also yesterday evening at peak time.
A long term solution would be of course adding queueing / bw shaping also on each CPEs, blocking that traffic at an early stage
Thank you for any suggestion
Leo
Attached:
- at shaper ccr, strange traffic from customers, 18Mbps more than expected
garbage_from_internal_network.jpg - at natting ccr, everything normal
no_garbage_passed_to_nat.jpg