Network interface Rx Drops

slagjan · Tue Jan 19, 2010 2:17 pm

I have strange problem

I have a routerOS 4.2 using for pppoe server (local auth - no radius) with 2000 sessions. Trafic up to 50Mbps. PPS up to 10000pps
CPU load is up to 25% , RAM 2 GB (used 10%)
on ether interface (my public network card) im geting RX drops up to 200 000 per day . This is afecting some realtime services

Any idea about this problem/bug ?
Where should a look in to ?

Chupaka · Fri Jan 22, 2010 4:40 pm

how much CPU cores? do you use some queues?

slagjan · Mon Jan 25, 2010 10:44 pm

4 cores Quad AMD

i have upgrade to latest v4.5 , same things , more traffic more drops. I have change the network card but same again

Chupaka · Tue Jan 26, 2010 1:28 pm

queues? number of online users?

slagjan · Wed Jan 27, 2010 3:09 am

I use profiles with predefined speeds (128k, 512k, 1m, 2m, 3m, 4m)
every secret has profile which is giving client predefined speed and public ip address from pools

I have up to 1200 pppoe sesions active

Chupaka · Wed Jan 27, 2010 4:46 pm

1200 simple queues is madness. I think, you should move to PCQ:

create 2 pcq queues for each profile limit (upload and download), remove 'Rate' from the Profiles and add 'Address List'. then mark packets according to dynamic address-lists and send them to appropriate queues

slagjan · Fri Feb 05, 2010 7:00 am

Please can you give me some examples especially on mark packets according to dyn address list .
I have about 20 profiles .

Chupaka · Fri Feb 05, 2010 4:30 pm

/ip firewall mangle add chain=forward src-address-list=group1 action=mark-packet new-packet-mark=group1_upload
/ip firewall mangle add chain=forward dst-address-list=group1 action=mark-packet new-packet-mark=group1_download

where 'group1' is address-list stated in Profile

slagjan · Sun Feb 07, 2010 2:53 am

Thank you Chupaka

But the problem is still here
i had use second mikrotik to be usermanager and move all secrets there
now i have no local auth. or queues . Just radius controled pppoe clients

So now PPPoE server is just pure pppoe concentrator .
And still seeing rx drops there .
Any idea ?

Chupaka · Sun Feb 07, 2010 2:30 pm

was CPU load lowered?

slagjan · Sun Feb 07, 2010 9:25 pm

No
CPU load is same
I'm starting suspecting on hardware issues.

Chupaka · Sun Feb 07, 2010 11:20 pm

so now there's no more simple queues in your router?

slagjan · Mon Feb 08, 2010 1:20 am

no
just active pppoe sessions controlled from another mikrotik usermanager.
nothing else . no more local authentications

Mon Feb 08, 2010 8:22 am

Probably you still have 1200 dynamic change-mss rules in mangle - replace them by one global rule for all traffic.

slagjan · Mon Feb 08, 2010 5:00 pm

yes
i have there about twice more

how i do that ?

kostil · Mon Feb 15, 2010 10:57 am

I'm using this rule for my pppoe

/ip firewall mangle add action=change-mss chain=forward comment="Cut big packets for PPPoE" disabled=no new-mss=1440 protocol=tcp tcp-flags=syn tcp-mss=1441-65535

but I have same problems with Rx drops.
Then I use two integrated interface there is a little bit less drops then use only one. (ether1 - internet, ether2 - local clients vlans).

There are PPPoE,NAT,queue tree in my router and about 800 PPPoE clients with 200Mbit/s. Server is intel S3210SHLC with 1G RAM and Core2duo 3.0Ghz.

Any suggestions?

kostil · Mon Feb 15, 2010 11:09 am

4 cores Quad AMD

i have upgrade to latest v4.5 , same things , more traffic more drops. I have change the network card but same again

25% of 4 cores Quad cpu load is full load of one core. Mikrotik can't proper work with 4 cores. This meas that can used only 1/4 from your cpu. maybe this is a point.

I've trying 4 and 2 core processors on Intel and HP server - same result on all of them. If cpu is 4 cores - max load is 25% if 2 cores - 50%.

Look at your cpu usage graf, I bet it nether up greater than 30% on average.

sioannou · Mon Feb 15, 2010 2:01 pm

I have the same problem on some of my boxes.

Most of the times the problem came from having the speed and duplex auto-ngotiation on. What type of switch do you have connected to the box???
If it is a direct line set the speed and duplex your self and try again.

Some times the microtik reports that it is connected to 100 FDx but the switch on the other end is reporting 100 HDx.

Thanks,

Sotiris Ioannou

kostil · Mon Feb 15, 2010 3:11 pm

My mikrotik box connected to cisco catalyst 6500 series. I haven't see any duplex problems in cisco logs or mikrotik box. There is 1Gbit full duplex on both end.

sioannou · Mon Feb 15, 2010 3:15 pm

I've seen it with cisco before. Just try hard setting the options on both sides.

Sotiris

kostil · Tue Feb 16, 2010 9:59 am

this don't work for me

any other suggestions?

sioannou · Tue Feb 16, 2010 11:22 am

Hi Kostil,

Right the next step is the configuration on your switch and the configuration on the ROS.
I'm guessing the drops you have are due to a VLAN trunk. Most probably you have the prots connected to the router as a dot1q trunk. In cisco world the trunks transmit informational packets about states of the trunk on the native VLAN.
No this will show up under the interface of the ROS. So the next thing for you to try is to hard set the VLANs the port is member to and not specify that it is a trunk that allows all. If you want you can publish your configuration of the port and we can have a look.

Check the statistics on the switch most of the times this is followed by drops on the switch for undersize packets.

Thanks,

Sotiris

kostil · Tue Feb 16, 2010 12:18 pm

I have no drops or errors on my cisco switch. There is it port configuration

interface GigabitEthernet2/1
 description a05-nas01 uplink local
 no ip address
 switchport
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 7,54,2200-2279
 switchport mode trunk
 spanning-tree portfast

sh int gi2/1
GigabitEthernet2/1 is up, line protocol is up (connected)
  Hardware is C6k 1000Mb 802.3, address is fcfc.458d.acac (bia fcfc.458d.acac)
  Description: a05-nas01 uplink local
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 64/255, rxload 62/255
  Encapsulation ARPA, loopback not set
  Full-duplex, 1000Mb/s, media type is T
  input flow-control is off, output flow-control is on
  Clock mode is auto
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:49, output 00:00:43, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 2133
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 245293000 bits/sec, 49719 packets/sec
  5 minute output rate 251965000 bits/sec, 52171 packets/sec
     119410095471 packets input, 84457781207870 bytes, 0 no buffer
     Received 7823151 broadcasts (7714038 multicast)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     123184253878 packets output, 85537960094890 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

and the mikrotik port configuration and statistic

/interface ethernet
set 0 arp=enabled auto-negotiation=yes cable-settings=default comment="" disable-running-check=yes disabled=no full-duplex=yes mac-address=00:15:17:FA:CF:89 mtu=1500 name=uplink speed=1Gbps

/interface vlan
...
add arp=enabled comment="" disabled=no interface=uplink mtu=1500 name=127_248 use-service-tag=no vlan-id=2272
add arp=enabled comment="" disabled=no interface=uplink mtu=1500 name=127_249 use-service-tag=no vlan-id=2273
add arp=enabled comment="" disabled=no interface=uplink mtu=1500 name=127_250 use-service-tag=no vlan-id=2274
add arp=enabled comment="" disabled=no interface=uplink mtu=1500 name=127_251 use-service-tag=no vlan-id=2275
add arp=enabled comment="" disabled=no interface=uplink mtu=1500 name=127_252 use-service-tag=no vlan-id=2276
add arp=enabled comment="" disabled=no interface=uplink mtu=1500 name=127_253 use-service-tag=no vlan-id=2277
add arp=enabled comment="" disabled=no interface=uplink mtu=1500 name=127_254 use-service-tag=no vlan-id=2278
add arp=enabled comment="" disabled=no interface=uplink mtu=1500 name=127_255 use-service-tag=no vlan-id=2279

interface ethernet monitor 0 once 
                 status: link-ok
       auto-negotiation: done
                   rate: 1Gbps
            full-duplex: yes
  default-cable-setting: standard

Sorry I can't understand what you mean I must to try next. Could you post a config?

sioannou · Tue Feb 16, 2010 12:44 pm

Hello kostil,

Your configuration is correct as it is.

What I have noticed on the switch is that you have 2 interface resets. This can happen if packets queued for transmission were not sent within several seconds or sometimes a cable problem.

Also can you post a
sh int gi2/1 controller

Sotiris

kostil · Tue Feb 16, 2010 2:55 pm

it's the same as "sh int gi2/1".
2 resets is nothing if look at more than 6 million! drops in mikrotik interface.

There is some thing in mikrotik configuration I think. Maybe it's about queue tree?

queue tree with pcq config

queue tree print
0     name="Download" parent=global-out limit-at=0 priority=1 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s
...
7   name="4Mbit_down" parent=Download packet-mark=4Mbit_down limit-at=0 queue=pcq_4Mbit_down priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s 
...
19   name="4Mbit_up" parent=Upload packet-mark=4Mbit_up limit-at=0 queue=pcq_4Mbit_up priority=8 max-limit=0 burst-limit=0 burst-threshold=0 burst-time=0s 

queue type print
15 name="pcq_4Mbit_down" kind=pcq pcq-rate=4000000 pcq-limit=40 pcq-classifier=dst-address pcq-total-limit=20000 
16 name="pcq_4Mbit_up" kind=pcq pcq-rate=4000000 pcq-limit=40 pcq-classifier=src-address pcq-total-limit=20000

martini · Thu Feb 25, 2010 12:50 am

Problem is onbord LAN and low bus speed. I see drop even on 8 core server with PCI-X intel Lan.
When moved to Core-i5/i7 and PCI-Express Intel LAN drops are stopped. Before this have the same number of drops..

One more things - PCQ whith more than 100 pppoe users and traffic more than 200 Mbits dont work correct, especialy if rate more than 10 Mbits (even if you change Limit and Total-limit, on any version of ROS)

ну немогут они сделать балансировку по ядрам проца... возможно это и есть причина неработоспособности PCQ

ilirnako · Thu Mar 04, 2010 1:26 pm

I have the same problem .Too much Rx drops .The ports are Intel PRO 1000.

I think the problem is with Intel -drivers or issues related Mikrotik -Intel cards

Regards

Ilir

Chupaka · Thu Mar 04, 2010 6:16 pm

I think the problem is with Intel -drivers or issues related Mikrotik -Intel cards

in about two weeks you will be able to test RouterOS v5 - it has new kernel and new Intel NIC driver =)

0ldman · Fri Mar 05, 2010 4:25 am

I am having the same issues. Core i7 920, Realtek 8169 PCI gigabit, auto negotiation, forced gig, forced 100 half/full, same problem.

Once it does this, the Cisco kills the port for five minutes. I have swapped cards from 3c905c, same issues.

I think the issue is deeper. Could be something to do with VLAN's and trunking, can't argue that possibility, however, it has done it with only the 7200 router and a dummy gigabit switch.

Could it possibly be STP?

fatonk · Fri Mar 05, 2010 11:04 am

Hi,

I have faced the same issue, what I have done is that I disabled the single PCQ rule out of all rules in the queue tree and the problem was solved.

Regards.

Faton

janisk · Fri Mar 05, 2010 2:04 pm

if i have understood correctly - for gigabyte link you have to enable auto negotiation, because not only link speed is negotiated over the link. So there is no way someone can force 1GBit while not braking the link.

ilirnako · Fri Mar 05, 2010 8:09 pm

Faton

Please can you write me at inako@abcom.al or skype :ilirnako (flasim aty qe te sqaroj me konkretisht problemin )

Regards
Iliri

0ldman · Sat Mar 06, 2010 7:56 am

Hi,

I have faced the same issue, what I have done is that I disabled the single PCQ rule out of all rules in the queue tree and the problem was solved.

Regards.

Faton

Would you mind explaining that a bit?

fatonk · Mon Mar 08, 2010 4:03 pm

Hello,

I will explain in more detail:
I have installed a RouterOS based server for QoS delivery, and the setup involves lot of mangle rules that later apply at queue-tree rules. This server is configured in bridge mode, and it forwards around 900 Mbit/s of traffic, from all queue-tree rules only one rule had as a queue-type PCQ, where this rule was used to limit youtube per connection at 400 Kbit/s. But the problem started to occur when the youtube traffic started to exceed 250 Mbit/s and we started to face lot of rx drops what resulted with retransmissions etc, after having a look at queue-tree adding the column drop, it was obvious to see where the drops are occurring Youtube rule, after disabling it everything was fine, now youtube is limited differently.

Regards.

Faton

P.S. Ilir Nako write at fatonkurteshi@yahoo.com

0ldman · Wed Mar 10, 2010 9:28 am

So you're queue rule was actually causing the drops or was it PCQ itself? are you still using PCQ for the other queues?

fatonk · Wed Mar 10, 2010 10:28 am

No, I don't use PCQ for other rules, it was just one rule with PCQ and now I have none.

Regards.

Faton.

0ldman · Wed Mar 10, 2010 7:39 pm

Thanks!

I've been fighting with this problem for a while now. Hopefully removing the PCQ's will be the fix.

My PCQ don't limit the connection speed. If I understand it properly, only the number of connections.

meister · Mon Apr 19, 2010 2:44 am

Has anyone tried 5.0beta1 to see if the problem still exists? I am having the same issues.

I have tried, all with no success,
-changing from multi-cpu no to yes
-duplex/speed settings on switch and mt
-the customers vlans and the network vlan were on the same interface -changed them to different interfaces (helped a little)

The customers that are complaining are the ones using realtime traffic - voip ect.

keefe007 · Tue May 11, 2010 7:07 am

I'm seeing RX drops on a number of my interfaces as well. I'm running a dual-core system with version 3.30.

Keefe

nikhil · Wed Oct 06, 2010 8:53 am

If you use 1gigabit setting to autonegotiation reduces the RX drops substantially. I did this 2 weeks ago and the rx drops and tx drops have reduced to 1-3% of the original.

I stll want to take it to zero. I am seeing rx drops on the brodcom nics on our dell r200s and 2950s but no more on the intel nics. Using 3.27 on 2950 and 5.0rc1 on r200s with intel 1000pro-vt on 3.27 and 82576 quad port based nic on 5.0rc1

I would like to see this at zero as traffic flowing is not more than 150 mbps -200 mbps on the individual nics

NetworkPro · Fri Nov 11, 2011 9:56 pm

if i have understood correctly - for gigabyte link you have to enable auto negotiation, because not only link speed is negotiated over the link. So there is no way someone can force 1GBit while not braking the link.

Have someone tested what happens if we turn off gigabit auto-negotiation on both sides of the link?

Who is online