Packet Storm

Stryker777 · Sun Sep 10, 2006 3:49 pm

We are replacing a 2.8.24 RouterOS system with a brand new system running 2.9.30. I am help with this remotely.
Configs are set up the same but for some reason we are getting packet storms shortly after adding the new system. Unplug it and replug the old one and everything runs fine again.
They have told me the interfaces on the new box are all good and the new box is being plugged into the same exact port on the switch.

It does appear there may be a loop somewhere on the bridged network (bridged at the switch not on router). They have STP turned on. When we turned it off they lost 8 ports on the switch. Sure sounds looped. Still, is there any reason anyone can think of that the old router runs fine but when the new one is in place a storm starts that takes the network down?

Just looking for input. It seemed a bit strange from here. I wish I was on site there to check it out lol.
Thanks

Stryker777 · Mon Sep 11, 2006 12:56 am

Ive gone through hours of logs from the network. Some with the old mikrotik some with the new one. There is definately a broadcast storm with the new machine but not with the old. The only real difference I see in traffic before the storm is, when running on the Old machine there is some IGMP traffic being logged. When on the new machine that is not happening.
Can IGMP not passing cause a storm? Does the new version handle protocols differently than 2.8?
Thanks

dancuofzhills · Tue Sep 26, 2006 7:13 pm

I Have a bridged network running stp. I was running rstp, but it makes the storm far worse than with no stp or regular stp.
I too am getting broadcast storms. I've turned off all redundant connections that i normally leave enabled for backup purposes, i've looked everywhere for a loop but can't find one.
These storms are taking most of my network offline, and i have been battling it for two days. Every time it seems like i've found the source i shut off access to that piece of hardware, and then the problem is back later.

Stryker777 · Wed Sep 27, 2006 1:17 am

I feel the pain lol. Still havent sorted this one out. It is 800 miles away so I can not actually see anything to find the issue. Have built another new system and same problem. The thing that is confusing me the most is why it works fine with the old one but not the new. The configs are the same.

sten · Wed Sep 27, 2006 3:51 am

Any of you guys running HP Procurve switches?
Procurves have a history of going nuts in some cases and then everything going through them will be broadcasted.
Maybe something similar with your switches?

This sort of thing tends to happen whenever the switch's bridge host table is getting close to full and the programmers used a hash table to store the table data. Often programmers will permit X number of lookups before broadcasting so that it wont loop forever on a nearly full table.

my 2 bits *shrugs*

Stryker777 · Wed Sep 27, 2006 9:13 am

The network i am working on is not. Thanks for the idea though. I had them swap out switches already, no luck. I'm confused lol. Wish I was down there to physically look at it all.

dancuofzhills · Wed Sep 27, 2006 6:34 pm

I am not using the procurve switch, but i have fixed my storm problem.. at least for now anyway. My border router(MT v 2.9.30) has multiple ip addresses assigned to it, and one of those addresses was 172.16.0.57. The primary ip for my router is 172.16.0.1.
i ran a program called iptraf (kind of like etherreal) and it shows this about 200 times a second:

Tue Sep 26 12:28:13 2006; ICMP; eth1; 56 bytes; source MAC address 001095791e1c; from 0.0.0.0 to 172.16.0.57; dest unrch (proto)

I got this coming from all different mac addresses, pretty much all the devices that i looked up the mac address for are cpe bridge panels on my customers houses. The panels are all tranzeo or zcomax hardware with atmel and marvell chipsets.

I do not really understand why having this ip address enabled causes the packet storm, i also dont understand how an icmp request can come from 0.0.0.0

The problem is solved for now, but since i dont understand what caused it i am worried that it will happen again.

If anyone can shed some light on this please let me know!
thanks

janisk · Thu Sep 28, 2006 8:31 am

172.16.0.57 and 172.16.0.1 on diferent ports on router?

if so - then here you go.

sten · Thu Sep 28, 2006 5:13 pm

I am not using the procurve switch, but i have fixed my storm problem.. at least for now anyway. My border router(MT v 2.9.30) has multiple ip addresses assigned to it, and one of those addresses was 172.16.0.57. The primary ip for my router is 172.16.0.1.
i ran a program called iptraf (kind of like etherreal) and it shows this about 200 times a second:

Tue Sep 26 12:28:13 2006; ICMP; eth1; 56 bytes; source MAC address 001095791e1c; from 0.0.0.0 to 172.16.0.57; dest unrch (proto)

I got this coming from all different mac addresses, pretty much all the devices that i looked up the mac address for are cpe bridge panels on my customers houses. The panels are all tranzeo or zcomax hardware with atmel and marvell chipsets.

I do not really understand why having this ip address enabled causes the packet storm, i also dont understand how an icmp request can come from 0.0.0.0

The problem is solved for now, but since i dont understand what caused it i am worried that it will happen again.

If anyone can shed some light on this please let me know!
thanks

routeros, responds to all ip addresses configured on the router on all arp capable interfaces, not just the interface the ip address is configured.
that alone wont explain the 0.0.0.0 addresses. however any ip capable host can transmit packets with any source address it wants. that is why you have ingress filtering on routers. ingress filtering is to filter source addresses to make sure packets dont have wrong source address.

dancuofzhills · Thu Sep 28, 2006 5:46 pm

I know that MT will respond to all ip addresses it is configured with, the reason i have the two ips is because one of them dst-nats to another computer on the other side of the MT router.
One thing i still wonder about is whether or not the problem was with my MT router, or with the other devices that were sending the packets. I am pretty sure it is the MT because it is less likely that so many other devices would start having a problem all of a sudden.

sten · Sun Oct 01, 2006 8:01 am

what's the possibilities of loops?

dancuofzhills · Mon Oct 02, 2006 7:30 pm

I have double checked and double checked again for loops and verified that there are none. Everything works great whe i leave the additional ip address of the router disabled. It is very weird... I normally do have redundant connections on my network that i let spanning tree control, but i had those all disabled for troubleshooting when i was having trouble. All my backup connections are live again with no problems unless i enable that alternate ip addy.

Packet Storm

Packet Storm

Broadcast storms

Solved for now..

Re: Solved for now..

N/A

Who is online