[UFO Chicago] help in investigating a possible packet storm

[Politik Durden]

Hello all, 

Going to a client site at 6 AM tomorrow because at about 5 PM today (Thursday) all network traffic started getting really really slow.

Here's what I know:

- no recent changes (no new switch, NIC, changes to static routes, config changes, patches/upgrades, etc)

- about a dozen switches feed into a 3COM switch (no model #s yet). ballpark of 2 to 3 hundred nodes total

- no protocols are used, all devices are in "dumb" mode and act as just a plain 'ol switch. some can be managed but no features (snmp, etc) are turned on.

- most nodes *seem* to be pingable from both sides of the firewall, but everything is just crawling. 

- nothing (reports, scripts, etc) is timing out, but everything is just super super slow.

They tried swapping out switches one at a time to narrow down the culprit and that helped for a bit, but then traffic slowed down again and they couldn't really do any more during production hours.

Theories: 

- Can one bad port cause this kind of a traffic jam ? They started diags on all the major nodes (server NICs, the central 3COM switch, etc) but nothing obvious so far. 

- Some sort of protocol/feature was turned on by mistake and now all the switches are confused ? A quick "topeka" (ha!!) points to stories of spanning tree causing these kinds of traffic jams.

- Somehow a loop got introduced ? 

What I really need is suggestions on a good free traffic tool, something we can install on two or three laptops and put each switch through its paces. Any ideas ? 

Thanks in advance for your comments. This lot always points me in the right direction :-)



      
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://ufo.chicago.il.us/pipermail/ufo/attachments/20100401/e9495585/attachment.htm