[UFO Chicago] help in investigating a possible packet storm

[Greg Neumarke]

On 4/1/2010 11:15 PM, Politik Durden wrote:
> Hello all,
>
> Going to a client site at 6 AM tomorrow because at about 5 PM today 
> (Thursday) all network traffic started getting really really slow.
>
> Here's what I know:
>
> - no recent changes (no new switch, NIC, changes to static routes, 
> config changes, patches/upgrades, etc)
>
> - about a dozen switches feed into a 3COM switch (no model #s yet). 
> ballpark of 2 to 3 hundred nodes total
>
> - no protocols are used, all devices are in "dumb" mode and act as 
> just a plain 'ol switch. some can be managed but no features (snmp, 
> etc) are turned on.
>
> - most nodes *seem* to be pingable from both sides of the firewall, 
> but everything is just crawling.
>
> - nothing (reports, scripts, etc) is timing out, but everything is 
> just super super slow.
>
> They tried swapping out switches one at a time to narrow down the 
> culprit and that helped for a bit, but then traffic slowed down again 
> and they couldn't really do any more during production hours.
>
> Theories:
>
> - Can one bad port cause this kind of a traffic jam ? They started 
> diags on all the major nodes (server NICs, the central 3COM switch, 
> etc) but nothing obvious so far.
>
> - Some sort of protocol/feature was turned on by mistake and now all 
> the switches are confused ? A quick "topeka" (ha!!) points to stories 
> of spanning tree causing these kinds of traffic jams.
>
> - Somehow a loop got introduced ?
>
> What I really need is suggestions on a good free traffic tool, 
> something we can install on two or three laptops and put each switch 
> through its paces. Any ideas ?
>
> Thanks in advance for your comments. This lot always points me in the 
> right direction :-)
>
>
Don't discount the possibility that one of the client stations is 
causing the problem. There could be a virus or trojan infection, an 
install of a Peer-to-peer file sharing app that is sucking up all the 
bandwidth. Even a lot of people trying to watch streaming video can slow 
things down. (March madness?)

I would check the switches that can be smart and get those features 
turned on. You might also check for updated firmware for the switches. 
Then use whatever monitoring ability is built into the switch to see if 
any one port is producing a lot of traffic. This might be accessed by a 
web interface on the switch, a telnet session, or even a serial port 
connection to the switch.

As you said, a loop can also cause trouble. You might try to check 
everywhere there is a switch and make sure nothing is plugged in 
incorrectly.

Another possiblity is a Denial of Service attack from the outside. Does 
this only affect internet connections or are internal connections to, 
say, file servers, slow?

-Greg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://ufo.chicago.il.us/pipermail/ufo/attachments/20100402/b30cce22/attachment.htm