Network grinds to a halt- where to start ?

[ssAPT]

Ok so I am not really a network guru but I know enough to create a VLAN or 2 on Procurve switches etc. My network consists of about 50 users , 6 servers incl exchange and file/print and a single Procurve 5406zl switch

Last friday my network just came to a halt. Access to servers was slow and then died completely. Once I was satisfied it wasn't the servers I just rebooted everything including the switch

Now my question is what commands could I have run on the switch to try and get an idea of what was going on and hopefully lead me in the direction of the problem ?

I know PCM+ is a good piece of software to have but let's assume thats not an option right now.

Thanks

[Javed Padinhakara]

Good starting point would be the syslog on the switch.

You can execute the command
"log -r" on the console and see the various log messages to get some idea what was happening during the outage.

There is a web-based view too for your 5206 ( simply do a http:/// ) where you can navigate to the Overview>Status page to see the list of Alerts from the switch. This also may give you some idea on the gory happenings.

ps: you mentioned only one switch- a 5406- in your environment. In that case having net management software like PCM+ may not be helpful I guess - you can rather access the switch directly via console/telnet/web-ui and carry out administration.

[Alan_152]

The 1st two places I would look are the following:

1) bring up the task manager on each of your servers/workstations and look at network utilization.

2) log into your switch and look at per port network utilization and error rates.

My guess is that you had a NIC start jabbering or you have a piece of malware somewhere in your network.

[Kevin Richter_1 · Accepted solution]

Start from the most basic. Use the LEDs to see what kind of activity you are (or are not) having on your network. When a network "locks up", there is either no traffic passing (some version of dead) or an unmanageable overload of traffic (paralyzed by a loop, broadcast storm or uncontrolled multicast flood.) Use the Activity LEDs to get your first gauge of what's going on.

Once you access the CLI of the switch, check to see if the switch is "OK" or if it is locked up or overloaded. A "show system" command will show you the CPU utilization...which will usually be at 99% if you have a loop or other condition in your network overwhelming the switch and all other devices.

I like the first reply's suggestion to check the syslog or "Event Log." Using a "show log" command at the CLI works. I'd recommend issuing a "menu" command and browsing the Event Log there instead. It's easier to scroll through entries and see if and when errors (excessive broadcast; high collision or drop rate; CRC errors) might have started occurring.

In the Event Log or at the CLI, you can watch for or check "show boot" to see any crash history for the switch. Not likely to be any since you had to reload the switch (if it crashed, it crashed to recover from a locked up condition.)

Since the ProCurve 5406 switch is managed, starting your troubleshooting from there is a good way to take advantage of the features you've purchased to help manage your network. If you don't see any "clues" from the LED's, logs, counters, etc., you may need to consider what other items in your network might effect the entire network (router/firewall, DNS server, etc.) Sometimes when "everything stops," it turns out all we've really checked is access to the Internet or some other shared/routed resource and it turns out that basic layer 2 traffic is still working. Confirm that "everything" has really stopped by checking to see if pings within the vlan still work even when you can't get out to other resources.

[Kevin Richter_1]

If you need to reboot to recover, consider issuing a "reload" command from the CLI or hitting the "Reset" button. Powering off by pulling the plug to restart will lose your log entries. If you have used or use the reset button in the future, you can still collect data from your "event" after getting the network back online.