HP9000 system hangs

[Nodoz]

I've got a strange new problem. This system is not in production, in fact it has no users but me and root.

After 1-2 days I can no longer telnet to it, but I can still ping, the console also locks up and a reboot is required it get it operational again...

I'm not able to find anything in the sys log that indicates a problem, I'm wondering if some cron job or something else might be getting out of controller using up resources and finally hanging the system.

I'm pretty much a HP newbie, so any help would be greatly appreciated. Even the dumbest & easiest ideas are welcome.

I would like to snoop around a little trying to find out what's wrong before I reinstall the OS.

Thx

[T G Manikandan]

Any system problem the first thing to do is to check the
/var/adm/syslog/syslog.log file

check whether you have any "SCSI errors" or disk "POWERFAILED" messages

REvert with the messages

[Eugeny Brychkov]

I this system has GSP then log into it and check status:
SS
PS
SL (error logs, no filtering)
Please make sure that LAN console is NOT connected to public network, but private local management lan. High broadcast traffic on the lan console port can hang system until traffic will come down.
In worst case you can issue TC command from GSP initating Transfer of Control and then send crashdump to HP for analisys
Eugeny

[Leo Simon]

There are several items must be check to make sure this is a hardware problem or software problem.

The best way is to check this problem by creating a crashdump file and analysed it by the help of HP engineer.

To create a crashdump file
First you must check your memory size and let the /var/adm/crash directory space is available for grater than your memory size.

Second check the crashdump program is configure
right which mean crashdump program is up and running.
The file is in /etc/rc.config.d directory
For HP-UX 10.20 file name is savecore
For HP-UX 11.0 file name is savecrash

Third if your server hang you can just press TOC button is your server to create a crash dump.

If you suspect cron job might be the reason for this you can check using crontab -l for
checking cron that automatically running
Or check /var/spool/cron directory for cron job program that will run automatically

I hope this will answer your problem

[Thamm]

hi,
what about your network-interface configuration?
If there is a mismatch between switchport and interface configuration you might get this problems after a while.
You can check your configuration using the following comands:
netstat -in # find out your interaces
lanadmin -x lanI # with I inteerface number

The result might be:
2128 -> lanadmin -x 0
Current Speed = 100 Full-Duplex Auto-Negotiation-OFF

If the swith uses Half Dublex you will run into problems.

Regards,
Stefan.

[Nodoz]

Ok a couple new bits of info, it's happeneing in 5-15 mins and not 24 hours like I first thought and the attached mouse and keyboard are also locked up. Hitting the power button is the only way to get it working that I'm aware of...

Running iostat 1 99 over the network gives me this, At the bottom it simply hangs no more output. My other telnet window I had going also hung, about the same time I did a df.

I'll try and reproduce this and let you know.

device bps sps msps
c0t5d0 0 0.0 1.0
c0t5d0 21 6.9 1.0
c0t5d0 8 1.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 1 1.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 8 1.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 0 0.0 1.0
c0t5d0 8 1.0 1.0

[Keely Jackson]

Just a thought, do you have the latest patches isntalled?

Cheers
Keely

[Eugeny Brychkov]

Please during hang period make sure that lan console port do not experience broadcast storm - simply disconnect it from the network to see if machine will recover
Eugeny

[Nodoz]

This machine is deep behind a firewall, and it has nothing on it much other than the base HP install, probably doesn't have any of the patches, but this is something I'm going to correct.

I noticed it seems to hang up after getting a error starting a raid manger software, which isn't surprising since there is no raid attached any longer. I removed the software and so far no lockup.

Is there a limit to roots email file ? I just deleted about 6+ megs of mail for root also...

I will work on each of these suggestions tommorow, and thx for all the input.

Thx

[Steven E. Protter]

I'm attaching a web doc telling you how to analyze q4 dump output if needed for referral to HP Support.

P

[Nick Wickens]

I guess you have checked that none of your main filesystems (/stand / /var etc are full or nearly full - use the bdf command - ideally you should only be 75% used.

[Nodoz]

Just a FYI, It would seem that it is now fixed

# uptime
12:07pm up 17:31, 3 users, load average: 0.00, 0.01, 0.02

I've learned a lot about HP in the last 24 hours, thx for everyones input...

It would seem that the raid manager was causing the hang because it could not find an array to monitor.

At least that's what I think today :)

[Nodoz]

Just a few feedbacks

T G - I think you nailed it. After I emliated everything else, it turns out the raidmanager error in the sys log might have been the culpret.

Eugeny - I'm not sure what a GSP is. Not sure what the SS, SL and TC commands do since they dont' seem to work on my console.

Leo - Good info I'll try that next time.

Thamm - lanadmin doesn't work on my system...
Error: This option is not supported for this hardware.
Bummber :( Good info though.

Keely - Patches are good, I just inherited this system so I'm not at fault :)

Steven - Didn't see your attachment....

Nick - File systems were good, all below 20%. But good advice.

[Nodoz]

Just a few feedbacks

T G - I think you nailed it. After I eliminated everything else, it turns out the raid manager error in the sys log might have been the culprit.

Eugeny - I'm not sure what a GSP is. Not sure what the SS, SL and TC commands do since they don't' seem to work on my console.

Leo - Good info I'll try that next time.

Thamm - lanadmin doesn't work on my system...
Error: This option is not supported for this hardware.
Bummber :( Good info though.

Keely - Patches are good, I just inherited this system so I'm not at fault :)

Steven - Didn't see your attachment....

Nick - File systems were good, all below 20%. But good advice.

[V. V. Ravi Kumar_1]

Hi,
may be a patch issue. search for network patches on that version of OS.