topic Re: rp3410 problem in Operating System - HP-UX

rp3410 problem

Alain St-Laurent — Tue, 11 Apr 2006 10:15:46 GMT

Hello everyone,

On our new rp3410 on 3 different occasions (over 2 months) the system seemed to be locked, and we had to reboot to gain back access.

When we looked at some of the system logs,
we didn't find anything unusual.
We have reason to believe that the system was not hanged and that it was still running.
But we could not login or telnet to it, although we could ping it.

The syslog indicates some inetd activities and there is a device listed as unknown in the ioscan...
0/7/2/0 unknown PCI Display (10025159)

We are running version B.11.11 (U 9000/800) and I think the most recent patches have been installed.

Those events stopped us from going into production with that server and now it's been a month and the system is stable.

Any clue as what happened, and will it happen
again ???

Re: rp3410 problem

Antonio Cardoso_1 — Tue, 11 Apr 2006 11:22:52 GMT

Hi

When system was in such 'hanged' state, did you attempt to login using the WEB console ?

Re: rp3410 problem

A. Clay Stephenson — Tue, 11 Apr 2006 11:33:18 GMT

A box that cannot be telneted but can be pinged is a classic symptom of a dead inetd. The first thing I would do is search the patch database for inetd-related (e.g. PHNE_28328 the inetd cumulative patch) and apply them. Because I don't like to get up in the middle of the night and login at the console, one of my standard cronjobs on every UNIX server is a script that runs every 5 minutes and makes sure than inetd is running and if not restarts it. It's just one of those things you learn to do after decades of UNIX experience. The cron Band-aid is no substitute for fixing the problem but it's still a useful trick.

Re: rp3410 problem

Patrick Wallek — Tue, 11 Apr 2006 11:33:51 GMT

It almost sounds as if the 'inetd' daemon died for some reason.

'ping' works at a much lower level that telnet and does not need the inetd daemon.

Since this is a new machine you should have support. Did you call HP for help with this issue?

I also seriously doubt the the 'PCI DISPLAY' line has anything to do with this. That indicates to me that you may have a video card installed, but have not installed the driver for it yet.

Without a lot more information and troubleshooting, it is very hard to know what happened and whether or not it will happen again.

Re: rp3410 problem

Torsten. — Tue, 11 Apr 2006 11:47:34 GMT

regarding the ioscan entry:

The build-in VGA is not supported - the line in ioscan is normal and not a problem.
(It's only used in the Integrity version)

See

http://docs.hp.com/en/A7136-96003/A7136-96003.pdf

Re: rp3410 problem

John Dvorchak — Tue, 11 Apr 2006 12:34:09 GMT

Did you check the console when it was "Hung". If the console is non-responsive I'd have to vote for a HUNG system. Very rare. If the console was responsive then I'd vote for inetd dying.

Ping, since it is a ICMP echo, occurs several layers below the OS. Even if the OS is hung will probably still return pings if the machine booted and configured the NIC with an IP stack successfully. Usually the only thing that ping really tells us is that the LAN is working between you and the target machine, and the target is powered up.

Re: rp3410 problem

Alain St-Laurent — Tue, 11 Apr 2006 12:57:56 GMT

Thanks everyone, for the clue on the PCI display.

His there any way to prove that inetd failed, is there any sure way to make it fail again ?

I tried some rlogin rmsh rcp in a script
that looped and it has been running for days
and did not give any errors

We want to replicate the problem and then
we will be sure that we fixed it.

Re: rp3410 problem

Antonio Cardoso_1 — Tue, 11 Apr 2006 14:16:11 GMT

Hi Alain,

I come again with question regarding the WEB console or console access.
WEB console is a management LAN interface independant from production LAN interfaces, providing a remote access to system console even if inetd service is down or your system is in low run level.
Regarding troubleshooting of your issue, this might help you performing "hot" debug.

You search the system for a core file potentially generated when system blocked. (if some appears, use 'file' command to determine which application caused the core)

Re: rp3410 problem

Alain St-Laurent — Tue, 11 Apr 2006 14:31:25 GMT

Hi Antonio,

The first time this occured somebody was
able to access the web console and initiate
a reboot.

The second time this occured we didn't
have to reboot. The only user on the system tried again half an hour later an was able to access the server.

By the way we confirmed that the patch
are installed.

drwxr-xr-x 5 root sys 96 Nov 23 23:43 PHNE_28328

Re: rp3410 problem

Antonio Cardoso_1 — Wed, 12 Apr 2006 01:14:46 GMT

So, difficult with this information to guess what happened.
Case this occurs again, you should attempt to check a few things on the server prior to reboot it:
- log on using the console
- run top / ps -ef commands to check what processes are there and using CPU.
- check inetd is running
- check network connectivity (ping to broadcast address of your LAN)
- collect system information using SEP's script
http://www.hpux.ws/system.perf.sh
(SEP's forum profile is: http://forums1.itrc.hp.com/service/forums/publicProfile.do?userId=CA569821&forumId=1)

Re: rp3410 problem

Tom Henning — Wed, 12 Apr 2006 05:59:00 GMT

You stated that when the system was non-responsive, you could not login or telnet to it. Do you have secure shell installed and could you ssh to the box? It runs as a seperate deamon process and does not (near as I can tell) run via inetd. If inetd had died you should have been able to get to the box via ssh.

I have a 3410, it's been runnning for some months now and I've experienced no problems. It is very lightly loaded, though so not sure I'm a good example for this one.

Re: rp3410 problem

Alain St-Laurent — Wed, 12 Apr 2006 14:05:02 GMT

We don't have secure shell installed.

Is there any way to prove that inetd
failed ?

the logs in syslog.log don't mention
a failure.

is there any way to stress inetd.

Re: rp3410 problem

Mridul Shrivastava — Thu, 13 Apr 2006 03:43:56 GMT

If you want to analyse what exactly happened than we have to enable inetd daemon logging and wait for this problem to reoccure then we'll find some entries regarding the issue.
To enable the inetd daemon logging edit the /etc/rc.config.d/netdaemons
and change the following parameter
INETD_ARGS=-l
this will enable inetd logging and if this happens in future so we'll have some logs for this issue.
Meanwhile you pls check the server for all network related patches. Since when patch updation is not done and check for the available patches and install.

Re: rp3410 problem

Alain St-Laurent — Thu, 04 May 2006 15:21:32 GMT

The problem occured again but a couple days late the HP technician found a memory problem.
A memory chip was replaced.

the inetd activity, was normal after all.

Thanks everyone.