Server Hung Urgent!!

[George_Dodds]

Hi

I've got an L2000 that is hung.

Cant telnet or access through console!

I can ping the box but thats it.

Any ideas or is it a hard reboot job.

Ta

George

[Stefan Farrelly]

Have you got another server that can remsh to your hung server as root without a password ?

if so you can try; remsh -l root /usr/sbin/reboot

Can you ftp to it ? (just to see if its working).

Is your server configured to allow remote reboot requests ?

[Leif Halvarsson_2]

Hi,
Is console completly dead or is login just very slow. If there is problem with NIS (I don't know if you use NIS) login sometimes becomes very slow.

[Massimo Bianchi]

Are you sure it isn't over loaded ?

In case you decide for a ToC, remember to take the dump, so you can open a call to HP and have the dump analyzed.


What run on that server ? Is there a NFS share that might be hanging the system ?

If remsh work, can you try to stop some applications?

HTH,
Massimo

[Stefan Farrelly]

another thought....

ask around - perhaps someone else has left a root windows logged into it so you can issue a reboot from it, or check whats going on.

[T G Manikandan]

Might be problems with the hard disks.

make sure you check the /var/adm/syslog/OLDsyslog.log file after the reset

Either you can do a reset from GSP
or just do a hard boot.

[twang]

another method:
connect GSP, issue RS to reset the system instead of hard reset the machine.

[twang]

one more thing, check your system after reboot:
1. /var/adm/crash
2. /var/tombstones

[George_Dodds]

I cant ftp to the server but there is an nfs mount to another server that is still active.

When i log into the console there is no prompt all existing telnet session are hanging.

Tried using console cable on another server and it was ok.

I havent had the ok the try the remote reboot but remsh should be enabled.

[Massimo Bianchi]

Hi,
when you get remsh,
try to shutdown the nfs services

/sbin/init.d/nfs.client stop
/sbin/init.d/nfs.server stop
/sbin/init.d/nfs.core stop

This commands may hang, but if they succedd i think they will free the server.

or stop the net

/sbin/init.d/net stop

After you can have only access by console, but might work.


Massimo

[George_Dodds]

Damn tried remsh and no joy, confused why nfs mount are still active on the other server though.

Allough both machines are connect to a xp256 and the nfs filesystem are on there.

[Massimo Bianchi]

wild shot: extract the fiber channel from the server for 15 seconds and then plug it backup.

it worked in a couple of situation.

Massimo

[George_Dodds]

I can see one of the local disks has a lot of io activity.

tried to see error in gsp log but cant get past first entry which was for last week.

[Jeff Schussele]

Hi George,

Sure sounds like you have a runaway process that's consuming EVERY single CPU cycle. IF you can find an existing open root window and IF you can get any response from it to determine & kill the offending PID(s) then that's about all you can do.
If that's not an option then I'm afraid your only recourse is to get into GSP & issue the TC (TOC) command & analyze the dump.

Good Luck/Good Hunting,
Jeff

[George_Dodds]

It Lives!!!!

here's a new one.

Hard booted the system in the end.

primary boot disk wasnt found in isl search path.

booted to alt path ran ioscan and no sign of primary disk.

Since it looked like a dead disk i tried a spare disk so i could restore to it, which was found ok.

To humour myself i tried the dead disk in another server and it booted fine!

So i've put the resurrected disk back in the prod server and it booted without any problems.

The only thing i can think of is that the disk slightly unseated itself, not enough to lose power but enough to lose io.

I'm running a check script just to make sure the disk is ok.

Thanks for all the relpies. :)