topic Re: server rebooted automatically in Operating System - HP-UX

server rebooted automatically

Bejoy C Alias — Fri, 06 May 2005 05:02:04 GMT

Hi gurus,

for the first time ever after installing the server, it restarted yesterday automatically.
There is nothing in the shutdown log to show anybody rebooted the server . There is no error logs in the syslog or gsp also .
any way to trace the reason ?

rgds

Re: server rebooted automatically

Darrel Louis — Fri, 06 May 2005 05:04:23 GMT

Hi,

Can you see anything in the /var/adm/crash?

Have you checked your cron or at log file, also do a crontab -l.

Darrel

Re: server rebooted automatically

Pete Randall — Fri, 06 May 2005 05:05:53 GMT

Check your /etc/shutdownlog too.

Pete

Re: server rebooted automatically

Devender Khatana — Fri, 06 May 2005 05:07:40 GMT

Hi,

How many power supplies server is running on ? Is it the only server on the power source it is connected to? It could be because of power source problem? Can be a hardware issue also.

Check /var/adm/crash files with latest time stamp.

HTH,
Devender

Re: server rebooted automatically

Bejoy C Alias — Fri, 06 May 2005 06:46:24 GMT

Hi,
I already specified that there is no logs in the shutdownlog.
/var/adm/crash is empty.

Re: server rebooted automatically

Victor BERRIDGE — Fri, 06 May 2005 07:18:08 GMT

Hi,
Just thoughts...
How does your /etc/rc.log.old finish? (well look at the sequence from the last 100 lines...).
Is the machine beeing brought down or the file is at the last level since it booted?

All the best
Victor

Re: server rebooted automatically

Kent Ostby — Fri, 06 May 2005 07:29:36 GMT

Bejoy .. two places to check.

#1) Check the tombstones:

grep May /var/tombstones/ts99

If a date from the GREP shows the approximate time your box failed, you had a hardware error and you should contact HP hardware and supply the ts99 file.

#2) If that doesn't work, and this box has service logs or GSP logs at the console, check those for power fail messages or overtemp messages.

Best regards,

KEnt m. Ostby

Re: server rebooted automatically

Steve Lewis — Fri, 06 May 2005 08:04:59 GMT

One of the following happened:

Either your power supply failed completely without warning;

Or somebody turned off the power or unplugged it and will not admit it;

Or (if its an old D class) someone pressed the TOC button onthe back;

Or someone went into the GSP and selected Reset CTRL/B-RS-Y;

Or (least likely) your CPU, memory etc failed completely, but then it probably would not have come back up again.

A PANIC, shutdown or reboot would have been logged, so I think that the power, TOC or GSP reset is the most likely.

Re: server rebooted automatically

Devender Khatana — Fri, 06 May 2005 08:08:45 GMT

Hi,

It is important to mention here that if this is a old D-class maachine it could have Power Supply / DC ON/OFF Switch problem. This was a known problem in old D-Class server, which used to reboot intermidently without any error due to these problems.

Which model is this ?

HTH,
Devender

Re: server rebooted automatically

Bejoy C Alias — Sun, 08 May 2005 22:09:04 GMT

Hi,

The server model is RP5470 .

This server is having two power supplies, so i think there should not be any pblm due to that. Even if somebody switched off the power and rebooted, the server should have done fsck during the next boot, but fsck is not executed during the reboot .

For GSP reset , i think that will be logged.

Again if it is a TOC , that will be logged.

What is best way to check whether during the last boot fsck ran or not . I checked the lost+found directories in the file systems and those are empty .

rgds

Re: server rebooted automatically

Isralyn Manalac_1 — Sun, 08 May 2005 22:44:13 GMT

What were the last couple of entries in OLDsyslog.log? Did you get any tombstone files?

Regards,

Isralyn

Re: server rebooted automatically

Suraj Singh_1 — Sun, 08 May 2005 23:12:44 GMT

It seems there is no harm to the filesystems after the reboot.

How accessible is the server physically? Is there any possibility that the root password is with somebody? What is the time stamp on shutdownlog file? What is the o/p of "who -b" command?

I would suspect Power source only, UPS coudl have tripped, or something like that.

Regards

Re: server rebooted automatically

Sudeesh — Sun, 08 May 2005 23:50:10 GMT

Hi Bejoy,
Try your luck with Console logs in GSP. This strore 20kb of console data and if you dont do much activity on console you can see the boot time messages.

Ctrl+B
MP>CL

Look for something like this:

/sbin/ioinitrc:
/dev/vg00/lvol1:log replay in progress
/dev/vg00/lvol1:replay complete - marking super-block as CLEAN
/sbin/krs_sysinit:
* The module 'ipf' has been loaded.
* The module 'rng' has been loaded.
* The module 'gvid_info' has been loaded.

/sbin/bcheckrc:
Checking for LVM volume groups and Activating (if any exist)
Volume group "/dev/vg00" has been successfully changed.
Activated volume group
Volume group "/dev/vgLab" has been successfully changed.
Resynchronized volume group /dev/vg00
Resynchronized volume group /dev/vgLab
vxfs fsck: sanity check: root file system OK (mounted read/write)
Checking vxfs file systems
vxfs fsck: /dev/vg00/rlvol3:/dev/vg00/rlvol3 is currently mounted
vxfs fsck: /dev/vg00/rlvol1:/dev/vg00/rlvol1 is currently mounted
Running parallel replay fsck ...
/dev/vg00/rlvol8:log replay in progress
/dev/vg00/rlvol5:log replay in progress
/dev/vg00/rlvol6:log replay in progress
/dev/vg00/rlvol7:log replay in progress
/dev/vg00/rlvol4:log replay in progress
/dev/vg00/rlvol5:replay complete - marking super-block as CLEAN
/dev/vg00/rlvol6:replay complete - marking super-block as CLEAN
/dev/vg00/rlvol4:replay complete - marking super-block as CLEAN

Hope this helps.

Sudeesh

Re: server rebooted automatically

Bejoy C Alias — Mon, 09 May 2005 04:22:38 GMT

There is no valuable information in OLDsyslog. The time stamp of the shutdownlog is of one month before , when we have done a manual reboot . GSP console log contains some logs but it is of no use.

Re: server rebooted automatically

Naveej.K.A — Mon, 09 May 2005 04:35:11 GMT

Hi,

What is/was the state of the Front panel when you saw the server??

Are your two power supplies using different power source?? If they are from the same UPS, there are chances that the UPS might have tripped..

Do you have the history enabled in root's profile...If yes check when the last reboot/shutdown command was executed.

Also check if there are any EMS events from /var/opt/resmon/log...

Regards,
Naveej

Re: server rebooted automatically

Unni Krishnan — Mon, 09 May 2005 07:14:22 GMT

HI bejoy

I still feel that there must be some Hardware problem And please run bdf to see any filesystems are getting full especially /tmp filesystem.

Re: server rebooted automatically

Bejoy C Alias — Mon, 09 May 2005 07:40:58 GMT

After rebooting the server , the attention LED was blinking , no fault LED . The last log in GSP was
ALERT LEVEL: 10 = Boot possible, functionality lost .

Some other machines are also using the same ups which not rebooted , so the ups was not triped.

There is no errors in EMSMonitor log.

The last reboot was on Apr 12 which is the last one logged in shutdown log .

There is no filesystems which are used more than 50 % , so i dont think that it is because of a full file system.