server rebooted automatically

[Bejoy C Alias]

Hi gurus,

for the first time ever after installing the server, it restarted yesterday automatically.
There is nothing in the shutdown log to show anybody rebooted the server . There is no error logs in the syslog or gsp also .
any way to trace the reason ?

rgds

[Darrel Louis]

Hi,

Can you see anything in the /var/adm/crash?

Have you checked your cron or at log file, also do a crontab -l.

Darrel

[Pete Randall]

Check your /etc/shutdownlog too.


Pete

[Devender Khatana]

Hi,

How many power supplies server is running on ? Is it the only server on the power source it is connected to? It could be because of power source problem? Can be a hardware issue also.

Check /var/adm/crash files with latest time stamp.

HTH,
Devender

[Bejoy C Alias]

Hi,
I already specified that there is no logs in the shutdownlog.
/var/adm/crash is empty.

[Victor BERRIDGE]

Hi,
Just thoughts...
How does your /etc/rc.log.old finish? (well look at the sequence from the last 100 lines...).
Is the machine beeing brought down or the file is at the last level since it booted?


All the best
Victor

[Kent Ostby]

Bejoy .. two places to check.

#1) Check the tombstones:

grep May /var/tombstones/ts99

If a date from the GREP shows the approximate time your box failed, you had a hardware error and you should contact HP hardware and supply the ts99 file.

#2) If that doesn't work, and this box has service logs or GSP logs at the console, check those for power fail messages or overtemp messages.

Best regards,

KEnt m. Ostby

[Steve Lewis]

One of the following happened:

Either your power supply failed completely without warning;

Or somebody turned off the power or unplugged it and will not admit it;

Or (if its an old D class) someone pressed the TOC button onthe back;

Or someone went into the GSP and selected Reset CTRL/B-RS-Y;

Or (least likely) your CPU, memory etc failed completely, but then it probably would not have come back up again.

A PANIC, shutdown or reboot would have been logged, so I think that the power, TOC or GSP reset is the most likely.

[Devender Khatana]

Hi,

It is important to mention here that if this is a old D-class maachine it could have Power Supply / DC ON/OFF Switch problem. This was a known problem in old D-Class server, which used to reboot intermidently without any error due to these problems.

Which model is this ?

HTH,
Devender

[Bejoy C Alias]

Hi,

The server model is RP5470 .

This server is having two power supplies, so i think there should not be any pblm due to that. Even if somebody switched off the power and rebooted, the server should have done fsck during the next boot, but fsck is not executed during the reboot .

For GSP reset , i think that will be logged.

Again if it is a TOC , that will be logged.

What is best way to check whether during the last boot fsck ran or not . I checked the lost+found directories in the file systems and those are empty .

rgds

[Isralyn Manalac_1]

What were the last couple of entries in OLDsyslog.log? Did you get any tombstone files?

Regards,

Isralyn

[Suraj Singh_1]

It seems there is no harm to the filesystems after the reboot.

How accessible is the server physically? Is there any possibility that the root password is with somebody? What is the time stamp on shutdownlog file? What is the o/p of "who -b" command?

I would suspect Power source only, UPS coudl have tripped, or something like that.

Regards

[Sudeesh]

Hi Bejoy,
Try your luck with Console logs in GSP. This strore 20kb of console data and if you dont do much activity on console you can see the boot time messages.

Ctrl+B
MP>CL



Look for something like this:


/sbin/ioinitrc:
/dev/vg00/lvol1:log replay in progress
/dev/vg00/lvol1:replay complete - marking super-block as CLEAN
/sbin/krs_sysinit:
* The module 'ipf' has been loaded.
* The module 'rng' has been loaded.
* The module 'gvid_info' has been loaded.


/sbin/bcheckrc:
Checking for LVM volume groups and Activating (if any exist)
Volume group "/dev/vg00" has been successfully changed.
Activated volume group
Volume group "/dev/vgLab" has been successfully changed.
Resynchronized volume group /dev/vg00
Resynchronized volume group /dev/vgLab
vxfs fsck: sanity check: root file system OK (mounted read/write)
Checking vxfs file systems
vxfs fsck: /dev/vg00/rlvol3:/dev/vg00/rlvol3 is currently mounted
vxfs fsck: /dev/vg00/rlvol1:/dev/vg00/rlvol1 is currently mounted
Running parallel replay fsck ...
/dev/vg00/rlvol8:log replay in progress
/dev/vg00/rlvol5:log replay in progress
/dev/vg00/rlvol6:log replay in progress
/dev/vg00/rlvol7:log replay in progress
/dev/vg00/rlvol4:log replay in progress
/dev/vg00/rlvol5:replay complete - marking super-block as CLEAN
/dev/vg00/rlvol6:replay complete - marking super-block as CLEAN
/dev/vg00/rlvol4:replay complete - marking super-block as CLEAN





Hope this helps.

Sudeesh

[Bejoy C Alias]

There is no valuable information in OLDsyslog. The time stamp of the shutdownlog is of one month before , when we have done a manual reboot . GSP console log contains some logs but it is of no use.

[Naveej.K.A]

Hi,

What is/was the state of the Front panel when you saw the server??

Are your two power supplies using different power source?? If they are from the same UPS, there are chances that the UPS might have tripped..

Do you have the history enabled in root's profile...If yes check when the last reboot/shutdown command was executed.

Also check if there are any EMS events from /var/opt/resmon/log...

Regards,
Naveej

[Unni Krishnan]

HI bejoy

I still feel that there must be some Hardware problem And please run bdf to see any filesystems are getting full especially /tmp filesystem.

[Bejoy C Alias]

After rebooting the server , the attention LED was blinking , no fault LED . The last log in GSP was
ALERT LEVEL: 10 = Boot possible, functionality lost .

Some other machines are also using the same ups which not rebooted , so the ups was not triped.

There is no errors in EMSMonitor log.

The last reboot was on Apr 12 which is the last one logged in shutdown log .

There is no filesystems which are used more than 50 % , so i dont think that it is because of a full file system.