System went down unexpectedly

[Jagadesh_2]

Dear Admins,

We had a server down yesterday. Iam trying to find the root cause as why the server went down. I analysed all the dmesg, syslog.log,OLDsyslog.log,rc.log,rc.log.old still i am not able to come to a conclusion as why the server went down unexpectdly. I am here by attaching the logs. Requesting you to let me know why the server went down. Hope your suggessions will help me to prevent the system to go down in future.

Please find the dmesg output below
dmesg output:
****************
btlan1: Initializing 10/100BASE-T card at 10/4/8.......
Networking memory for fragment reassembly is restricted to 100839424 bytes
Logical volume 64, 0x1 configured as ROOT
Logical volume 64, 0x2 configured as SWAP
WARNING: Logical volume for Dump expected but not found
Swap device table: (start & size given in 512-byte blocks)
entry 0 - major is 64, minor is 0x2; start = 0, size = 98304
Defaulting dump to primary swap
WARNING: Could not find HPA for dump device 0xffffffff
WARNING: Could not use primary swap for dump. Dump is disabled.
Starting the STREAMS daemons.
9245XB HP-UX (B.10.20) #1: Sun Jun 9 06:31:19 PDT 1996

Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 1048576 Kbytes, lockable: 924772 Kbytes, available: 959348 Kbytes

Using 1020 buffers containing 7968 Kbytes of memory.


more /etc/shutdownlog
.
.
07:20 Thu Jan 26, 2006. Halt:

I checked the ioscan and there is no hardware issues found. When tried to bring up the server we found that there ia inconsistency in the filesystem. Then we fixed it and brought up the server.

Thanks in advance for your help

[Jagadesh_2]

Attaching the rc.log, rc.log.old & syslog.log

[Pete Randall]

Is there anything in /var/adm/crash?


Pete

[baiju_3]

Hi,

Search for core file generated .If found any ,can get HP support to analyse what went wrong.Check the chasis logs for any h/w problem reported .

Also see log files in /var/tombstones and OLDsyslog.log.




Regards bl.

[Patrick Wallek]

If none of the files show anything, especially if /etc/shutdownlog does not show any reason for a crash, then chances are the reason has to do with power.

If power is cut suddenly, a power supply flakes out, a UPS flakes out, whatever, then there will typically NOT be any messages in any of the logs files.

A sudden loss of power does not provide any opportunity for anything to be written to log files.

[Don Morris_1]

rc.log.old indicates the system was shut down, not a power failure:

SVRMGR>
Server Manager complete.
completed shutting down ARCH2 Thu Jan 26 12:19:45 GMT 2006
Script Completed

Stop NFS server subsystem
Output from "/sbin/rc2.d/K900nfs.server stop":



I'd definately look at /etc/shutdownlog to see if the shutdown was due to a panic or a shutdown request. I wouldn't expect a dump in /var/adm/crash because your syslogs indicate dump is disabled.

[Jagadesh_2]

Thanks for the sugessions,

The system went down at 07:20

$more /etc/shutdownlog
07:20 Thu Jan 26, 2006. Halt:

I am not finding any information for 07:20. Quiet strange.

This server is in a datacenter. If it was power problem then it could affect other servers too. Other servers were up and running. Is there any problem with the SMPS of this server? If so where it will log that information? This is a HP9000 K420 class server. This has only one SMPS. My doubt is with the SMPS. If you could let me know where the SMPS will log the failure information so that i can check and confirm the same.

[Jagadesh_2]

How to check for the SMPS logs. If there is an power failure due to SMPS then where it will log those errors.

[Darrel Louis]

Hi,

Have you checked your log-files on the gsp.
Also check your ts99 file for information.

Darrel