topic Re: server suddenly reboot without problem signal in Operating System - HP-UX

server suddenly reboot without problem signal

Sam_88 — Thu, 04 Sep 2003 15:47:55 GMT

I got a problem on my server (HP A500) since it reboots suddenly and I am new for HP-UX. So, please help. I am attach the syslog.log for reference.

Re: server suddenly reboot without problem signal

Pete Randall — Thu, 04 Sep 2003 15:50:53 GMT

Sam,

You need to look at the OLDsyslog.log, the /etc/shutdownlog, and crash dumps in /var/adm/crash, and any diagnostic logs (assuming you're running diagnostics).

Pete

Re: server suddenly reboot without problem signal

Jean-Luc Oudart — Thu, 04 Sep 2003 15:51:17 GMT

Did you check the other logs to see if that's a "proper" reboot or unwanted one.

check this link :
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x335fcc0a76c1ae4da5364278123d54c8,00.html

Rgds,
Jean-Luc

Re: server suddenly reboot without problem signal

Steven E. Protter — Thu, 04 Sep 2003 15:52:13 GMT

Check the NIC card, cable and switch, it appears that your LAN connection was interupted. This could result in the issues you've noted in this thread.

From the outisde on a Unix/Linux box. ping hostname

If the ping times gradually climb, you need a NIC card replacement.

dmesg

Check for hardware faults.

ioscan, look for NW_HW or unclaimed devices.

Check /var/adm/crash for crash dumps.

If there is one, follow the attached guide to analyze the dump. You may need a patch.

SEP

Re: server suddenly reboot without problem signal

Helen French — Thu, 04 Sep 2003 15:56:06 GMT

What version of MC/SG (MC ServiceGuard cluster software) are you using? Applying these patches might solve your issue:

For HP-UX 11.X, MC/SG version A.11.09 - PHSS_27158:

http://www2.itrc.hp.com/service/cki/patchDocDisplay.do?patchId=PHSS_27158

For HP-UX 11.X, MC/SG version A.11.14 - PHSS_29122:

http://www2.itrc.hp.com/service/cki/patchDocDisplay.do?patchId=PHSS_29122

Re: server suddenly reboot without problem signal

Pete Randall — Thu, 04 Sep 2003 16:00:23 GMT

Sam,

Since the last entry I see in your syslog is someone su'ing to root, I have to wonder if this is somehow connected. Do you know who did the su? Do you know what they did after they su'd? What does /etc/shutdownlog say?

Pete

Re: server suddenly reboot without problem signal

E. Wong — Thu, 04 Sep 2003 20:30:48 GMT

Please check for possibles cron jobs that may trigger a reboot.

run

# crontab -l

and check for each of the jobs scheduled.

Re: server suddenly reboot without problem signal

Sam_88 — Fri, 05 Sep 2003 11:08:33 GMT

Thnaks for you all info.

Here is the OLDsyslog.log file. Please help me to see whether the system crash is due to hardware problem or other.

Re: server suddenly reboot without problem signal

Pete Randall — Fri, 05 Sep 2003 11:12:14 GMT

Sam,

I don't think we're seeing all of the OLDsyslog.

Could you just do a "tail -100 /var/adm/syslog/OLDsyslog.log" and attach that output?

Pete

Re: server suddenly reboot without problem signal

Sam_88 — Fri, 05 Sep 2003 11:21:46 GMT

Thanks for your reply

Here is the modified OLDsys.log

Thanks

Re: server suddenly reboot without problem signal

Pete Randall — Fri, 05 Sep 2003 11:28:29 GMT

Sam,

I still don't see anything. Did you see anything suspicious in the shutdown log? Anything under /var/adm/crash? With what we've been able to see so far, there's no way to tell what caused the reboot.

Pete

Re: server suddenly reboot without problem signal

Kent Ostby — Fri, 05 Sep 2003 11:51:30 GMT

Sam -- The key file is going to be /etc/shutdownlog.

Please post the output from:

tail /etc/shutdownlog

If shows something like :

reboot after panic: trap type 1 HPMC
or
reboot after panic: isr.ior = (some number)

then you're probably looking at a HW problem and we need to look at /var/tombstones/ts99 to see if there is current data INSIDE the file.

Please post shutdownlog information as that will tell us if you have a likely HW or SW problem.

Re: server suddenly reboot without problem signal

Sam_88 — Mon, 08 Sep 2003 11:41:35 GMT

Hi all,

Here is the /etc/shutdownlog form the problem A500 server.

06:18 Tue May 15, 2001. Halt: (by shkodb02!root)
07:15 Tue May 15, 2001. Reboot: (by shkodb02!root)
02:33 Thu May 17, 2001. Halt: (by shkodb02!root)
23:00 Thu May 17, 2001. Reboot: (by shkodb02!root)
02:09 Sat May 19, 2001. Reboot: (by shkodb02!root)
12:05 Wed May 30, 2001. Reboot: (by shkodb02!root)
16:24 Wed May 30, 2001. Reboot: (by shkodb02!root)
17:09 Wed May 30, 2001. Reboot: (by shkodb02!root)
18:14 Wed May 30, 2001. Reboot: (by shkodb02!root)
16:17 Fri Jun 1, 2001. Reboot: (by shkodb02!root)
10:35 Tue Jun 5, 2001. Reboot:
10:54 Thu Jun 14, 2001. Halt: (by shkodb02!root)
10:54 Fri Jun 15, 2001. Reboot: (by shkodb02!root)
17:24 Tue Jul 24, 2001. Halt:
15:52 Sun May 11, 2003. Halt: (by shkodb02!root)
10:26 Tue May 13, 2003. Halt: (by shkodb02!root)
db800
15:27 Wed May 14, 2003. Reboot: (by shkodb02!root)ior = 0'505a400.80000000'6f
01:37 Fri May 16 2003. Reboot after panic: , isr.ior = 0'd503800.80000000'74
67800
10:30 Fri May 16, 2003. Reboot: (by shkodb02!root)
11:16 Fri May 16, 2003. Reboot: (by shkodb02!root)
11:29 Fri May 16, 2003. Reboot: (by shkodb02!root)
15:29 Sun May 18, 2003. Halt: (by shkodb02!admin1)
11:49 Thu Aug 21, 2003. Reboot:
11:59 Thu Aug 21, 2003. Halt: (by shkodb02!root)
12:11 Thu Aug 21, 2003. Reboot:
13:50 Thu Aug 21, 2003. Reboot: (by shkodb02!root)
02:20 Fri Aug 22 2003. Reboot after panic: , isr.ior = 0'fb22400.80000000'60
b5800
10:41 Fri Aug 22 2003. Reboot after panic: , isr.ior = 0'a220400.80000000'88
d9800
10:48 Fri Aug 22, 2003. Reboot: (by shkodb02!root)
23:55 Sun Aug 24, 2003. Halt: (by shkodb02!root)
02:46 Mon Aug 25, 2003. Halt: (by shkodb02!admin5)
03:13 Mon Aug 25, 2003. Reboot:
12:54 Thu Aug 28 2003. Reboot after panic: Spinlock deadlock!
10:34 Sun Aug 31 2003. Reboot after panic: , isr.ior = 0'63d1c00.80000000'50

Also, the ts99.dat file is attached

Thanks very much

Sam

Re: server suddenly reboot without problem signal

Pete Randall — Mon, 08 Sep 2003 11:45:35 GMT

Sam,

I get a 404 error when I try to open the ts99 file, but obviously your system paniced and re-booted. An analysis of the ts99 file might help to narrow down the cause.

Pete

Re: server suddenly reboot without problem signal

Sam_88 — Mon, 08 Sep 2003 12:05:31 GMT

Hi Pete,

I attached the file again. Please help me to see whether anything goes wrong.

Thanks

Sam

Re: server suddenly reboot without problem signal

Pete Randall — Mon, 08 Sep 2003 12:18:29 GMT

Sam,

All I can tell with my limited knowledge is that it looks like you had a HPMC (High Priority Machine Check). That will definitely result in a panic and reboot, but why you had the HPMC still needs to be determined. Hopefully someone will be able to diagnose something from your ts99 file.

Pete

Re: server suddenly reboot without problem signal

Kent Ostby — Mon, 08 Sep 2003 12:37:01 GMT

Sam --

Definitely looking at a HW problem and probably a CPU.

You're probably should open a HW call so you can get the correct diagnosis and get the situation fixed right away.

Best regards,

Kent M. Ostby

Re: server suddenly reboot without problem signal

Sam_88 — Mon, 08 Sep 2003 12:52:02 GMT

Hi Pete,

Thanks for your advise on my problem.
Base on the log files, Is that problem on the processor1 for my case? I am getting crazy on this situtation as it reboots without any signals and it is so confuse. Anyway, thanks for your quick response.

Sam

Re: server suddenly reboot without problem signal

Kent Ostby — Mon, 08 Sep 2003 12:55:32 GMT

Again, if you have access to HW support, you're probably better off going to them to get a definitive answer.

If I had to guess at something, I'd look at CPU 0 since it did not report in at the time of the reboot (i.e. the No HPMC Chassis Codes logged) .

Normally, all CPUs should report in to the ts99 file if there is some sort of crash.

Best regards,

Kent M. Ostby