topic Re: service guard TOC : not updated in /etc/shutdownlog in Operating System - HP-UX

service guard TOC : not updated in /etc/shutdownlog

joseph51 — Mon, 02 Aug 2010 13:07:29 GMT

Hi,

In our environment one server rebooted because of unavailability of Heart beat . I believe the service guard issued a TOC . But its not upated in /etc/shutdownlog .and there is no crash dump generated . the time stamp for the /var/adm/crash/crash.0 is today the same time when the server rebooted . But the content of this directory is very old data .

Please tell me . why the shutdownlog not updated and there is no crashbump generated

Re: service guard TOC : not updated in /etc/shutdownlog

Steven E. Protter — Mon, 02 Aug 2010 17:19:13 GMT

Shalom,

TOC acts like a power off. It is sudden and not necessarily logged.

Since it is not a crash, a crash dump need not be created.

TOC is what the cluster does to one node or the other when heartbeat or quorum server is lost and the cluster wants to avoid data corruption via split brain syndrome.

Split brain is when two nodes try and access the same shared data at the same time.

There should be some evidence on the logs, but TOC is very sudden. The surviving node that is not hammered should log the event.

SEP

Re: service guard TOC : not updated in /etc/shutdownlog

Dennis Handly — Mon, 02 Aug 2010 18:51:04 GMT

>it's not updated in /etc/shutdownlog and there is no crash dump generated.

As SEP said, the log is only updated with shutdown(1m). Do you have crash dumps configured?

>But the content of this directory is very old data.

You probably should remove it if it is that old.

Re: service guard TOC : not updated in /etc/shutdownlog

melvyn burnard — Tue, 03 Aug 2010 06:50:56 GMT

well if Serviceguard cauused a TOC, it WOULD be logged in the shutdownlog and a dump attempted, however, if you are using Oracle and THAT loses "heartbeat", I have seeen Oracle call a reboot -q, which does NOT log a dump or to shutdownlog!
Are you perjaps using Oracle?

Re: service guard TOC : not updated in /etc/shutdownlog

joseph51 — Tue, 03 Aug 2010 08:05:18 GMT

Hi Melvyn,

There is oracle process runnign on this server

root@dczv2:/sysadm/utils : ps -ef | grep -i pmon
oracle 660 1 0 Aug 2 ? 0:07 ora_pmon_DCZV
root@dczv2:/sysadm/utils :

And the shutdown log updated on July 23 . not for this time crash .
This crash happened on AUg 2nd .

root@dczv2:/sysadm/utils : ls -l /etc/shutdownlog*
-rw-r--r-- 1 bin bin 1707 Jul 23 11:19 /etc/shutdownlog
root@dczv2:/sysadm/utils :

Re: service guard TOC : not updated in /etc/shutdownlog

melvyn burnard — Tue, 03 Aug 2010 09:03:18 GMT

I would check the OLDsyslog.log from the time of the crash, and check to see ifthere were any Oracle issues logged, as this may have been the cause.
reboot -q does a reboot "quietly", and doesw NOT log to shutdown.log, it would also have done a reboot and not a crash in all probability

Re: service guard TOC : not updated in /etc/shutdownlog

Stephen Doud — Tue, 03 Aug 2010 10:58:42 GMT

A Serviceguard-generated TOC usually produces a message of this form in the /etc/shutdownlog:

Reboot after panic: SafetyTimer expired, INIT, IIP:0xe000000002011670 IFA:0xe0000001371705b0

If you are not seeing this type of message, then investigate (crashconf(1m)) whether the server is able to perform a memory dump (TOC) properly.
Such panic messages are pulled from the dump space when the system boots. If the dump space (typically swap space) does not have a legitimate dump, then no messages are placed in /etc/shutdownlog.

Re: service guard TOC : not updated in /etc/shutdownlog

joseph51 — Tue, 03 Aug 2010 11:30:21 GMT

Tnak you for your valuable information ...

Re: service guard TOC : not updated in /etc/shutdownlog

joseph51 — Tue, 03 Aug 2010 11:32:08 GMT

Thanks to all for giving a valuable answers