service guard TOC : not updated in /etc/shutdownlog

[joseph51]

Hi,

In our environment one server rebooted because of unavailability of Heart beat . I believe the service guard issued a TOC . But its not upated in /etc/shutdownlog .and there is no crash dump generated . the time stamp for the /var/adm/crash/crash.0 is today the same time when the server rebooted . But the content of this directory is very old data .

Please tell me . why the shutdownlog not updated and there is no crashbump generated

[Steven E. Protter]

Shalom,

TOC acts like a power off. It is sudden and not necessarily logged.

Since it is not a crash, a crash dump need not be created.

TOC is what the cluster does to one node or the other when heartbeat or quorum server is lost and the cluster wants to avoid data corruption via split brain syndrome.

Split brain is when two nodes try and access the same shared data at the same time.

There should be some evidence on the logs, but TOC is very sudden. The surviving node that is not hammered should log the event.

SEP

[Dennis Handly]

>it's not updated in /etc/shutdownlog and there is no crash dump generated.

As SEP said, the log is only updated with shutdown(1m). Do you have crash dumps configured?

>But the content of this directory is very old data.

You probably should remove it if it is that old.

[melvyn burnard]

well if Serviceguard cauused a TOC, it WOULD be logged in the shutdownlog and a dump attempted, however, if you are using Oracle and THAT loses "heartbeat", I have seeen Oracle call a reboot -q, which does NOT log a dump or to shutdownlog!
Are you perjaps using Oracle?

[joseph51]

Hi Melvyn,

There is oracle process runnign on this server

root@dczv2:/sysadm/utils : ps -ef | grep -i pmon
oracle 660 1 0 Aug 2 ? 0:07 ora_pmon_DCZV
root@dczv2:/sysadm/utils :

And the shutdown log updated on July 23 . not for this time crash .
This crash happened on AUg 2nd .

root@dczv2:/sysadm/utils : ls -l /etc/shutdownlog*
-rw-r--r-- 1 bin bin 1707 Jul 23 11:19 /etc/shutdownlog
root@dczv2:/sysadm/utils :

[melvyn burnard]

I would check the OLDsyslog.log from the time of the crash, and check to see ifthere were any Oracle issues logged, as this may have been the cause.
reboot -q does a reboot "quietly", and doesw NOT log to shutdown.log, it would also have done a reboot and not a crash in all probability

[Stephen Doud · Accepted solution]

A Serviceguard-generated TOC usually produces a message of this form in the /etc/shutdownlog:

Reboot after panic: SafetyTimer expired, INIT, IIP:0xe000000002011670 IFA:0xe0000001371705b0

If you are not seeing this type of message, then investigate (crashconf(1m)) whether the server is able to perform a memory dump (TOC) properly.
Such panic messages are pulled from the dump space when the system boots. If the dump space (typically swap space) does not have a legitimate dump, then no messages are placed in /etc/shutdownlog.

[joseph51]

Tnak you for your valuable information ...

[joseph51]

Thanks to all for giving a valuable answers