Questions about dumping of the error log buffers during crash

[quorum]

IA64 V8.4-2

Whenever system crashed we received following message :

**** No supported device(s) found in DUMP_DEV
**** Unable to dump error log buffers to remaining shadow set members
** Dumping memory to HBVS unit 11401

We configured the local disk DKA100 as DOSD dump device (created directory & dumpfile , added 4 to DUMPSTYLE), and added this disk as dump device in BOOT_OPTIONS
The "No supported device(s) found" message disappeared, but it was still complaining about the shadow members :

**** Unable to dump error log buffers to remaining shadow set members
** Dumping memory to device #01 in the DUMP_DEV environment variable

The system disk is a 3 member shadowset ($1$DGA11401,$1$DGA12401 and $1$DGA13401)
After adding these 3 members as DUMP device in BOOT_OPTIONS, this message also disappeared,
but during a crash it causes an additional delay of about 2 minutes to discover the links :

3-JUL-2019 18:20:45.85|EFI VMS_DUMP_DEV_ Options list:
3-JUL-2019 18:20:45.85|
3-JUL-2019 18:20:45.85|Entry Description
3-JUL-2019 18:20:45.85|----- -------------------------------------------------------------------------
3-JUL-2019 18:20:45.86| 1 $1$DKA100 PCI(0|2|0|0) Sas(5000C50077427C29,Lun0)
3-JUL-2019 18:20:46.05| 2 $1$DGA11401 PCI(0|a|0|1) Fibre(22020002AC01C11F,Lun2000000000000)
3-JUL-2019 18:20:46.05| 3 $1$DGA11401 PCI(0|9|0|0) Fibre(21010002AC01C11F,Lun2000000000000)
3-JUL-2019 18:20:46.05| 4 $1$DGA11401 PCI(0|9|0|0) Fibre(23010002AC01C11F,Lun2000000000000)
3-JUL-2019 18:20:46.05| 5 $1$DGA11401 PCI(0|9|0|1) Fibre(20020002AC01C11F,Lun2000000000000)
3-JUL-2019 18:20:46.25| 6 $1$DGA11401 PCI(0|9|0|1) Fibre(22020002AC01C11F,Lun2000000000000)
3-JUL-2019 18:20:46.25| 7 $1$DGA11401 PCI(0|a|0|0) Fibre(21010002AC01C11F,Lun2000000000000)
3-JUL-2019 18:20:46.25| 8 $1$DGA11401 PCI(0|a|0|0) Fibre(23010002AC01C11F,Lun2000000000000)
3-JUL-2019 18:20:46.26| 9 $1$DGA11401 PCI(0|a|0|1) Fibre(20020002AC01C11F,Lun2000000000000)
3-JUL-2019 18:20:46.26| 10 $1$DGA12401 PCI(0|9|0|0) Fibre(21010002AC019E86,Lun2000000000000)
3-JUL-2019 18:20:46.45| 11 $1$DGA12401 PCI(0|9|0|0) Fibre(23010002AC019E86,Lun2000000000000)
3-JUL-2019 18:20:46.45| 12 $1$DGA12401 PCI(0|9|0|1) Fibre(20020002AC019E86,Lun2000000000000)
3-JUL-2019 18:20:46.45| 13 $1$DGA12401 PCI(0|9|0|1) Fibre(22020002AC019E86,Lun2000000000000)
3-JUL-2019 18:20:46.45| 14 $1$DGA12401 PCI(0|a|0|0) Fibre(21010002AC019E86,Lun2000000000000)
3-JUL-2019 18:20:46.46| 15 $1$DGA12401 PCI(0|a|0|0) Fibre(23010002AC019E86,Lun2000000000000)
3-JUL-2019 18:20:46.65| 16 $1$DGA12401 PCI(0|a|0|1) Fibre(20020002AC019E86,Lun2000000000000)
3-JUL-2019 18:20:46.65| 17 $1$DGA12401 PCI(0|a|0|1) Fibre(22020002AC019E86,Lun2000000000000)
3-JUL-2019 18:20:46.65| 18 $1$DGA13401 PCI(0|9|0|0) Fibre(21010002AC01C11E,Lun3000000000000)
3-JUL-2019 18:20:46.85| 19 $1$DGA13401 PCI(0|9|0|0) Fibre(23010002AC01C11E,Lun3000000000000)
3-JUL-2019 18:20:46.85| 20 $1$DGA13401 PCI(0|9|0|1) Fibre(20020002AC01C11E,Lun3000000000000)
3-JUL-2019 18:20:46.85| 21 $1$DGA13401 PCI(0|9|0|1) Fibre(22020002AC01C11E,Lun3000000000000)
3-JUL-2019 18:20:46.85| 22 $1$DGA13401 PCI(0|a|0|0) Fibre(21010002AC01C11E,Lun3000000000000)
3-JUL-2019 18:20:46.85| 23 $1$DGA13401 PCI(0|a|0|0) Fibre(23010002AC01C11E,Lun3000000000000)
3-JUL-2019 18:20:46.86| 24 $1$DGA13401 PCI(0|a|0|1) Fibre(20020002AC01C11E,Lun3000000000000)
3-JUL-2019 18:20:47.05| 25 $1$DGA13401 PCI(0|a|0|1) Fibre(22020002AC01C11E,Lun3000000000000)
3-JUL-2019 18:20:47.05|--------------------------------------------------------------------------------
3-JUL-2019 18:20:47.05|25 entries found.
3-JUL-2019 18:20:47.05|
3-JUL-2019 18:20:47.05|
3-JUL-2019 18:20:50.88|Enter your choice: *EXIT*
3-JUL-2019 18:20:51.68|SYSTEM/ROCK>
3-JUL-2019 18:21:24.55|SYSTEM/ROCK> mc opccrash
3-JUL-2019 18:21:25.70|
3-JUL-2019 18:21:25.73| Quorum: 2 (of 3 votes); this node contributes 1 vote
3-JUL-2019 18:21:25.93| Quorum disk contributes 1 vote.
3-JUL-2019 18:21:35.38|
3-JUL-2019 18:21:35.60|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:21:35.60|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:21:36.00|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:21:36.00|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:21:39.07|
3-JUL-2019 18:21:39.22|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:21:39.22|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:21:39.63|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:21:39.63|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:21:42.60|
3-JUL-2019 18:21:42.65|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:21:42.65|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:21:43.05|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:21:43.25|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:21:46.09|
3-JUL-2019 18:21:46.27|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:21:46.27|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:21:47.68|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:21:47.68|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:21:49.60|
3-JUL-2019 18:21:49.70|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:21:49.70|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:21:50.10|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:21:50.10|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:21:53.12|
3-JUL-2019 18:21:53.32|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:21:53.32|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:21:53.73|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:21:53.73|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:21:56.62|
3-JUL-2019 18:21:56.75|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:21:56.75|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:21:57.15|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:21:57.15|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:22:19.29|
3-JUL-2019 18:22:19.50|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:22:19.50|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:22:19.91|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:22:19.91|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:22:22.76|
3-JUL-2019 18:22:22.93|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:22:22.93|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:22:23.33|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:22:23.33|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:22:26.27|
3-JUL-2019 18:22:26.35|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:22:26.35|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:22:26.75|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:22:26.75|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:22:29.75|
3-JUL-2019 18:22:29.98|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:22:29.98|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:22:30.38|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:22:30.38|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:22:52.95|
3-JUL-2019 18:22:53.14|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:22:53.14|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:22:53.54|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:22:53.54|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:22:56.46|
3-JUL-2019 18:22:56.56|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:22:56.56|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:22:56.96|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:22:56.96|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:22:59.92|
3-JUL-2019 18:22:59.98|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:23:00.19|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:23:00.39|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:23:00.39|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:23:03.40|
3-JUL-2019 18:23:03.61|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:23:03.61|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:23:04.01|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:23:04.01|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:23:06.91|
3-JUL-2019 18:23:07.03|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:23:07.03|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:23:07.44|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:23:07.44|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:23:10.42|
3-JUL-2019 18:23:10.46|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:23:10.46|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:23:10.86|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:23:10.86|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:23:13.88|
3-JUL-2019 18:23:14.08|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:23:14.08|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:23:14.49|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:23:14.49|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:23:17.37|
3-JUL-2019 18:23:17.51|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:23:17.51|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:23:17.91|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:23:17.91|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:23:20.85|
3-JUL-2019 18:23:20.93|PGQBT-I-INIT-UNIT, boot driver, PCI device ID 0x2532, FW 4.04.04
3-JUL-2019 18:23:20.93|PGQBT-I-BUILT, version X-39, built on Sep 19 2017 @ 07:19:02
3-JUL-2019 18:23:21.34|PGQBT-I-LINK_WAIT, waiting for link to come up
3-JUL-2019 18:23:21.34|PGQBT-I-TOPO_WAIT, waiting for topology ID
3-JUL-2019 18:23:24.65|
3-JUL-2019 18:23:25.50|** Dumping memory to device #01 in the DUMP_DEV environment variable
3-JUL-2019 18:23:25.57|
3-JUL-2019 18:23:25.57|**** Starting compressed selective memory dump at 3-JUL-2019 18:21...
3-JUL-2019 18:23:33.02|................................................................................
3-JUL-2019 18:23:51.55|................................................................................
3-JUL-2019 18:24:01.82|................................................................................
3-JUL-2019 18:24:12.09|................................................................................
3-JUL-2019 18:24:21.76|................................................................................
3-JUL-2019 18:24:39.89|...............................................................................
3-JUL-2019 18:24:39.89|** System space, key processes, and key global pages have been dumped.
3-JUL-2019 18:24:40.09|** Now dumping remaining processes and global pages...
3-JUL-2019 18:24:43.71|..........................
3-JUL-2019 18:24:43.71|...Complete ****
3-JUL-2019 18:24:43.71|
3-JUL-2019 18:24:43.71# SYSTEM SHUTDOWN COMPLETE

Questions :

If we have a local DOSD dump disk, is it still required to add the system disk shadow members as dump devices ?

What is the disadvantage of not defining the dump device in BOOT_OPTIONS (and ignore the warning messages) ?    Are the error log buffers not saved anyway into the dumpfile ?

Where are these errlog buffers saved, I can't see any new file created on the local disk :

$ dir dka100:<000000...>*.*/size/nohead/notrail/wid=file=60
DKA100:<000000>000000.DIR;1 1
DKA100:<000000>BACKUP.SYS;1 0
DKA100:<000000>BADBLK.SYS;1 0
DKA100:<000000>BADLOG.SYS;1 0
DKA100:<000000>BITMAP.SYS;1 4377
DKA100:<000000>CONTIN.SYS;1 0
DKA100:<000000>CORIMG.SYS;1 0
DKA100:<000000>GPT.SYS;1 96
DKA100:<000000>INDEXF.SYS;1 1564
DKA100:<000000>SECURITY.SYS;1 1
DKA100:<000000>SYS107.DIR;1 1
DKA100:<000000>VOLSET.SYS;1 0
DKA100:<SYS107>SYSEXE.DIR;1 1
DKA100:<SYS107.SYSEXE>SYSDUMP.DMP;1 2000000

Bart

[Volker Halle]

Bart,

the errorlog buffers in memory are saved in the dumpfile and can be manually extracted from SYSDUMP.DMP with

$ ANALYZE/CRASH dev:<dir>SYSDUMP.DMP

SDA> CLUE ERRLOG   ! saves errlog entries into CLUE$ERRLOG.SYS

SDA> EXIT 

But the errlog entries saved in the dumpfile will not be (automatically) added into SYS$ERRORLOG:ERRLOG.SYS

The crash code also tries to save the errlog entries from memory into SYS$SYSTEM:SYS$ERRLOG.DMP. The errlog entries from that file will be automatically read during boot (by SYSINIT) and put into SYS$ERRORLOG:ERRLOG.SYS.

See:  http://www.decus.de/slides/sy2004/20_04/1l04.pdf

Volker.

[quorum]

Thanks Volker, I missed the fact that the errlog buffers are never saved on DOSD disk, but always to systemdisk.

If we have defined DUMP device in BOOT_OPTIONS as

Local DOSD disk, system disk shadow member 1,system disk shadow member 2, system disk shadow member 3

Am I right that

- system dump will be saved to first reachable disk (most probably the local disk)

- error log buffer will be saved to ALL of the 3 shadow members, so that whatever disk we reboot from, sysinit/errfmt will find the saved error entries ?

[Volker Halle · Accepted solution]

Bart,

during a crash, the errlog buffers will be saved first to the system disk (SYS$ERRLOG.DMP) and - if using a shadowed system disk - to all currently valid shadowset members of the system disk as defined and reachhable via DUMP_DEV_nn.

Then SYSDUMP.DMP will be written to the DOSD device. The DOSD device must also be defined via a DUMP_DEV_nn entry.

Volker.