topic Re: SYSDUMP.DMP - retaining contents of in Operating System - OpenVMS

SYSDUMP.DMP - retaining contents of

John A. Beard — Fri, 30 Jan 2009 22:08:55 GMT

Hi folks,

I would be grateful for some advice on the issue of retaining the contents of SYSDUMP.DMP and SYS$ERRLOG.DMP (plus other files that might get used when a crash occurs)

Environment: AlphaServer ES47 (partitioned) running OpenVMS 7.3-2. DUMPSTYLE is set to 9

Irrespective if there was a system crash or a complete power failure, can you tell me what I need to do to make sure the contents of these files are not overwritten or reinitialized when the system is brought back up. We want to ensure that valid content is available for examination by HP's technical staff if needed.

I am aware of using ANAL/CRASH to copy the contents to a seperate file, or alternatively using BACKUP with /IGNORE=NOBACKUP. My question really relates to what happens once we have had a failure (system crash or perhaps a prelonged total power outage)

For this example, let's say that the Restart control setting is set to HALT. If we are at the cheveron prompt, will simply performing a straight BOOT erase the contents of either of the above files, or perhaps not provide enought information for later diagnosis. if the later was true then do I need to issue a CRASH command to write additional information out to the dump files?

The reason I made reference to a power outage was because we have recently experienced two outages that cut power to the CPU. Even if a straigh power loss would not write informaton out to the dump files, I would still like to know what is the safeguard steps we need to take when a 'normal' crash occurs.

Below is a brief extract form HP describes part of the problem....

When the system went down last night it generated a CDL or crash data log. The CDL was written to the VMS error log. In the case, I can see that it was a CLOCK POWER FAULT VOLTAGE FAILURE. The CPU module has VRM’s or voltage regulator modules on the board. Theses VRM’s can power down the module/partition. On earlier versions of the firmware, they had problems with False environmental errors. They would sense errors when there really were none. I’m recommending two things. One, upgrade the firmware because there were problems in earlier versions. I don’t think this is the fix but it should be done and Two, replace the DUO/CPU module because the VRM’s are on it.

Re: SYSDUMP.DMP - retaining contents of

Steven Schweda — Fri, 30 Jan 2009 22:41:28 GMT

I thought that the whole idea of a system
dump file was to have something to examine
after you re-boot. What good would it be if
it got wrecked by a re-boot?

You can expect it to get wrecked
(overwritten) by the next _crash_.

Re: SYSDUMP.DMP - retaining contents of

John A. Beard — Fri, 30 Jan 2009 22:45:25 GMT

I appreciate that the contents will get overwritten should there be another crash. My original question related to what happens the contents of the dump files when a boot is performed....what happens to the original contens...stays intact, do we need to perform any other operation to ensure we retain as much useful info as possible.

Re: SYSDUMP.DMP - retaining contents of

Jan van den Ende — Sat, 31 Jan 2009 09:07:36 GMT

John,

it just stays intact.

Now, _IF_ you use the PAGEFILE for dumping (remember the prices of disks, say 20+ years ago) _THEN_ you _MUST_ save it before regular system use resumes. The construct can still be found in use, but at todays prices it really deserves a review.

Nowadays, the second reason for saving takes precedence: ANOTHER crash before this one has been analysed.

Adding the ANAL/CRASH COPY command to the boot sequence is the prefered, simple way to get around that. Int hat case, a review of the need to retain old copies after a crash is advisable.

This combination covers for all cases, AFAIK.

hth

Proost.

Have one on me.

jpe

Re: SYSDUMP.DMP - retaining contents of

John A. Beard — Sat, 31 Jan 2009 11:01:24 GMT

Thanks for the info Jan,

Where should we apply the ANAL/CRASH COPY cmmand in the startup sequence. Do we modify one of the DEC (ooops, HP) site independent procedures or simply insert it in he begining of systartup_vms.com

Re: SYSDUMP.DMP - retaining contents of

John A. Beard — Sat, 31 Jan 2009 12:32:49 GMT

Most of my enquiries stem from one of my colleagues telling me that if the system crshes and you simply perform a straight reboot, then you loose all the information relating to what was happening on the system at the time of the crash.... just trying to get to the bottom of this.

I may have this all wrong but when a system crashes I gather that it takes a snapshot of main memory but does not actually write to the dump files at that time, but rather to a series of snapshot files.

When the system eventually reboots, the boot process checks the details about the previous system shutdown and availablity of these snapshot files. If it finds that previous shutdown was not normal, and there are snapshot files, then it will start to build the DUMP file using the information in the snapshot file. After the system is up, the critical information relating to the crash would then be available for analysis.

If the above was in any way correct, then is there a need to execute the CRASH command at the cheveron prompt?
______________________

An additional question in rgards to another dump file that appears in discussion threads along with SYSDUMP.DMP, SYS$SYSTEM:SYS$ERRLOG.DMP

Should this file (like sysdump.dmp) also be copied as part of the boot sequence to preserve its contents, and if so what is the command to do so...

I apologize if this is starting to turn into a one stop session in all things relating to crashes, but I really would like to get a better initial grasp as to what is the correct course of action.

Re: SYSDUMP.DMP - retaining contents of

Volker Halle — Sat, 31 Jan 2009 13:04:57 GMT

John,

if a system crashes, it first dumps the current in-memory errlog entries to SYS$SYSTEM:SYS$ERRLOG.DMP, then it tries to write system memory into SYSDUMP.DMP.

During boot, the errlog entries from SYS$ERRLOG.DMP are automatically extracted and appended to ERRLOG.SYS, so you don't loose any possible errlog entries from the time immediately preceeding the crash. There is no need to copy/save SYS$ERRLOG.DMP.

CLUE$STARTUP is automatically run during startup and examines SYSDUMP.DMP. If the dumpfile has not yet been been analyzed, it creates the CLUE summary text file of the crash footprint in CLUE$COLLECT:CLUE$node_ddmmyy_hhmm.LIS.

If you want to automatically save a copy of the SYSDUMP.DMP file after a system crash, you should use the CLUE$SITE_PROC logical name (please see the SDA documentation for details):

http://h71000.www7.hp.com/doc/82final/6549/6549pro_001.html#invokin

You only need to use the >>> CRASH command under 2 circumstances:

- if you want to force a crash on a hung system. Type CTRL-P (or press HALT button), then type >>> crash

- if the system has halted by itself (should not normally happen on OpenVMS) and you did NOT have AUTO_ACTION set to RESTART. Typing >>> CRASH will then try to write a dump. With just >>> B in this case, you will loose ALL information about the problem, as there will be no errlog entries saved and no dump written.

Volker.

Re: SYSDUMP.DMP - retaining contents of

John A. Beard — Sat, 31 Jan 2009 14:39:19 GMT

Thanks Volker..... You explaind things really well

We were debating the issue of having the system restart control set to HALT or RESTART, but that was primarily related to issues surrounding various applications automaticly restarting where there might be problems.

So if I'm reading things correctly, if we have it set to HALT and the system crashes, the only way we can guarantee saving all the valid information would be to initiate the CRASH command... and conversely, if we did have it set to RESTART then we would not have to initiate CRASH.

Re: SYSDUMP.DMP - retaining contents of

Volker Halle — Sat, 31 Jan 2009 15:46:42 GMT

John,

under normal circumstances, OpenVMS will not issue a HALT instruction in kernel mode. But there are some situations, in which the CPU may halt due to some error (e.g. a kernel stack not valid halt) or the PC gets loaded with an address, which points to data instead of instructions and there happens to be a ZERO in that data stream.

If you have not >>> SET AUTO_ACTION RESTART, you will typically loose all context information in these case, as a normal operator would just type >>> B, if he sees the system at the console prompt.

Also in case of those unexpected HALTs, the system will remain down until operator intervention. With AUTO_ACTION = RESTART, the system will write a dump and reboot automatically (depending on the setting of BUGREBOOT, default=1).

Volker.

Re: SYSDUMP.DMP - retaining contents of

John Gillings — Sun, 01 Feb 2009 20:41:25 GMT

John,

Preserving the dump by taking a complete copy with the SDA COPY command is obviously the "gold standard", but even with cheap disk space, dumps are large, and there is a limit to how many can be retained in the long term.

All is not lost if you accidently overwrite a dump, as the system automatically gebnerates a CLUE log on reboot if the system crashes. See SYS$ERRORLOG:CLUE$node_date_time.LIS. These are text files containing the crash "footprint", whicch is often sufficient information to identify the reason for the crash. See SYS$STARTUP:CLUE$STARTUP.COM.

Re: SYSDUMP.DMP - retaining contents of

John A. Beard — Mon, 02 Feb 2009 12:54:54 GMT

Hi John,

Thank you for your input...

Where should we apply the ANAL/CRASH COPY cmmand in the startup sequence... site independent procedures or simply insert it in he begining of systartup_vms.com .

We backup everything to a newtwork storage system (retention policy by default set to 365 days), so once we were to get a copy of the dump file we can delete the copy without hogging disk space.

This is the last question I have on the subject and I would like to thank everyone for their very detailed and clear responses...you guys are and endless source of useful information

Re: SYSDUMP.DMP - retaining contents of

Volker Halle — Mon, 02 Feb 2009 13:05:23 GMT

John,

it is suggest to use the CLUE$SITE_PROC method to copy the dumpfile during startup. This is the offically recommended method to save dumpfiles.

Recommended method:

Create an SDA procedure that includes the COPY command
Ensure the COPY command is the last command
Define logical name CLUE$SITE_PROC (/SYSTEM)
If using DOSD, ensure disk is mounted and logical name CLUE$DOSD_DEVICE defined

Possible difficulty
If data disks not mounted, collection may not work
May prefer to delay copy, but will lose dump with another crash
Do separate collection with COLLECT /SAVE (I64 only)

Using this method also prevents the dumpfile from being copied on every boot.

Volker.

Re: SYSDUMP.DMP - retaining contents of

Volker Halle — Mon, 02 Feb 2009 13:10:31 GMT

John,

here is a pointer to the presentation of the OpenVMS engineer, who is mainting this part of OpenVMS:

http://de.openvms.org/TUD2006/19_Making_the_most_out_of_a_system_crash.pdf

Consider this information to be the ultimate resource about system dumps. This presentation is from 2006, but it's the best you can get without going to an OpenVMS Bootcamp like event yourself.

Volker.

Re: SYSDUMP.DMP - retaining contents of

John A. Beard — Mon, 02 Feb 2009 13:23:36 GMT

Excellent resonses to my questions..thanks folks.