Open VMS system log requirement.

[Anjan Ganguly]

I have one Open VMS (Version 7.2-1) alpha DS10 system where my customised applications are running.On 11-Feb-2011 at 18:00:00 hrs the system gets hangged and and customised applications stops functioning.I have checked with the customised applications logs but not able find out any problem from it.But somebody has restarted the system at around 18:30:00 hrs and restored the normal function of the running applications.Since I am not able to understand the state of the system from my application logs, I like to know that is there any way to see the system activity (or any kind of logs), from where I can make out what exactly happened just before the hang and from before the restart i.e. from 17:55:00hr to 18:29:00 hrs.Can some body help me in this regard.Basically an History is required for me to analyze the system activity that causes the trouble.

[Volker Halle]

Anjan,

by default, you only have the accounting file, operator log and error log available on OpenVMS for after-the-fact system analysis. If you are running some performance analysis tools (e.g. T4 or PSDC), you may also be able to extract some data from those utilities.

Did just your application hang or the complete OpenVMS system ? Has the OpenVMS system been rebooted - check $ WRITE SYS$OUTPUT F$GETSYI("BOOTTIME") ? Maybe the system crashed ($ TYPE CLUE$HISTORY) ?

Volker.

[Anjan Ganguly]

Yes system was rebooted and as per the physical Inspection at the time of the incident we are not able to login to the system from a remote machine.I have my system connected in LAN.

[Volker Halle]

Anjan,

so we are talking about a possible 'system hang'. The reason could be software or hardware. In a situation like this, one should first try to PING the system. If that works, you know that at the the IP drivers still work. If if fails, it may be a network problem or a hardware hang.

Next go the the console terminal and try to login. Do you get a Username: prompt ? If the system appears to hang, press the HALT button, then - if the console prompt appears - type >>> CRASH

If it's not a hardware hang, this will cause a system crash to be written and the system will automatically boot afterwards. You can analyze the system crash and try to find the reason for the hang.

If the system does not react on pressing the HALT button, it's a hardware hang and it may be very hard to analyze that.

Volker.

[MichelleP_1]

For the future, you might also take a look at installing and using Availability Manager (http://goo.gl/YsTWX).

You'll need the Data Collector running on the host and the Analyzer running on at least one other VMS or Windows system. You may still have to crash the system, but you just might be able to: 1) see what's happening first, 2) avoid a crash with one of the available "fixes", and/or 3) get more information from the Analyzer log file about what was happening during that time.

The Data Collector runs at the device driver level, so information/access may be available even when the system is hung (just not crashed completely).

[Anjan Ganguly]

Volker,

I was able to ping at the time of hang.But as the event has already occured on 11-Feb-2011 at around and after that the system is rebooted manually, can I analyze the root cause of the so called hang/crash.Actually I like to know what exactly happened during the period i.e. from 17:45hrs to 18:30 hrs on 11-Feb-2011.Is there any way?And does crash gives the details happened before manually reboot?

[Anjan Ganguly]

MichelleP ,

Can u please tell me how to see the system events happened by availability manager.I have that installed in my system.So How can I see the PAST Events (Events just before the reboot)?

[MichelleP_1]

Anjan,

Did you have the Analyzer running on a separate system that was monitoring the node in question?

If so, go to that system and you'll find log files in the program installation directory. Mine (on a Windows node) are in C:\Program Files\Hewlett-Packard\HP Availability Manger V3.0-1. Look for the log file(s) for the particular time frame, starting with AnalyzerEvents_Conn0_.log. This should have all the same events that show up in the bottom half of the Analyzer main window.

If your analyzer node is OpenVMS, check the *AMDS* logicals for the location. I believe you'll find them in AMDS$AM_LOG or more specifically AMDS$AM_ROOT:[AMDS$AM]

The logs can be quite large and "noisy" depending on how much customization you've done with the thresholds and the activity of your system.

I hope this helps. I've used these logs many times for post-event analysis.

[P Muralidhar Kini]

Hi Anjan,

Check the following links for HP Availabity manager details -
http://h71000.www7.hp.com/openvms/products/availman/docs.html

For Documentation, refer
http://h71000.www7.hp.com/openvms/products/availman/6552pro.html
* Chapter 5 - Getting Information About Events

To understand your problem, you need to provide the details
1) Was it a process hang or system hang ?
2) How was the system rebooted ? Was it force crashed ?

Regards,
Murali

[Volker Halle]

Anjan,

for after-the-fact analysis on OpenVMS, by default you only have OPERATOR.LOG, Accounting and errorlog and any application logfile. In case of a system hang, only a forced crash will help you, in case of an application hang, you might be able to do some online analysis, if you have the time and know what to look for. A forced crash is ALWAYS better than just a boot, as with a plain boot, you losse ALL the state information of the system (including errlog entries not yet written to the disk).

To do after-the-fact analysis with AMDS, you need to have it running continuously (on the client running the AMDS GUI) or at least start it, if the system/application hangs.

Volker.

[David Phelps]

Anjan, When your system rebooted after 18:30, VMS would likely have started a new generation of your OPCOM log (sys$manager:operator.log). A good place to start is to identify the prior version of your OPCOM log, and type out the last 100 or 200 line of it.
DIRECTORY/DATE/SIZE=ALL sys$manager:operator.log
Identify the OPCOM log that was recording up to the time of the system hang.
type/tail=200 sys$manager:operator.log;-1 (for example, for one generation prior to the current OPCOM log)

[Anjan Ganguly]

Dear David

In my system I can see only last 2 versons of Operator.log in sys$manager directory.
So I was not able to make out form the latest one as I do not get any clue for what the system goes down.I have only the messages when the system started after reboot.
Is there any way to increase this no. of files of OPERATOR.log to store.Does the system decide any way that how many files of Operator.log it should keep !!!!!!!!!!!

[Jan van den Ende]

Anjan,

by default the system keeps ALL of them.

Somewhere/somehow YOU (or your system) does the PURGing.
The most likely place would be during bootstrap.
Carefully check your bootstrap procedure (and the things called from it) for
$ PURGE OPERATOR.LOG
(be aware of command abbreviations and LogicalNames!!).

A BACKUP of shortly after 11-feb might well still contain that version. Obviously you have rebooted once again since then, OR, you have a Logfile refresh-and-purgr procedure active. (if so, then THAT should also be modified to retain more logfiles)

hth

Proost.

Have one on me.

jpe

[Jon Pinkley]

Anjan,

It is also possible the file is set to autopurge to two versions. Since it is easy to check for this possibility, I would check that before searching for something that is using the purge command.

Do a directory/full sys$manager:operator.log and look at the end of the line that begins with "File attributes: "

If it does not say "No version limit", then something has done a set file/version=x on the file. This version limit is not really an attribute of the file itself, but of the directory entry for the file name.

Use set file/version=x to set a higher limit, or set to 0 for "No version limit"

Jon