Diagnostic log file grows huge

[Stuart Powell]

When I run "/sbin/init.d/diagnostic start" I find that the log file diaglogd_activity_log in directory /var/stm/logs/sys starts growing very large, rather quickly. The contents from the head of the file are shown below.

th9kkux.thcvnet.hercfilm
root
A.03.00
diaglogd
driver_name
A.01.02
tllibio
driver_name
A.03.00
diaglogd
driver_name
A.01.02
tllibio
driver_name

It continues this patter through out the files. Any Ideas?

Stuart

[Bill Hassell]

A runaway diagnostic log seems to occur right after installing the diags -- they need some time to 'cook', to prepare internal files. If this is interrupted, runaways seem to occur. Download the latest version of the diags (not from the CD) and reinstall. Once installed, don't reboot for a few hours.

[Stuart Powell]

I have downloaded the Support tools and have installed version A.52.00. I'm still seeing the same problem with these new entries in the /var/stm/logs/sys/diaglogd_activity_log file every minute:
A.01.02
tllibio
driver_name
A.03.00
diaglogd
driver_name
There are not patches listed for HP-UX 11.11.

Any ideas on what I might be missing.

[Andrew Merritt_2]

Hi Stuart,
That's not a plain text file. To see what message is being logged (which you'll need to do to know how to stop it), run STM and view the diaglogd activity log.

If you use 'cstm', use 'dacl' and select 'diaglogd'. In 'mstm' or 'xstm', select "System | Daemons | Daemon Activity Log ...| diaglogd".

Andrew

[Stuart Powell]

I ran cstm on the diaglog file and got the following entry repeating endlessly:

Tue Aug 29 13:04:27 2006: Diaglogd daemon failed to get hardware path for I/O error entry to be logged. Entry will be logged with NULL hardware path.

Tue Aug 29 13:04:27 2006: The io_query call failed with io_errno (15) when attempting to get the Context Dependent I/O module name for the device. An STM_KEY_TOKEN_DEF (15) io_errno indicates that the specified item does not exist in the I/O tree node.

Possible Causes/Recommended Action:

The driver is not placing the data in the I/O tree node. This could be a defect or it may merely mean the the driver does not implement this feature.

The device is misconfigured. Reconfigure the
device making sure that the correct device driver is utilized.

However, this doesn't help me solve the problem situation. Any ideas.

Stuart

[Andrew Merritt_2]

Hi Stuart,
I appreciate that you don't yet have the answer, but we can make some progress now we know what the messages are.

Check you have a recent version of OnlineDiags installed; if not, upgrade.

If you have a support contract with HP, I would recommend thinking about opening a call with them at this point.

The other thing to do is to work out which devices are referred to. The lines immediately before the ones you've quoted will include the device paths which are causing the problem. You'll need to correlate these with an ioscan output to see if there's something odd about them. What are they for, when were they created, does rerunning 'ioscan' (without the -k option) clear things? When did the problem start? Has new hardware been added?

Sorry I can't give a definite answer at this stage, but basically STM is trying to access the hardware, and is getting an unexpected response when it does so. It should cope with any supported devices, so the most likely explanations are either you've got an old version of OnlineDiags and some new hardware, or there's something bogus about the device paths.

Andrew

[Andrew Merritt_2]

Ok, just realised you do have A.52.00, so ignore the comments about upgrading.

Andrew

[Stuart Powell]

Andrew;
Your thoughts were very helpful in troublshooting. A few weeks ago I had to change the PID setting on the switches in our SAN to accomidate a newer switch. When I changed that value and reset the switches all of the SAN hardware was assigned new instance numbers. I had not cleaned everything up on the systems that are experienceing the hardware problem. So I went in and got rid of most of the special files and addresses associated with the NO_HW.
Now I have four entries for each switch, one for each port that did have array device:
ext_bus 26 8/4/1/0.3.30.0.0 fcparray NO_HW INTERFACE FCP
Array Interface
ext_bus 16 8/4/1/0.3.30.255.0 fcpdev NO_HW INTERFACE FCP
Device Interface
ext_bus 0 8/4/1/0.3.31.0.0 fcparray NO_HW INTERFACE FCP
Array Interface
ext_bus 2 8/4/1/0.3.31.255.0 fcpdev NO_HW INTERFACE FCP
Device Interface
I will begin looking how to get rid of these ioscan entries. I can't reboot the systems at this time.
The diaglogd_activity_log is still growing.
Stuart

[Andrew Merritt_2]

Hi Stuart,

Just running 'ioscan -fn' should do what you need.

If it doesn't, it's safe to delete the dialogd_activity_log file if it's getting too large, that shouldn't break anything.

Andrew

[Andrew Merritt_2]

The other point I just realised I haven't mentioned is that the reason diaglogd is logging the errors is because it is dealing with events passed by the device driver, and doesn't recognise the path name. The point being that one of the drivers is detecting some sort of hardware problem, most likely when a device is being accessed. You may be able to see what these errors are by using 'logtool' in STM (I'm not sure exactly whether the problem with the unrecognised paths would prevent the events appearing in logtool or not).

It could be a problem due to the ioscan data being out of sync, so could resolve itself when that is sorted.

Andrew