/var/adm/messages empty

[Darrell Tschakert]

Hi,
We have a 9000 that no longer updates it's /var/adm/messages file properly.
Below is the output of the following commands for the problem machine and a

working machine:

uname -a
who -r
ll /var/adm/messages
ll /var/adm/msgbuf

PROBLEM MACHINE

HP-UX imsp1 B.11.00 U 9000/800 ... ...

. run-level 3 Jan 7 16:56 3 0 S

imsp1
-rw-r--r-- 1 root root 0 Jan 7 16:57 /var/adm/messages
-rw-r--r-- 1 root sys 4096 Dec 13 16:55 /var/adm/msgbuf
=================================================


PROPERLY WORKING MACHINE

HP-UX imsp3 B.11.00 U 9000/800 ... ...

. run-level 3 Oct 13 13:57 3 0 S

-rw-r--r-- 1 root root 4322 Dec 30 17:55 /var/adm/messages
-rw-r----- 1 root sys 4096 Dec 30 17:55 /var/adm/msgbuf
=================================================

Even though the messages file is zero length, entering "dmesg" gives me the

current information from the last boot. Entering "dmesg -" gives me nothing.

The permissions and ownerships on /var and /var/adm are the same.

Both machines have this line in there root crontabs:

05,15,25,35,45,55 * * * * /usr/sbin/dmesg - >> /var/adm/messages

Any ideas as to why our messages file is always zero length?

For testing purposes, can anyone tell me of a simple way to cause additional

entries to be added to the messages file on a properly working machine. For

example, pulling the Network cable from the machine will generate an error, but I

would like something that I can use from the command line.

Thank you,

Darrell Tschakert
EEOC

[Ken Hubnik_2]

Not sure if it matters but the permissions are different of this file.

rw-r--r-- 1 root sys 4096 Dec 13 16:55 /var/adm/msgbuf

-rw-r----- 1 root sys 4096 Dec 30 17:55 /var/adm/msgbuf

[Patrick Wallek]

I just looked at 2 of my machines and /var/adm/messages is 0 on 1 machine and does not exist on the other.

In the current versions of HP-UX (10.X and 11.x) I do not think that /var/adm/messages is used.

'dmesg' doesn't use the /var/adm/messages file. The /var/adm/syslog/syslog.log file is the normal logfile for HP-UX.

[John Poff]

Hi,

I think your /var/adm/messages file gets updated by the dmesg program. Typically the dmesg gets run by cron and redirects output to /var/adm/messages. Here is a snippet from 'man dmesg':


dmesg looks in a system buffer for recently printed diagnostic
messages and prints them on the standard output. The messages are
those printed by the system when unusual events occur (such as when
system tables overflow or the system crashes). If the - argument is
specified, dmesg computes (incrementally) the new messages since the
last time it was run and places these on the standard output. This is
typically used with cron (see cron(1)) to produce the error log
/var/adm/messages by running the command:

/usr/sbin/dmesg - >> /var/adm/messages

every 10 minutes.



Check to see if root's crontab is different between the two boxes, and see if the one that has messages working has dmesg in crontab.

JP

[Pete Randall]

The bad (0 length) file is happening because your cron job is doing dmesg -, which you said doesn't work. So, the question is, why doesn't dmesg - work?

Pete

[Pete Randall]

Darrell,

What does your /var/adm/msgbuf file look like?

$ ll /var/adm/msgbuf
-rw-r--r-- 1 root sys 4096 Jan 10 09:31 /var/adm/msgbuf


According to the man page, that is the "memory scratch file for - option"


Pete

[Pete Randall]

Duhhhh! Never mind. You already posted that.

Pete

[Pete Randall]

The date on your failing machine's /var/adm/msgbuf is old. Try renaming it (or moving it somewhere else) and running dmesg - when I tried it, my msgbuf was re-created.

Pete

[John Poff]

Hi,

Ok. Now I've had some coffee and read your ENTIRE post this time [boy, I'm glad it's Friday].

I'm not sure what is wrong. For a suggestion to get something in the message buffer, I noticed on one of my 11.00 boxes that it complained about an NFS mount. Do you have any NFS mounts you could temporarily make it complain about?

Too bad that dmesg doesn't seem to have anything like the 'logger' command for the syslog.

JP

[Darrell Tschakert]

I should have mentioned that we have seven machines. Six of them handle the messages file perfectly. The one does not.
Pete. "dmesg -" is working properly. It behaves that way on all the other machines yet they have regularly updated messages file. The msgbuf file looks just like what I would expect it to look like, only it is outdated. I know the msgbuf file is old - that's part of the problem.
John, I have a pretty good understanding of "mesg -". I put the line in the man entry in each of our seven HP
9000's a few months ago and they all work fine except this one.
Ken, You are right about the permissions on the file - good eye. I gave a poor example. However, I have messages files on other working machines with the same permissions as the none working machine.

I was hoping that this was one of those common/easy problems that was quickly fixed. Not so. I think I'll just live with it. I can always ru "dmesg" and look at the output.

thanks,
Darrell Tschakert

[Pete Randall]

Darrell,

Back to your original question - can anyone tell me of a simple way to cause
additional

entries to be added to the messages file on a properly working machine.

How about copying /stand/vmunix (or some large file) to an inconsequential, nearly full file system. If you can fill one that won't hang your machine, that should generate vx-nospace messages.

Pete

[Frank Slootweg]

Did you follow Pete's January 10, 2003 14:41 PM GMT advice, i.e. rename msgbuf and then run "dmesg -"?

Also do you see anything special when you do

dmesg >/tmp/messages # Note *no* "-".
diff /var/adm/msgbuf /tmp/messages

I expect the diff(1) to give a *few* differences, but not many.

[Darrell Tschakert]

Frank,
Running "dmesg - " after renameing mgbuf gives me a new msgbuf.

Comparing msgbuf to the output of dmesg tells me that the first line is very slightly different - msgbuf has some binary numbers and the other has the date at the beginning of the first line.


Darrell T

[Frank Slootweg]

So, *except for the empty /var/adm/messages file*, everything looks normal!

Isn't it possible/likely that someone/something has just emptied /var/adm/messages?

If I were you, I would just do:

dmesg >>/var/adm/messages # Note *no* "-".

Based on your findings, that should create a correct/non-empty messages file.

Then, after the first few crontab-invoked runs of dmesg, check again if the messages file is still correct/non-empty. Then check every hour, etc.. Hopefully that will tell if/when the messages file is emptied again.

FYI, we have a (10.20) system, which I do not manage, which also has an empty messages file. On that system, the messages file has (about) the timestamp of the system boot.

[Darrell Tschakert]

Frank,
No, it is not possible that someone just emptied the messages file. I would not post just because of one zero length messages file. I should get a new, none zero length messages and msgbuf file everytime I boot - that's the way the other machines work. This machine gives me a zero length messages file every time I boot, and it's been like this since I came to this job months ago. That is the whole problem - messages file is not properly created at boot.

I could do a "dmesg > /usr/adm/messages" every time I boot. But, I don't want to. I would rather have it work the way it is supposed to work. I would like a new msgbuf and messages every time I boot AND I want messages to have some size.

If I run dmesg into the messages file, it does not go to zero length until I reboot - you must have misunderstood what I meant to say.

Darrell T

[Bill Hassell]

dmesg does not log anything to any file. It is simply a tool to read the kernel's message buffer. The dmesg - option is used to suppress already displayed text. Once you've typed dmesg -, all subsequent dmesg - commands will be empty until there is something new in the kernel's buffer.

The only way to get dmesg data into a file is by script, typically through cron. I personally don't like the filename 'messages' since it does not relate to the command that produces the logfile, so my cron entry is:

5,15,25,35,45,55 * * * * /sbin.dmesg - >> /var/adm/dmesg.log

The /usr/newconfig directory has a sample dmesg log entry where the 'messages' filename is used.

[Frank Slootweg]

OK, so the problem only occurs at boot.

The only thing I can suggest then is to *disable* the crontab entry, boot and look how things look then, i.e. is there a messages file, if so what is in it, does the first (crontab-based, let it only run every half) hour or so) "/usr/sbin/dmesg - >> /var/adm/messages" correctly create/update the messages file, etc..

I.e. *something* must be/go wrong on the machine during boot and, after the fact, we can not find out what it is.

[Frank Slootweg]

Out of interest, I looked into this some more.

On a standard system, /var/adm/messages is cleared by /sbin/rc2.d/S206clean_adm and cron(1M) is started by /sbin/rc2.d/S730cron.

It would be interesting to see how that is on your system and to check /etc/rc.log to verify that S206clean_adm and S730cron are indeed run in that order and to check /var/adm/cron/log to see when cron was started and when "/usr/sbin/dmesg - >> /var/adm/messages " was run for the first time.

If you really want to know, you could put extra debugging statements in the S-scripts to see when things are still OK and when they are wrong. But please do so only in a copy of these scripts (i.e. preserve the original timestamps, etc.) and be *very* careful, because errors in the rc(1M) related scripts can very easily lead to a non-bootable system. To be on the safe side, move the original scripts completely out of the way, i.e. for example to /.

[Darrell Tschakert]

CLOSED -GIVEN UP ON THIS PROBLEM - THANKS

Frank,
Thank you for your reply. Yes the /sbin/rc2.d files appear to be running in the proper order. The rc.log files shows the S206clean_adm output on line 362 and the S730cron output at line 601. /sbin/rc2.d and /sbin/init.d/* files are also exactly the same as on the properly working machines - right down to size of the files that they are linked to, perms, etc., etc. It all look the way I would expect it to look.

The S206clean_adm appears to be doing it's work since it religiously mv's the old messages to OLDmessages and zero's out a new messages file. However, it appears never to get the first output of dmesg AND msgbuf never gets refilled. It seems that there is something missing - something like "rm msgbuf; dmesg > messages" I have grep'ed all of the /sbin/rc?.d/* and /sbin/init.d/* files for "messages", "msgbuf", and "dmesg" but found no leads.

It's time to give up on this one. I will take your advice and comment out the "... dmesg - >> /var/adm/messages" line from root's crontab and see what happens the next time we reboot the problem machines. I say machines (plural) now because I have found another machine out of order - earlier it was one bad machine out of seven. We have decided to decommission these very two machanes anyway. The problem never was critical, but it is bothersome since some of my shell scripts depend upon a messages file. I was spanked by one of my bosses yesterday for having wasted so much time on this problem. I only post now because I am at home on my own time.

Thanks,

Darrell Tschakert