topic Re: Getting error using dmesg in Operating System - HP-UX

Getting error using dmesg

Sushil Singh_1 — Thu, 02 May 2002 00:32:44 GMT

Hi All friends,
Whenever I use dmesg, I get the following output. Please help.
vxfs: mesg 001: vx_nospace - /dev/vg00/lvol5 file system full (1 block extent)

vxfs: mesg 001: vx_nospace - /dev/vg00/lvol5 file system full (1 block extent)

vxfs: mesg 001: vx_nospace - /dev/vg00/lvol5 file system full (1 block extent)

But when I do bdf , the following is the output

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 212992 62613 141021 31% /
/dev/vg00/lvol1 207381 27272 159370 15% /stand
/dev/vg01/lvol11 2097152 1020284 1009615 50% /vendor
/dev/vg00/lvol7 1048576 842086 194630 81% /var
/dev/vg00/lvol6 1048576 673305 351838 66% /usr
/dev/vg01/lvol15 102400 44740 54118 45% /user_files
/dev/vg01/lvol14 4194304 3724242 441056 89% /u01/app/oracle
/dev/vg01/lvol12 524288 14251 478194 3% /u01/app/home
/dev/vg01/lvol13 1470464 1107992 344713 76% /u01/app/asms
/dev/vg00/lvol5 1576960 31310 1453738 2% /tmp
/dev/vg00/lvol4 524288 458009 62179 88% /opt
/dev/dsk/c3t2d0 584580 584580 0 100% /SD_CDROM

Re: Getting error using dmesg

A. Clay Stephenson — Thu, 02 May 2002 00:40:27 GMT

Hi:

The dmesg buffer is not 'real time' as is the bdf command. At some earlier time, /tmp was filling up and what you are seeing is an artifact of that event. The process that was writing the large files probably died and as soon as that happened the filesystem space was returned. Those messages will stay in the circular dmesg buffer until the contents are overwritten with new syslog messages.

Re: Getting error using dmesg

Michael Tully — Thu, 02 May 2002 00:44:49 GMT

Hi,

This message indicates that /tmp was full
at some stage after the last reboot. The
best place to check is in the
/var/adm/syslog.syslog.log file to see
when. Given that you bdf output no longer
shows the filesystem as being full indicates
that the process or processes that filled
the filesystem have terminated.

One good thing is you can have a look in
your log file (name as above) to see
whether this is a one-off occurance. If
it is not, this is a good indicator that
you may need to either increase or point
your working files to a different filesystem.
Generally some of these come from the
application being used.

HTH
~Michael~

Re: Getting error using dmesg

Sushil Singh_1 — Thu, 02 May 2002 00:45:24 GMT

Thanks Mr Stephenson,
That make more sense. Is there any way I can reset the buffer instead of rebooting this machine. Some time I get this error on production machine and its very hard for us to reboot that machine. Also is there any way to find out the culprit which is causing the dmesg buffer to be filled.

Thanks
Sushil Singh

Re: Getting error using dmesg

Michael Tully — Thu, 02 May 2002 00:53:26 GMT

From the man page of 'dmesg'

dmesg looks in a system buffer for recently printed diagnostic messages and prints them on the standard output. The messages are those printed by the system when unusual events occur (such as when system tables overflow or the system crashes). If the - argument is
specified, dmesg computes (incrementally) the new messages since the last time it was run and places these on the standard output. This is typically used with cron (see cron(1)) to produce the error log /var/adm/messages by running the command:

# /usr/sbin/dmesg - >> /var/adm/messages
every 10 minutes.

HTH
~Michael~

Re: Getting error using dmesg

A. Clay Stephenson — Thu, 02 May 2002 00:56:59 GMT

The dmesg buffer is intended to be filled; it just wraps around when the end of buffer is reached. There is no need to 'clear' it and it's rather pointless when machines should stay up for months on end. It's intentionally a fairly small buffer. You should really look in the syslogs for more data. Look in the /var/adm/syslog directory. The file syslog.log will probably give you more data. The best tool to find out what is filling /tmp is 'lsof' which is available for free download from any of the HP-UX Porting Centre's.

Re: Getting error using dmesg

Bill Hassell — Thu, 02 May 2002 01:02:30 GMT

There are two things to accomplish. First, log all dmesg lines to a file with a timestamp. An example for a cron job entry is found in /usr/newconfig/var/spool/cron/crontab.root which reads (in part):

# log kernel diagnostic messages every 10 minutes
05,15,25,35,45,55 * * * * /usr/sbin/dmesg - >>/var/adm/messages

I don't agree with the suggested name for this file (messages) so I would recommend changing it to:

# log kernel diagnostic messages every 10 minutes
05,15,25,35,45,55 * * * * /usr/sbin/dmesg - >>/var/adm/dmesg.log

Add this to your root cronjobs (you should many to trim logs, watch for disk space problems, run backups, etc). Now, when you want to see what dmesg is reporting, use the command:

tail -50 /var/adm/dmesg.log

dmesg is a program that reads an in-core buffer of console messages and as such, it cannot be cleared. Once the buffere is full, old messages are dropped off and new ones are added at the bottom.

As as the reason for /tmp becoming full, this will take some work. /tmp is (and must always be) open to every user on the system. Less experienced Unix users and sysadmins will store all sorts of junk files in /tmp and then forget to remove them.

The correct directory is /var/tmp but since scripts and programs misuse the /tmp directory, you'll have to run a cronjob to clean out old files automatically. The also applies to /var/tmp. A good rule of thumb is to remove any file and directory in temp directories that is more than a week old. And for real problems, very large files (more than 3 to 5 megs) should be removed after 3 days.

To see how your /tmp area is being used, use the command:

du -kx /tmp | sort -rn | more

The largest directory (below /tmp) will be shown. If this is not very big (3 digits or less) then /tmp may have one or more large files. To see the files sorted by size, use the command:

ls -l | sort -rnk5

The owner of the files and directories will be shown but the programs that caused the program will not be determined.

Re: Getting error using dmesg

Mark Fenton — Thu, 02 May 2002 01:21:03 GMT

Bill, as an addendum to your post -- a word of caution on removing old files from /var/tmp. Some (extremely poorly written) programs store CRITICALLY IMPORTANT "hidden" files in the /var/tmp directory (the license indicator for microfocus cobol and certain cobol database programs) for instance. Removing these files can cause a few hours of fun and frustration for users and unfortunate DBAs/SAs.

<>

Re: Getting error using dmesg

Bill Hassell — Thu, 02 May 2002 01:35:32 GMT

Agreed...there are a lot of applications that need major repair. The manufacturer should provide a list of critical files that must remain in temporary directories.