topic Re: Strange syslog/dmesg message in Operating System - HP-UX

Strange syslog/dmesg message

joe_91 — Mon, 11 Jul 2005 12:18:56 GMT

I am having this message in syslog and as well as dmesg...
from syslog

vmunix:
full (1 block extent)vmunix: vxfs: mesg 001: vx_nospace - /dev/root file system

from dmesg:
vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)

the machine however does not mount the / filesystem in /dev/root. what does this mean? also the core file(from the appln) is directed to /dev/root. any thoughts please..

Many Thanks

Joe

Re: Strange syslog/dmesg message

Pete Randall — Mon, 11 Jul 2005 12:22:45 GMT

Joe,

Usually this results from the machine being brought up to multi-user mode directly from LVM maintenance mode. In LVM maintenance mode, / gets mounted as /dev/root and the correct way to return to multi-user mode is to reboot, rather than doing an init 3. You need to find and clear out what is filling up / and then reboot.

Pete

Re: Strange syslog/dmesg message

joe_91 — Mon, 11 Jul 2005 12:30:44 GMT

Please note that the machine does NOT mount as /dev/root .

Thanks

Joe.

Re: Strange syslog/dmesg message

Rick Garland — Mon, 11 Jul 2005 12:35:49 GMT

What is date stamp of the message?

Is it possible that this system was brought up from LVM maintenance sometime in the past?

Re: Strange syslog/dmesg message

TwoProc — Mon, 11 Jul 2005 12:44:07 GMT

Joe, this is just what "/" is always reported as when it gets full(/dev/root). I just did the following on a test server...

dd if=/dev/zero of=/zeroes bs=32k

It errored out with:

msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)
I/O error
739+0 records in
738+1 records out

and - my syslog has the following new entry in it now:
Jul 11 14:39:27 testserv vmunix: msgcnt 1 vxfs: mesg 001: vx_nospace - /dev/root file system full (1 block extent)

So, under the covers - /dev/root and / are the same thing. Nothing to worry about.

Re: Strange syslog/dmesg message

Devender Khatana — Mon, 11 Jul 2005 12:55:01 GMT

Hi,

If I remember correctly this use to be in the cases where you do not have LVM config on your root disk. In this event you do not have multiple file systems for /var , /usr , /home & all . All these are clubbed in a single file system & the device file for that should be /dev/root.

Output of bdf will clarify that.

Which OS version it is ?
Post the output of bdf.

HTH,
Devender

Re: Strange syslog/dmesg message

TwoProc — Mon, 11 Jul 2005 13:06:24 GMT

Looking in /stand/build/conf.c file ...

There is a line
dev_t rootmir;

and later on...

dev_t rootdev = makedev(-1,0xFFFFFF);

and - looking at /dev/root -
ls -al /dev/root
brw-r----- 1 bin sys 255 0xffffff Feb 26 2004 /dev/root

I can *guess* that /dev/root is created during the boot process as some sort of a mirrored memory pointer to "/" - or vice-versa, and the /dev/root block special file is left on the system as a default way to find the root file system. Could this be how the system gets to boot from an lvm device before it's even loaded? My guess is that it has something to do with it. Of course, that theory could be all wet. :-)

Re: Strange syslog/dmesg message

joe_91 — Mon, 11 Jul 2005 14:13:12 GMT

Jul 9 22:47:05 mn2300 vmunix:
full (1 block extent)vmunix: vxfs: mesg 001: vx_nospace - /dev/root file system

this is the timestamp. It was not after the system brought up from LVM maint mode. this happ after there was an installation of an application.

Thx

Joe

Re: Strange syslog/dmesg message

Devender Khatana — Mon, 11 Jul 2005 14:16:26 GMT

Joe,

Can you attach the bdf output. You will notice some file system 100% full there.

HTH,
Devender

Re: Strange syslog/dmesg message

TwoProc — Mon, 11 Jul 2005 14:17:42 GMT

Joe, run the test scenario (dd command) as I did above on a test server. It will convince that what you're seeing is normal for a full root file system, and prove to you that at least as far as this topic is concerned - /dev/root and / are the same thing, and the error you're seeing the syslog is normal for that condition.

Re: Strange syslog/dmesg message

Fabio Ettore — Mon, 11 Jul 2005 14:42:05 GMT

Hi Joe,

I saw that you gave Pete just one point but you will see he is right.
I had the same your problem (identic), infact in that instance no /dev/root was mounted on my system but continuously that message was logged in syslog.log.
As Pete said you need to find and clear out what is filling up /.

I would suggest you to begin by find command, for example:

find / -size +10000c
(to find what file is bigger than 10M),

find / -size +100000c
(to find what file is bigger than 100M)

and so on.

Hope that helps you.

Best regards,
Fabio

Re: Strange syslog/dmesg message

Alan Meyer_4 — Mon, 11 Jul 2005 14:46:53 GMT

Chances are it's a core file or a mis-directed output file of some sort

Re: Strange syslog/dmesg message

Bill Hassell — Mon, 11 Jul 2005 17:27:37 GMT

The /dev/root location is an internal method that LVM uses to get started and indeed, your bdf might show / is mounted from /dev/root. It is a survival device file in case there is something wrong LVM activation during bootup. The syslog message generator doesn't spend any time tracking down the true mountpoint.

So the message means: your / filesystem is (or was) full. To locate the source of the problem, do NOT look for big files! Many sysadmins have purged big files like vmunix or vg00.conf, only to regret the outcome. Instead, use du to analyze the directories and then look inside the biggest directories:

du -kx / | sort -rn | more

This will show you the largest directories starting with / and working down. To sort the files inside a specific directory, do this:

ll | sort -rnk5 | more

then look at the biggest file(s) to see if it is a mistake and can be removed.

Re: Strange syslog/dmesg message

joe_91 — Mon, 11 Jul 2005 18:07:27 GMT

thanks so much for all the replies. I apologize for giving pete just 1 point. It was indeed true that there was a core file created and the admin moved and removed that file which caused the /dev/root message. However we do have a code where tey are re-directing the application core files to a particular directory. so i am wondering how it went to the root filesystem.

Thanks

Joe

Re: Strange syslog/dmesg message

Bill Hassell — Mon, 11 Jul 2005 18:15:43 GMT

core files should never be generated in the / filesystem. It means that most likely, the application that created the core file is running as root (not good) and root's $HOME directory is / (not good). Always start your applications with a script and cd to an appropriate location like /var/tmp to collect your program 'droppings' (core file). And don't remove a core file until you find out what program created it:

file core

(the file command decodes the core file as to program name and error code)

Re: Strange syslog/dmesg message

joe_91 — Mon, 11 Jul 2005 18:23:03 GMT

Bill:

We still have the core file (just moved). it was from java. what i do not understand is we have already had this core issue before and are re-directing the core file to a tmp directory and also limiting it to a particular size. wonder how it landed in /.

Thanks

Joe.

Re: Strange syslog/dmesg message

Bill Hassell — Mon, 11 Jul 2005 18:47:14 GMT

The directory where a core file is stored is always the current value of $PWD (the working directory). So, whatever started this Java program either explicitly did a cd / or implicitly set PWD=/ because it runs as root and root's $HOME is /.

Re: Strange syslog/dmesg message

joe_91 — Mon, 11 Jul 2005 18:56:20 GMT

Thanks Bill and All who responded. What a great forum this is !!!!!

Best Regards

Joe