How To Umount /dev/root?

[dedham_ma_man]

Hello,

I am not a network-admin but rather a SW engineer who is trying to install 'net-snmp' onto our HPUX 11.23 PA (64bit) box.

In doing so, I am trying to increase the size of the /dev/root logical volume, because when I try to install something (net-snmp) as root, I keep getting the following error message:

vx -nospace /dev/root file system full.

And sure enough, when I check the amount of memory blocks available there, I get the following:

# df /dev/root
(/dev/root ): 0 blocks 2139030464 i-node
s

So, in trying to increase the size of the logical volume, the first step is to unmount /dev/root I believe.

Of course, I guess I have to be a superuser to do this, so I'm logged in as 'root'. Anyhoooo, from the mount point '/', when I issue the following command as 'root':

# umount /dev/root

I get the following:

mount: file system table may be corrupt
umount: makereversetablelist failed.

Am I doing this correctly? I've googled around and see other's who have gotten the 'vx -nospace /dev/root system full message, but don't see the solution. Does more hardware need to be added?

Any ideas or suggestions would help greatly. Thanks for your response.

dedham_ma_man

[F Verschuren]

/dev/root = / = /dev/vg00/lvol3

can you do a
bdf /

to see how big it is and an
uname -a
to see if this is normal

to extend the root filesystem a ignite restore is the best way to folow.
but normaly you can cleanup some wrongly pleased files so / is more free.

[Shah Sahib]

You need to see why root is full in the first place. If you built the system with only the minimal things on root (/tmp,/stand /var, /opt, /usr on separate file systems), then it is possible that you have a file or files that are eating up space. Look for files named "core"
and remove them.

Look in /dev for files that don't belong (A common mistake is to backup to tape and specify /dev/rmt/om instead of /dev/rmt/0m and end up with a file named the former which is quite large). If /stand is not a separate file system, you will have files
called vmunix*. vmunix is your kernel. vmunix.prev is the previous kernel in case you have problems with your current one and need to boot an alternate kernel. You may have others which you should not need. They
can be quite large. Look for junk in root's home directory (which is / by default. Move them somewhere else.

[Tim Nelson]

There may be two items here.

1) net-snmp typically does not install anything in the / filesystem. If you are compiling yourself then pick a different install destination. net-snmp was included in the iexpress package pre-compiled and homed to /opt/iexpress/net-snmp. Where did you get net-snmp and how is it being installed ?

2) root is probably already full even prior to your install. Fix this first before moving on.

As mentioned, in order to increase the root filesystem you will need to take an ignite image and reinstall the OS using the interactive mode of ignite to increase the filesystem sizes.

The other option is to find the files in root that are causing it to be full ( once again this is a static filesystem of about 400-600MB depending on your version of HPUX ) Look in /dev for any non device type files is a good start.

Only /dev and /etc and /sbin are part of / all other dirs should be mounted filesystems like /opt, /usr, /var, home etc...

[dedham_ma_man]

Thanks again all.

=======================================

F Verschuren:

In response to your request:

# bdf /
bdf: file system table may be corrupt

#uname -a
HP-UX lclb126 B.11.23 U 9000/800 197424686 unlimited-user license

==========================================

Shaw Sahib:

The 'net-snmp' tarball, which is approx 27.5 MG, is located in a temp directory (/Randy) from where I untar it. Upon untarring, there are many files and so I'm not sure how large the whole collection of these files are, but it's a lot of space taken as well as the *.tar file.

============================================

Tim Nelson:

There may be two items here.

< net-snmp typically does not install anything in the / filesystem.>

I haven't gotten to the point of 'make install'. The problem occurs after I ./configure, then as I doing 'make'.

The general procedure I use for installing 'net-snmp', is the same one I used on Solaris. As 'root', I create a temp folder (/Randy), then copy the net-snmp tarball there, then untarred it. From /Randy/net-snmp-5.4.1, I run ./configure, which works ok, then run 'make' to build the 'net-snmp' libraries for HPUX specific architecture and that's when I eventually start getting the vx -nospace .... message.


< root is probably already full even prior to your install. Fix this first before moving on. >

No doubt that's true. 'root' contains the net-snmp' tarball which is 27.5 MG. Then, after untarring, with all of the files that are extracted, the memory usage grows in order of magnitudes.

NOTE: This is the same procedure I used on Solaris to install 'net-snmp' and I didn't have this problem.



Sadly, I'm afraid I'll have to resort to this. But it's a pain around here because the machine is remote and I don't know who has the CD's for this machine or how I would gain the admin priveledges to do this - I'm just a SW Eng/developer not an admin.



True, and that apparently applies to the system I using.

[Eric SAUBIGNAC]

Hi,

as Shah and Tim said, first and very important : do some cleaning in /

I am a beat surprised by the message you get when you tried to umount /dev/root. Normally you should get this error message :

umount: cannot find /dev/root in /etc/mnttab
cannot unmount /dev/root

The reason is that you normaly have /dev/vg00/lvol3 mounted under / You will find /dev/root mounted as / only when you boot in a special mode. Has this system been booted in maintenance mode then brought directly in a multi user environment ? In this case it might be necessary to remove /etc/mnttab then issue a "mount -a" command. BUT DON'T DO IT BY YOURSELF, let your system admin doing that !!!

And other question regarding the message :

mount: file system table may be corrupt
umount: makereversetablelist failed.

This could be a very very bad news. I strongly suggest you make a copy of your system with Ignite BEFORE halting the system. At least you should copy / filesystem to a safe location in case you need to reinstall OS.

Eric

[Eric SAUBIGNAC]

"No doubt that's true. 'root' contains the net-snmp' tarball which is 27.5 MG."

You MUST remove the tarball package or move it to /var/tmp for example.

But i am not sure that if your root filesystem is corrupted it will be enough to get it clean

Eric

[Robert-Jan Goossens]

Hi,

Lets first clear the error "file system table may be corrupt"

# mv /etc/mnttab /etc/mnttab.old
# mount -v

What does bdf report about /root filesystem?

# bdf /

Regards,
Robert-Jan

[Eric SAUBIGNAC]

Ho, and the last one before WE : though i think like Shah an Tim that it should not be necessary to increase /, you will find at the following URL some good readings :

root extend http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1180526

Eric

[dedham_ma_man]

Robert-Jan:

Here's what happens when I do as you suggested:

# mv /etc/mnttab /etc/mnttab.old
# mount -v
mount: file system table may be corrupt
# bdf /
bdf: file system table may be corrupt
#

Eric:

This one's interesting. Ok, as 'root', I cleared all of my stuff from '/' (the /Randy folder with the net-snmp tar as you suggested). Then, as 'root' I go to /usr and created a folder called /usr/Randy - can't use /var/tmp because that folder gets cleared every nt. Then, I copy the net-snmp tar file there in /var/Randy. Then, I untar it. Then, I go into the /var/Randy/net-snmp-5.4.1 folder and run './configure' (with the relevant options). Then, as the net-snmp ./configure script is running, here's what happens:

using default "enterprise.net-snmp"
using default enterprise sysOID "NET-SNMP-MIB::netSnmpAgentOIDs..."
using default notifications "NET-SNMP-MIB::netSnmpNotifications"
using OS default send buffer size for server sockets
using OS default recv buffer size for server sockets
using OS default send buffer size for client sockets
using OS default recv buffer size for client sockets
checking if I need to feed myself to ksh... no
checking for gcc... yes

(then, it hangs for awhile ... several minutes which is bad).

(then, I log in as root from another terminal window and 'kill' the "configure" process.

Then, I start getting these messages inside of the original 'root' session where the hang occurred:


msgcnt 27400 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27404 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27408 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27412 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27416 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27420 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27424 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27428 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27432 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27436 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27440 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27444 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27448 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27452 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27456 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27460 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27464 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27468 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27472 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27476 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27480 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27484 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27488 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27492 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27496 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27500 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27504 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27508 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27512 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27516 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27520 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27524 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27528 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27532 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27536 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27540 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27544 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27548 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27552 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27556 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27560 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27564 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

msgcnt 27568 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 bl
ock extent)

[Robert-Jan Goossens]

/usr is also in the root filesystem.

Try using /opt

Could you post
# ll /etc/mnttab

[dedham_ma_man]

Robert-Jan:

I'll try using /opt instead.

Also, as requested:

# ll /etc/mnttab
-rw-r--r-- 1 root root 0 Nov 30 14:33 /etc/mnttab

[Robert-Jan Goossens]

-rw-r--r-- 1 root root 0 Nov 30 14:33 /etc/mnttab

the mnttab is 0 bytes!! this is giving you the "file system table may be corrupt" error.

create a bit of space in / move /etc/mnttab again and use "mount -a" to create e good mnttab.

[Robert-Jan Goossens]

Hi,

Please also provide the output of

# du -kx / | sort -rn | head -30

Regards,
Robert-Jan

[dedham_ma_man]

Robert-Jan:

When move the mntabb file to mntabb.old, then
do 'mount -a', I get the 'file system table may be corrupt' message. Am I missing a step here?

Also, as requested:

Is this actual space for each file system in the tree?
Thanks again for your input.

# du -kx / | sort -rn | head -30
156232 /
92168 /etc
63128 /sbin
56912 /etc/vx
33704 /etc/vx/static.d
32944 /etc/vx/static.d/build
24024 /etc/opt
22416 /etc/vx/type
17304 /sbin/fs
13368 /sbin/fs/vxfs
13224 /etc/opt/resmon
10408 /etc/vx/type/static
7544 /etc/opt/resmon/lbin
7080 /etc/vx/type/gen
4560 /etc/opt/wbem
4544 /etc/opt/wbem/mof
4232 /etc/opt/resmon/log
3792 /etc/opt/samba
3760 /etc/opt/samba/codepages
2760 /sbin/fs/hfs
2664 /etc/vx/type/fsgen
2528 /etc/emc
2512 /etc/emc/lib
2256 /etc/vx/type/raid5
1840 /etc/hparray
1656 /etc/lvmconf
1592 /etc/opt/cifsclient
1544 /etc/opt/cifsclient/unitables
1176 /sbin/init.d
1032 /etc/opt/resmon/lib
#

[Robert-Jan Goossens]

Hi,

No you are not missing a step.

The du -kx / will report a the total size of root filesystem and the size of every directory in the root filesytem

156232 / ---> 150 mb is normal for /root filesystem, so what is keeping your /root full???

Did someone remove a large (open) logfile today in the root filesystem?

Robert-Jan

[Dennis Handly]

You might be able to move this elsewhere, and add a symlink, until you figure out where the space is going:
32944 /etc/vx/static.d/build

[dedham_ma_man]

Robert-Jan:

Here's a snapshot of the root ( / ):
Not sure if this helps; I'll poke around underneath and see if I see anything else that's clogging it up:

# ls -l
total 1730
-rw------- 1 root sys 900 Aug 31 2005 .adb_hist
-rw------- 1 root sys 7573 Sep 12 2005 .bash_history
-rw-rw-rw- 1 root sys 79 Apr 26 2005 .bashrc
-rw------- 1 root sys 0 Jun 29 12:27 .history
-r--r--r-- 1 bin bin 1007 Sep 14 2004 .profile
-rw------- 1 root sys 4750 Aug 18 2006 .q4_hist
-rw------- 1 root sys 1368 Dec 12 2005 .q4history
-r-xr-xr-x 1 root sys 5310 Jul 10 2006 .q4rc.pl
-rw-rw-rw- 1 root sys 0 Nov 30 01:03 .rhosts
-rw------- 1 root sys 0 Nov 29 22:40 .sh_history
drwx------ 2 root root 96 Dec 12 2005 .ssh
drwxr-xr-x 6 root sys 8192 Apr 12 2005 .sw
drwxr-xr-x 2 root sys 96 Apr 12 2005 SD_CDROM
lr-xr-xr-t 1 root sys 8 Aug 6 2004 bin -> /usr/bin
drwxrwxrwx 2 root sys 96 Apr 1 2005 cdrom
lrwxrwxrwx 1 root sys 18 Feb 15 2006 clearcase -> /disks/pp
tools-cam
---------- 1 root sys 0 Nov 29 17:07 con2
dr-xr-xr-x 13 bin bin 8192 Nov 29 09:15 dev
dr-xr-xr-x 1 root root 1 Nov 30 14:33 disks
-rw-r--r-- 1 root sys 148 Jan 10 2007 drv_list.out
dr-xr-xr-x 1 root root 1 Nov 30 15:11 emc
dr-xr-xr-x 35 bin bin 8192 Nov 30 15:02 etc
drwxr-xr-x 2 root root 8192 Aug 31 11:00 home
dr-xr-xr-x 1 root root 1 Nov 29 09:15 hwapps
-rwxr-xr-x 1 root sys 500520 Sep 13 2004 inq.HP
lr-xr-xr-t 1 root sys 8 Aug 6 2004 lib -> /usr/lib
drwxr-xr-x 2 root root 96 Aug 6 2004 lost+found
drwxrwxrwx 3 root sys 96 Nov 1 2004 mnt
dr-xr-xr-x 2 root root 2 Nov 29 15:41 net
drwxrwxrwx 14 mukuns tty 512 Nov 30 05:21 odb_tmp
dr-xr-xr-x 68 bin bin 8192 Nov 30 15:00 opt
drwxr-xr-x 2 root sys 96 Jan 9 2007 panicfs
lrwxrwxrwx 1 root sys 14 Nov 29 06:35 plutotest -> /var/plut
otest
drwxrwxrwx 2 root sys 8192 Aug 10 09:38 plutotest_corefiles
-rw-rw-rw- 1 root sys 6833 Nov 28 07:10 pptest_shutdown.log
dr-xr-xr-x 12 bin bin 8192 Nov 29 09:00 sbin
dr-xr-xr-x 1 root root 1 Nov 29 09:15 snfs
dr-xr-xr-x 9 bin bin 1024 Nov 29 09:13 stand
dr-xr-xr-x 1 root root 1 Nov 29 09:15 swapps
drwxrwxrwx 11 root root 16384 Nov 30 15:02 tmp
dr-xr-xr-x 28 bin bin 8192 Sep 28 15:12 usr
dr-xr-xr-x 28 bin bin 8192 Nov 30 14:45 var
drwxr-xr-x 2 root root 196608 Nov 29 09:18 view
drwxr-xr-x 25 root sys 8192 Nov 16 01:44 vobs
#

[Robert-Jan Goossens]

maybe df -k will report the filesystems.

try and post
# df -k

[dedham_ma_man]

Nothing seems to happen:

# df -k
# pwd
/

[Robert-Jan Goossens]

Just to check, could you post the output of

# lvdisplay /dev/vg00/lvol3

[dedham_ma_man]

# pwd
/
# lvdisplay /dev/vg00/lvol3
--- Logical volumes ---
LV Name /dev/vg00/lvol3
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 0
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 504
Current LE 63
Allocated PE 63
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict/contiguous
IO Timeout (Seconds) default

[Robert-Jan Goossens]

Hi,

Your root filesystem should be something like 450 mb, du -kx reported that there is currently 150 mb being used. So something is using 300 mb in your root filesystem.

You might have to schedule a reboot to free up the space in /

Robert-Jan

[F Verschuren]

ther is properly a open file that is removed so you do not see it but lvm is alsow not removing the free space. propery it is still writting data to it so if you can not reboot the folowing staps has to be done quickely:

# mv /etc/lvmconf/*.old /tmp
# mv /etc/mnttab /etc/mnttab.old
# mount -v

What does bdf report about /root filesystem?

# bdf /

to detect what file is removed and still showing i think you can see it whit but I do not know for sure
ff /dev/vg00/lvol3 |awk '{ print $1 }' |while read line
do
ls -d $line
echo $line
done |grep -i not

ignore the /tmp files but look if ther is a file not find. If this does not work you have to have lsof installed if you run lsof you see all open files and ls -d shows you if this file is still thare.

ps /view is that a seperate vg?

[Dennis Handly]

>/view is that a separate vg?

It is probably separate MVFS (clearcase).