vxfs: NOTICE: msgcnt 25 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block extent)

[Aungshuman Paul]

Hello,

Can anyone help me regarding this issues:

I have an HP-UX B.11.31 system. Certainly I found the root lvm is full 100%.
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1044544 4024 100% /
and dmesg show the following error :
vxfs: NOTICE: msgcnt 25 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block extent)
But when I calculate the mounted directory of the root lvm ( /, sbin and /etc), I did not find the 1 GB space. It is identical to other system.

I need help to identify the problem and free some space from this lvm.

I shall wait for your valuable feedback.

Aunsghu

[avizen9]

please post bdf -l output and also vgdisplay -v

thanks

[Sharma Sanjeev]

Hi Anugshuman

Please check for Large files in /

pwd
/
find / -xdev -size +10000000c -exec ls -lrt {}\;

Than zip or remove files which is not required
Regards
Sanjeev

[Steven E. Protter]

Shalom,

No files other than small configuration files belong on the root file system.

cd /
du -k | sort -rn | more

Find out what is stored there and does not belong.

HP-UX systems are not designed for use with a single file system like default Linux installs.

SEP

[Aungshuman Paul]

Please find the bdf -l and vgdisplay -v below :

# bdf -l
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1044544 4024 100% /
/dev/vg00/lvol1 1835008 169016 1653064 9% /stand
/dev/vg00/lvol8 8912896 641304 8209784 7% /var
/dev/vg00/lvol7 4653056 3044256 1596288 66% /usr
/dev/vgerpprdraid1/lvusrsaptrans
5242880 18366 4897989 0% /usr/sap/trans
/dev/vgerpprdraid1/lvusrsapPRD
8388608 19137 7846387 0% /usr/sap/PRD
/dev/vg00/lvtmp 7340032 26988 6855989 0% /tmp
/dev/vgerpprdraid1/lvsapmntPRD
5242880 18366 4897989 0% /sapmnt/PRD
/dev/vgerpprdraid1/lvoracle
1048576 17338 966792 2% /oracle
/dev/vgerpprdraid1/lvoraclestage102_64
7340032 18880 6863587 0% /oracle/stage/102_64
/dev/vgerpprdraid1/lvoraclePRD
1048576 17339 966789 2% /oracle/PRD
/dev/vgerpprdraid5/lvoraclePRDsapreorg
1048576 21104 1019456 2% /oracle/PRD/sapreorg
/dev/vgerpprdraid5/lvoraclePRDsapdata4
419430400 33920 416119952 0% /oracle/PRD/sapdata4
/dev/vgerpprdraid5/lvoraclePRDsapdata3
419430400 33920 416119952 0% /oracle/PRD/sapdata3
/dev/vgerpprdraid5/lvoraclePRDsapdata2
419430400 33920 416119952 0% /oracle/PRD/sapdata2
/dev/vgerpprdraid5/lvoraclePRDsapdata1
419430400 3938744 412245696 1% /oracle/PRD/sapdata1
/dev/vgerpprdraid5/lvoraclePRDsapcheck
1048576 21104 1019456 2% /oracle/PRD/sapcheck
/dev/vgerpprdraid5/lvoraclePRDsapbackup
1048576 21104 1019456 2% /oracle/PRD/sapbackup
/dev/vgerpprdraid1/lvoraclePRDoriglogB
5242880 18366 4897989 0% /oracle/PRD/origlogB
/dev/vgerpprdraid1/lvoraclePRDoriglogA
5242880 18366 4897989 0% /oracle/PRD/origlogA
/dev/vgerpprdraid1/lvoraclePRDoraarch
83886080 37641 78607919 0% /oracle/PRD/oraarch
/dev/vgerpprdraid1/lvoraclePRDmirrlogB
5242880 18366 4897989 0% /oracle/PRD/mirrlogB
/dev/vgerpprdraid1/lvoraclePRDmirrlogA
5242880 18366 4897989 0% /oracle/PRD/mirrlogA
/dev/vg00/lvol6 8880128 5322360 3530016 60% /opt
/dev/vg00/lvol5 131072 5792 124312 4% /home
/dev/vgerpprdraid5/lvdump
26214400 21872 25987912 0% /dump
DevFS 6 6 0 100% /dev/deviceFileSystem
#
# vgdisplay -v | more
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 10
Open LV 10
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 4353
VGDA 2
PE Size (Mbytes) 32
Total PE 4343
Alloc PE 2041
Free PE 2302
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 1.0
VG Max Size 2228736m
VG Max Extents 69648

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 1792
Current LE 56
Allocated PE 56
Used PV 1

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 8192
Current LE 256
Allocated PE 256
Used PV 1

LV Name /dev/vg00/lvol3
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 32
Allocated PE 32
input
Used PV 1

LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 512
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vg00/lvol5
LV Status available/syncd
LV Size (Mbytes) 128
Current LE 4
Allocated PE 4
Used PV 1

LV Name /dev/vg00/lvol6
LV Status available/syncd
LV Size (Mbytes) 8672
Current LE 271
Allocated PE 271
Used PV 1

LV Name /dev/vg00/lvol7
LV Status available/syncd
LV Size (Mbytes) 4544
Current LE 142
Allocated PE 142
Used PV 1

LV Name /dev/vg00/lvol8
LV Status available/syncd
LV Size (Mbytes) 8704
Current LE 272
Allocated PE 272
Used PV 1

LV Name /dev/vg00/lvswap
LV Status available/syncd
LV Size (Mbytes) 24576
Current LE 768
Allocated PE 768
Used PV 1
LV Name /dev/vg00/lvtmp
LV Status available/syncd
LV Size (Mbytes) 7168
Current LE 224
Allocated PE 224
Used PV 1


--- Physical volumes ---
PV Name /dev/disk/disk1_p2
PV Status available
Total PE 4343
Free PE 2302
Autoswitch On
Proactive Polling On


VG Name /dev/vgerpprdraid5
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 9
Open LV 9
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 29599
VGDA 2
PE Size (Mbytes) 64
Total PE 29599
Alloc PE 26064
Free PE 3535
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 1.0
VG Max Size 29599g
VG Max Extents 473584

--- Logical volumes ---
LV Name /dev/vgerpprdraid5/lvoracleclient
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapdata1
LV Status available/syncd
LV Size (Mbytes) 409600
Current LE 6400
Allocated PE 6400
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapdata2
LV Status available/syncd
LV Size (Mbytes) 409600
Current LE 6400
Allocated PE 6400
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapdata3
LV Status available/syncd
LV Size (Mbytes) 409600
Current LE 6400
Allocated PE 6400
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapdata4
LV Status available/syncd
LV Size (Mbytes) 409600
Current LE 6400
Allocated PE 6400
Used PV 1

LV Name /dev/vgerpprdraid5/lvdump
LV Status available/syncd
LV Size (Mbytes) 25600
Current LE 400
Allocated PE 400
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapcheck
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapbackup
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapreorg
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c16t0d2
PV Status available
Total PE 29599
Free PE 3535
Autoswitch On
Proactive Polling On


VG Name /dev/vgerpprdraid1
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 12
Open LV 12
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 2399
VGDA 2
PE Size (Mbytes) 64
Total PE 2399
Alloc PE 2160
Free PE 239
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 1.0
VG Max Size 2399g
VG Max Extents 38384

--- Logical volumes ---
LV Name /dev/vgerpprdraid1/lvusrsapPRD
LV Status available/syncd
LV Size (Mbytes) 8192
Current LE 128
Allocated PE 128
Used PV 1

LV Name /dev/vgerpprdraid1/lvusrsaptrans
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1

LV Name /dev/vgerpprdraid1/lvsapmntPRD
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRDoriglogA
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRDoriglogB
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1
LV Name /dev/vgerpprdraid1/lvoraclePRDmirrlogA
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRDmirrlogB
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRDoraarch
LV Status available/syncd
LV Size (Mbytes) 81920
Current LE 1280
Allocated PE 1280
Used PV 1

LV Name /dev/vgerpprdraid1/lvoracle
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclestage102_64
LV Status available/syncd
LV Size (Mbytes) 7168
Current LE 112
Allocated PE 112
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRD
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRD102_64
LV Size (Mbytes) 81920
Current LE 1280
Allocated PE 1280
Used PV 1

LV Name /dev/vgerpprdraid1/lvoracle
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclestage102_64
LV Status available/syncd
LV Size (Mbytes) 7168
Current LE 112
Allocated PE 112
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRD
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRD102_64
LV Status available/syncd
LV Size (Mbytes) 8192
Current LE 128
Allocated PE 128
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c10t0d3
PV Status available
Total PE 2399
Free PE 239
Autoswitch On
Proactive Polling On

[RUET]

Two things:

- firt may be you still have a process or a zombie process that has an open file (may be a large one). Memory FS metadata are still counting these data. In this case, if you reboot the server you will cleanup this bad context.
- you can also have a mount directory (like /opt or /var) that is masking some files under the mount point. By exemple, if you unmount /var, and cd /var, may be you will find some data that are masked by thje mount ..

hope this help

regards

Patrick

[Aungshuman Paul]

Already I reboot the system several time but no result. And how to umount the var or opt? in single user mode?
For your further assistance I upload the syslog file.

[Robert-Jan Goossens_1]

Hi,

Please post the output of;

# du -xk / | sort -rn | more

Regards,
Robert-Jan

[RUET]

Yes single user is the best
because you have a light and clean process context

or for each mounted filesystem:
umount /mount-point
cd /mount_point
ls
rm -fr * if needed ..
cd /
mount /mount_point

If no user activity you can do a fuser -ku on the logical volume if the FS is busy

patrick

[V. Nyga]

Hi,

single user mode will NOT help, because the root directory ('/') is full!

As already said before - check for large files!
Check for example /dev - are there ordinary files?
There should be none!
Maybe someone wqanted to write to a tape and told the wrong device (typo error?).
Then a large file is created in /dev or /dev/rmt.

Check for these!
Also pay attention with rebooting!
If root-directory is full you maybe can't login with root any more!
Stay logged in and make the tests we told you.

Volkmar

[Sharma Sanjeev]

Hi

As suggested before u need to check for large files under root,than mv or delete unwanted files.

please post find results

Regards
Sanjeev

[V. Nyga]

Hi again,

to speed up the 'du' command you can try to umount as many directories as possible (oracle directories ...).
Just try a 'umount -a'.

You also can do a 'init 1' to go to the first stage of boot up.
Then you can umount more disks.

>But when I calculate the mounted directory of the root lvm ( /, sbin and /etc), I did not find the 1 GB space.

What's about /dev, /home, /bin ...?

V.

[RUET]

Volkar
I 'm quite sure you are wrong or didn't understand what I wanted to show.

I know that / is full
but may be it's masked data or directories by mount point.

If you create a 2GB files under /opt before mounting a /opt Filesystem.
If then you mount /opt, it will mask the 2GB files. You are no more able to find it, but it's still there and consuming data space in the / file system ...
and you are no more able to find it .. except unmoounting /opt

[Ganesan R]

Hi Aungshuman,

There are few possiblities as people are explained earlier. Common one is open files. If someone have killed one process which has open files on root filesystem, then that space will not be released until you kill the process or reboot the server. You already rebooted the server so it is out.

Now,
>>>But when I calculate the mounted directory of the root lvm ( /, sbin and /etc), I did not find the 1 GB space. It is identical to other system.<<<

Did you calculated each and every subdirectory under root. You may missed out some directory or subdirectory. The best way is use this command to find which directory consumes how much space. So that you can dig into the big directory and delete the unwanted files.

#du -xk / |sort -nr

You can also list the first 20 big directories under root by this command

#du -xk / |sort -nr |head -20

[V. Nyga]

@PR - yes, you're right
Files under mounted directories are possible.
But it makes more sense to search for strange files first.

Btw. - there's no /dev/vg00/lvol4 mounted, also it's defined:
LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 512
Current LE 16
Allocated PE 16
Used PV 1

Normally this is /tmp - there's a new logical volume defined:
/dev/vg00/lvtmp 7340032 26988 6855989 0% /tmp

This could be a good candidate for hidden files ....

Please tell us your results Aunsghu ...

V.

[Aungshuman Paul]

Hi,

One of my colleague solve the problem. As he informed , there was a crash dump file in the /var/adm/crash. Even though I did not check yet. But I have some confussion, carsh folder under var lvm.I will inform you after know the problem and its resolution.

Aungshu

[Sharma Sanjeev]

Hi Aunsghu

Yes, if some crash file is their that can take lot of space.

As per ITRC rules if you are happy with replies please assign points

Regards
Sanjeev

[Cleiton L. Siqueira]

Hi,

First you can verify /etc/lvmconf/*.old files.
Sometimes, this files are large.

Then you can use the lsof app, to verify if any process in memory is using a file that was deleted before.

[Bill Hassell · Accepted solution]

> /dev/vg00/lvol3 1048576 1044544 4024 100% /

Hint: use bdf / to show a single location such as a mountpoint.

/dev/vg00/lvol8 8912896 641304 8209784 7% /var

There appears to be a lot of room in /var. What do the commands:

bdf /var/adm/crash
ll -d /var/adm/crash

show? If it shows "/" at the end, then this (potentially) very large directory is really on /, a very bad location. Does the ll -d command show a symbolic link to /?

> /dev/vg00/lvol3 1048576
This is very large for "/". Apparently, someone has stored non-OS related files in /, another very bad idea. Note also that looking for big files is the wrong approach -- look for big directories. Use this command to find the biggest directories:

du -kx / | sort -rn | head

The two largest directories in / must be /sbin and /etc, each about 30-75 MB. Anything else at the top of the list is in the wrong directory. You can move mistakes like this without affecting your programs by creating another lvol or finding space in a more appropriate directory such as /opt, then move all the files to this new location. Then replace the original directory name with a symbolic link to the new location.

OS files will not fill up 1GB of / space so your problem is related to a program that is storing data in the wrong location.

[Aungshuman Paul]

Thanks,

Now problem is solved. And it was due to /var/adm/crash file which was belong to root lvm.

Thanks again for everyone's help

[Aungshuman Paul]

Problem solved.