vxfs: NOTICE: msgcnt 25 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block extent)

[Aungshuman Paul]

Hello,

Can anyone help me regarding this issues:

I have an HP-UX B.11.31 system. Certainly I found the root lvm is full 100%.
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1044544 4024 100% /
and dmesg show the following error :
vxfs: NOTICE: msgcnt 25 mesg 001: V-2-1: vx_nospace - /dev/root file system full (1 block extent)
But when I calculate the mounted directory of the root lvm ( /, sbin and /etc), I did not find the 1 GB space. It is identical to other system.

I need help to identify the problem and free some space from this lvm.

I shall wait for your valuable feedback.

Aunsghu

[avizen9]

please post bdf -l output and also vgdisplay -v

thanks

[Sharma Sanjeev]

Hi Anugshuman

Please check for Large files in /

pwd
/
find / -xdev -size +10000000c -exec ls -lrt {}\;

Than zip or remove files which is not required
Regards
Sanjeev

[Steven E. Protter]

Shalom,

No files other than small configuration files belong on the root file system.

cd /
du -k | sort -rn | more

Find out what is stored there and does not belong.

HP-UX systems are not designed for use with a single file system like default Linux installs.

SEP

[Aungshuman Paul]

Please find the bdf -l and vgdisplay -v below :

# bdf -l
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1048576 1044544 4024 100% /
/dev/vg00/lvol1 1835008 169016 1653064 9% /stand
/dev/vg00/lvol8 8912896 641304 8209784 7% /var
/dev/vg00/lvol7 4653056 3044256 1596288 66% /usr
/dev/vgerpprdraid1/lvusrsaptrans
5242880 18366 4897989 0% /usr/sap/trans
/dev/vgerpprdraid1/lvusrsapPRD
8388608 19137 7846387 0% /usr/sap/PRD
/dev/vg00/lvtmp 7340032 26988 6855989 0% /tmp
/dev/vgerpprdraid1/lvsapmntPRD
5242880 18366 4897989 0% /sapmnt/PRD
/dev/vgerpprdraid1/lvoracle
1048576 17338 966792 2% /oracle
/dev/vgerpprdraid1/lvoraclestage102_64
7340032 18880 6863587 0% /oracle/stage/102_64
/dev/vgerpprdraid1/lvoraclePRD
1048576 17339 966789 2% /oracle/PRD
/dev/vgerpprdraid5/lvoraclePRDsapreorg
1048576 21104 1019456 2% /oracle/PRD/sapreorg
/dev/vgerpprdraid5/lvoraclePRDsapdata4
419430400 33920 416119952 0% /oracle/PRD/sapdata4
/dev/vgerpprdraid5/lvoraclePRDsapdata3
419430400 33920 416119952 0% /oracle/PRD/sapdata3
/dev/vgerpprdraid5/lvoraclePRDsapdata2
419430400 33920 416119952 0% /oracle/PRD/sapdata2
/dev/vgerpprdraid5/lvoraclePRDsapdata1
419430400 3938744 412245696 1% /oracle/PRD/sapdata1
/dev/vgerpprdraid5/lvoraclePRDsapcheck
1048576 21104 1019456 2% /oracle/PRD/sapcheck
/dev/vgerpprdraid5/lvoraclePRDsapbackup
1048576 21104 1019456 2% /oracle/PRD/sapbackup
/dev/vgerpprdraid1/lvoraclePRDoriglogB
5242880 18366 4897989 0% /oracle/PRD/origlogB
/dev/vgerpprdraid1/lvoraclePRDoriglogA
5242880 18366 4897989 0% /oracle/PRD/origlogA
/dev/vgerpprdraid1/lvoraclePRDoraarch
83886080 37641 78607919 0% /oracle/PRD/oraarch
/dev/vgerpprdraid1/lvoraclePRDmirrlogB
5242880 18366 4897989 0% /oracle/PRD/mirrlogB
/dev/vgerpprdraid1/lvoraclePRDmirrlogA
5242880 18366 4897989 0% /oracle/PRD/mirrlogA
/dev/vg00/lvol6 8880128 5322360 3530016 60% /opt
/dev/vg00/lvol5 131072 5792 124312 4% /home
/dev/vgerpprdraid5/lvdump
26214400 21872 25987912 0% /dump
DevFS 6 6 0 100% /dev/deviceFileSystem
#
# vgdisplay -v | more
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 10
Open LV 10
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 4353
VGDA 2
PE Size (Mbytes) 32
Total PE 4343
Alloc PE 2041
Free PE 2302
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 1.0
VG Max Size 2228736m
VG Max Extents 69648

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 1792
Current LE 56
Allocated PE 56
Used PV 1

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 8192
Current LE 256
Allocated PE 256
Used PV 1

LV Name /dev/vg00/lvol3
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 32
Allocated PE 32
input
Used PV 1

LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 512
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vg00/lvol5
LV Status available/syncd
LV Size (Mbytes) 128
Current LE 4
Allocated PE 4
Used PV 1

LV Name /dev/vg00/lvol6
LV Status available/syncd
LV Size (Mbytes) 8672
Current LE 271
Allocated PE 271
Used PV 1

LV Name /dev/vg00/lvol7
LV Status available/syncd
LV Size (Mbytes) 4544
Current LE 142
Allocated PE 142
Used PV 1

LV Name /dev/vg00/lvol8
LV Status available/syncd
LV Size (Mbytes) 8704
Current LE 272
Allocated PE 272
Used PV 1

LV Name /dev/vg00/lvswap
LV Status available/syncd
LV Size (Mbytes) 24576
Current LE 768
Allocated PE 768
Used PV 1
LV Name /dev/vg00/lvtmp
LV Status available/syncd
LV Size (Mbytes) 7168
Current LE 224
Allocated PE 224
Used PV 1


--- Physical volumes ---
PV Name /dev/disk/disk1_p2
PV Status available
Total PE 4343
Free PE 2302
Autoswitch On
Proactive Polling On


VG Name /dev/vgerpprdraid5
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 9
Open LV 9
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 29599
VGDA 2
PE Size (Mbytes) 64
Total PE 29599
Alloc PE 26064
Free PE 3535
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 1.0
VG Max Size 29599g
VG Max Extents 473584

--- Logical volumes ---
LV Name /dev/vgerpprdraid5/lvoracleclient
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapdata1
LV Status available/syncd
LV Size (Mbytes) 409600
Current LE 6400
Allocated PE 6400
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapdata2
LV Status available/syncd
LV Size (Mbytes) 409600
Current LE 6400
Allocated PE 6400
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapdata3
LV Status available/syncd
LV Size (Mbytes) 409600
Current LE 6400
Allocated PE 6400
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapdata4
LV Status available/syncd
LV Size (Mbytes) 409600
Current LE 6400
Allocated PE 6400
Used PV 1

LV Name /dev/vgerpprdraid5/lvdump
LV Status available/syncd
LV Size (Mbytes) 25600
Current LE 400
Allocated PE 400
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapcheck
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapbackup
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid5/lvoraclePRDsapreorg
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c16t0d2
PV Status available
Total PE 29599
Free PE 3535
Autoswitch On
Proactive Polling On


VG Name /dev/vgerpprdraid1
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 12
Open LV 12
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 2399
VGDA 2
PE Size (Mbytes) 64
Total PE 2399
Alloc PE 2160
Free PE 239
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 1.0
VG Max Size 2399g
VG Max Extents 38384

--- Logical volumes ---
LV Name /dev/vgerpprdraid1/lvusrsapPRD
LV Status available/syncd
LV Size (Mbytes) 8192
Current LE 128
Allocated PE 128
Used PV 1

LV Name /dev/vgerpprdraid1/lvusrsaptrans
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1

LV Name /dev/vgerpprdraid1/lvsapmntPRD
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRDoriglogA
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRDoriglogB
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1
LV Name /dev/vgerpprdraid1/lvoraclePRDmirrlogA
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRDmirrlogB
LV Status available/syncd
LV Size (Mbytes) 5120
Current LE 80
Allocated PE 80
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRDoraarch
LV Status available/syncd
LV Size (Mbytes) 81920
Current LE 1280
Allocated PE 1280
Used PV 1

LV Name /dev/vgerpprdraid1/lvoracle
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclestage102_64
LV Status available/syncd
LV Size (Mbytes) 7168
Current LE 112
Allocated PE 112
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRD
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRD102_64
LV Size (Mbytes) 81920
Current LE 1280
Allocated PE 1280
Used PV 1

LV Name /dev/vgerpprdraid1/lvoracle
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclestage102_64
LV Status available/syncd
LV Size (Mbytes) 7168
Current LE 112
Allocated PE 112
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRD
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 16
Allocated PE 16
Used PV 1

LV Name /dev/vgerpprdraid1/lvoraclePRD102_64
LV Status available/syncd
LV Size (Mbytes) 8192
Current LE 128
Allocated PE 128
Used PV 1


--- Physical volumes ---
PV Name /dev/dsk/c10t0d3
PV Status available
Total PE 2399
Free PE 239
Autoswitch On
Proactive Polling On

[RUET]

Two things:

- firt may be you still have a process or a zombie process that has an open file (may be a large one). Memory FS metadata are still counting these data. In this case, if you reboot the server you will cleanup this bad context.
- you can also have a mount directory (like /opt or /var) that is masking some files under the mount point. By exemple, if you unmount /var, and cd /var, may be you will find some data that are masked by thje mount ..

hope this help

regards

Patrick

[Aungshuman Paul]

Already I reboot the system several time but no result. And how to umount the var or opt? in single user mode?
For your further assistance I upload the syslog file.

[Robert-Jan Goossens_1]

Hi,

Please post the output of;

# du -xk / | sort -rn | more

Regards,
Robert-Jan

[RUET]

Yes single user is the best
because you have a light and clean process context

or for each mounted filesystem:
umount /mount-point
cd /mount_point
ls
rm -fr * if needed ..
cd /
mount /mount_point

If no user activity you can do a fuser -ku on the logical volume if the FS is busy

patrick

[V. Nyga]

Hi,

single user mode will NOT help, because the root directory ('/') is full!

As already said before - check for large files!
Check for example /dev - are there ordinary files?
There should be none!
Maybe someone wqanted to write to a tape and told the wrong device (typo error?).
Then a large file is created in /dev or /dev/rmt.

Check for these!
Also pay attention with rebooting!
If root-directory is full you maybe can't login with root any more!
Stay logged in and make the tests we told you.

Volkmar