/root file system full

[Kgreen]

Hi , I have two systems of the same config. But one has root file system showing just 15 % used and the other as 95 % used. Iam not able to make but whats wrong with the lsof or the list of files on the / file system.Here are the outputs attached

[Ivan Krastev]

Hi,

Try to search for some large files:

find / -size 10000k -type f

maybe some program produced core dump or big log files.

regards,
ivan

[Pete Randall]

Run "du -sk / |sort -n" to reveal where most of the space is being taken up. Typical causes are: core files in root's home directory, mistyped device files causing backups to be written to /dev (run "find /dev -type f" - anything returned is a problem). It is also possible for a file system to be unmounted, then have a job write to the unmounted file system and the output ends up in /. See if there are any file systems that you can unmount and look for files under the mount point.


Pete

[Michal Toth]

output of your find does not show any suspicious files

- doublecheck that your root fs are of equal total size (i'm not implying that you are stupid, but mistakes do happen :-) )
- use `du` on both systems and compare (your find was limited only to individual bigger files, du can reveal small files, but coming in massive numbers /aka infestation/)
- if no major differences then propably there is a file that was deleted, but it is still being accessed by an active process - use lsof to identify it (lsof )

[Kgreen]

That does not help


ecc6ta# find / -size 10000k -type f
find: Error in processing the argument 10000k
ecc6ta# find / -size 10000k -type f
find: Error in processing the argument 10000k
ecc6ta# find / -size 10000 -type f
/var/opt/OV/datafiles/coda00019
/var/opt/OV/datafiles/coda00020
/var/opt/OV/datafiles/coda00021
/var/opt/OV/datafiles/coda00022
/var/opt/OV/datafiles/coda00023
/var/opt/OV/datafiles/coda00024
find: cannot search /sapbasis/Trexsp9/SAPINST/UNIX/HP11_64
ecc6ta#
ecc6ta#
ecc6ta# bdf /var/opt/OV/datafiles/coda00021
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol7 8388608 3240464 5110232 39% /var
ecc6ta# find /dev -type f
ecc6ta#

[Kgreen]

i check all the mount points are mounted. All the du outputs are also same and lsof does not report any lost files. but still the / file system is 95 %.

ecc6ta# bdf /
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 2097152 1984392 111928 95% /

[Michal Toth]

Kgreen,

please attach output of du, bdf and lsof from both systems. Either you've missed something important or this is the biggest black-magic bug ever found within hpux.

[rmueller58]

sounds like a problem I've had when an NFS or CIFS mount fails where I am writing DB archive files and it writes to what ever directory I created under /root as the mount point.

[OldSchool]

try

find / -size +100000c -xdev

[john D_3]

Was it happened after creating /root filesystem?

May be before mounting different LV on /root, there were files under /root which exist on vg00/lvol3.

May be you can try to umount /root and see if there are files underneath it.

[Kgreen]

ecc6ta# bdf /oracle/ECT/920_64/bin/oracle
Filesystem kbytes used avail %used Mounted on
/dev/vg02ect/lv10 6291456 3841322 2297045 63% /oracle/ECT/920_64
ecc6ta# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 2097152 1984392 111928 95% /
/dev/vg00/lvol7 8388608 3243256 5107464 39% /var
/dev/vg00/lvol6 16777216 4054240 12623608 24% /usr
/dev/vg00/lvol5 2097152 1898200 197440 91% /tmp
/dev/vg00/lvol4 8388608 5503128 2863264 66% /opt
/dev/vg06ect/lv20 524288 16684 475937 3% /oracle
ecc4q.am.elcompanies.net:/sapbasis
136314880 112437968 22386688 83% /sapbasis
/dev/vg01/lv3 16625664 61236 15529179 0% /root
/dev/vg00/lvol8 8388608 207392 8117368 2% /home
/dev/vg01/lv2 16625664 20548 15567303 0% /var/adm/crash
/dev/vg06ect/lv23 524288 52982 441868 11% /usr/sap
ecc3d.am.elcompanies.net:/usr/sap/trans
25165824 17673256 7024392 72% /usr/sap/trans
/dev/vg00/lvol1 512499 111485 349764 24% /stand
/dev/vg06ect/lv8 524288 16617 475947 3% /oracle/client
/dev/vg06ect/lv24 4194304 17565 3915764 0% /sapwork
/dev/vg06ect/lv7 4194304 523903 3441008 13% /oracle/stage
/dev/vg06ect/lv6 2097152 1635462 435783 79% /oracle/stage/920_64
/dev/vg02ect/lv1 2097152 1719707 354040 83% /sapmnt/ECT
/dev/vg02ect/lv4 2097152 16982 1950163 1% /usr/sap/ECT
/dev/vg02ect/lv13 2097152 45070 1923889 2% /usr/sap/ECT/SCS01
/dev/vg02ect/lv11 2097152 19020 1948277 1% /oracle/ECT
/dev/vg02ect/lv12 2560000 2066986 462254 82% /usr/sap/ECT/DVEBMGS06
/dev/vg03ect/lv1 851968 139319 668114 17% /oracle/ECT/origlogA
/dev/vg03ect/lv2 851968 119079 687090 15% /oracle/ECT/origlogB
/dev/vg03ect/lv4 851968 119079 687090 15% /oracle/ECT/mirrlogB
/dev/vg03ect/lv3 851968 119079 687090 15% /oracle/ECT/mirrlogA
/dev/vg03ect/lv5 12533760 365684 11407638 3% /oracle/ECT/oraarch
/dev/vg03ect/lv6 3276800 38120 3036323 1% /oracle/ECT/saparch
/dev/vg04ect/lv1 30408704 25453592 4916408 84% /oracle/ECT/sapdata1
/dev/vg05ect/lv2 30408704 16504032 13796048 54% /oracle/ECT/sapdata2
/dev/vg04ect/lv3 30408704 21982488 8360392 72% /oracle/ECT/sapdata3
/dev/vg05ect/lv4 30408704 13360376 16915144 44% /oracle/ECT/sapdata4
/dev/vg04ect/lv5 30408704 12305600 17961680 41% /oracle/ECT/sapdata5
/dev/vg05ect/lv6 30408704 28403080 1990024 93% /oracle/ECT/sapdata6
/dev/vg04ect/lv7 30408704 9282040 20969112 31% /oracle/ECT/sapdata7
/dev/vg05ect/lv8 30408704 6096296 24122520 20% /oracle/ECT/sapdata8
/dev/vg02ect/lv7 2097152 37396 1931026 2% /oracle/ECT/sapreorg
/dev/vg02ect/lv10 6291456 3841322 2297045 63% /oracle/ECT/920_64
/dev/vg02ect/lv14 2097152 1605967 460498 78% /opt/vertex/ECT
ecc6ta# cd /ecc6ta# du -sk *
8 1
8 BE_ELC_ecc6t_200606290248.log
1632 DATA-PROTECTOR
0 HORCM
0 bin
0 ca_lic
0 cdrom
208 dev
170312 etc
0 export
0 ftpcmd_be
189376 home
0 ip
0 lib
0 lost+found
8 make_sys_image.log
8 mapfile
0 mnt
0 net
0 nfsm-
8 nohup.out
0 nsr
7058233 opt
139997294 oracle
0 pits
8 pvgfix.sh.log
40368 root
0 rpcmod-
12294063 sapbasis
1701237 sapmnt
16 sapr3.cfg
5 sapwork
131704 sbin
104 scripts
111485 stand
0 tcpm-
1881072 tmp
0 tmp_opt_vertex
0 udpm-
23166207 usr
3206080 var
ecc6ta#
ecc6ta#
ecc6ta# du -sk
289950084 .ecc4q# du -sk *
1632 DATA-PROTECTOR
0 HORCM
8 adviser.out
0 bin
0 ca_lic
0 cdrom
128 dev
0 do
0 done
10495066 dvl
16 ecc4q
158568 etc
0 fcmsutil
390632 home
204336 interface
0 lib
0 lost+found
8 make_sys_image.log
0 mnt
0 net
16 nohup.out
120 nsr
7473724 opt
144623250 oracle
0 pits
0 qa
141876 root
112362304 sapbasis
3508228 sapmnt
882653 sapwork
131744 sbin
143666 sbx
104 scripts
173558 stand
0 t
343544 tmp
8 tony
24689050 usr
3223336 var
ecc4q# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 2097152 313512 1769768 15% /
/dev/vg00/lvol1 512499 173558 287691 38% /stand
/dev/vg00/lvol7 8388608 3258760 5089872 39% /var
/dev/vg01/lv2 16777216 20580 15709353 0% /var/adm/crash
/dev/vg00/lvol6 16777216 3519144 13154520 21% /usr
/dev/vg02ECQ/lv2 4194304 3574699 580893 86% /usr/sap/ECQ
/dev/vg00/lvol5 2097152 360480 1723136 17% /tmp
/dev/vg02ECQ/lv8 4194304 900277 3088169 23% /sapwork
/dev/vg02ECQ/lv1 4718592 3528364 1115912 76% /sapmnt/ECQ
/dev/vg05ECQ/lv9 136314880 112857271 21993593 84% /sapbasis
/dev/vg01/lv3 16777216 163176 15575717 1% /root
/dev/vg02ECQ/lv3 524288 157985 343419 32% /oracle
/dev/vg02ECQ/lv5 4194304 17496 3915764 0% /oracle/stage
/dev/vg02ECQ/lv9 4194304 1633036 2401201 40% /oracle/stage/920_64
/dev/vg02ECQ/lv6 524288 84099 412733 17% /oracle/client
/dev/vg02ECQ/lv4 524288 35262 458493 7% /oracle/ECQ
/dev/vg02ECQ/lv7 2097152 56063 1913589 3% /oracle/ECQ/sapreorg
/dev/vg05ECQ/lv8 30408704 5095688 25115312 17% /oracle/ECQ/sapdata8
/dev/vg04ECQ/lv7 30408704 13903032 16376728 46% /oracle/ECQ/sapdata7
/dev/vg05ECQ/lv6 30408704 28403080 1990024 93% /oracle/ECQ/sapdata6
/dev/vg04ECQ/lv5 30408704 12305608 17961672 41% /oracle/ECQ/sapdata5
/dev/vg05ECQ/lv4 30408704 13360376 16915144 44% /oracle/ECQ/sapdata4
/dev/vg04ECQ/lv3 30408704 21982488 8360392 72% /oracle/ECQ/sapdata3
/dev/vg05ECQ/lv2 30408704 16504032 13796048 54% /oracle/ECQ/sapdata2
/dev/vg04ECQ/lv1 30408704 25536112 4834528 84% /oracle/ECQ/sapdata1
/dev/vg03ECQ/lv6 3276800 33872 3040280 1% /oracle/ECQ/saparch
/dev/vg03ECQ/lv2 851968 119079 687090 15% /oracle/ECQ/origlogB
/dev/vg03ECQ/lv1 851968 129679 677152 16% /oracle/ECQ/origlogA
/dev/vg03ECQ/lv5 12533760 1463231 10378685 12% /oracle/ECQ/oraarch
/dev/vg03ECQ/lv4 851968 119079 687090 15% /oracle/ECQ/mirrlogB
/dev/vg03ECQ/lv3 851968 119079 687090 15% /oracle/ECQ/mirrlogA
/dev/vg02ECQ/lv10 6291456 3942265 2202369 64% /oracle/ECQ/920_64
/dev/vg00/lvol4 8388608 5797624 2570960 69% /opt
/dev/vg02ECQ/lv14 2621440 1721883 843492 67% /opt/vertex/ECQ
/dev/vg00/lvol8 8388608 407800 7918496 5% /home
/dev/vg04ECQ/lv9 62914560 10527402 49113000 18% /dvl/interface
ecc3d.am.elcompanies.net:/usr/sap/trans
25165824 17689248 7009392 72% /usr/sap/trans
10.252.110.233:/qa/interface
31422384 206072 31216312 1% /interface
/dev/vg04ECQ/lv10 62914560 175618 58817763 0% /sbx/interface

================

[Kgreen]

nope iam having problem with / file system and not /root. But I tried dismounting that also.

[Kgreen]

Rex, /root is altogether a diff volume group

[Michal Toth]

at first glance, everything looks fine indeed...

well, if downtime is an option, then I'd umount those nfs mounts and then check again

or

to check fs consistency, you can split the mirroring for / and mount the new lv somewhere and work with that one

[Kgreen]

Yes you are right. I have that option in mind.But downtown is the issue. Its very strange that both systems are very identical except for the nfs mounts on which i dont see an issue. I have given up comparing at this moment but still if i run this command does not show any file thats big or hint to trace out whats using the / space

find / -size +100000c -xdev -exec ll -rt {} \; | more

[Bill Hassell]

Please post the following command:

du -kx / | sort -rn | head -20

Looking for big files is a waste of time because you do have legitimate files that might be considered to be big. The above command reports where the big directories are located. A 'normal' system would have a report like this:

78448 /
33000 /etc
32344 /sbin
14808 /etc/opt
13216 /etc/vx
12712 /root
11024 /etc/vx/type
8472 /etc/opt/resmon
5904 /sbin/fs
4336 /etc/vx/type/gen
3592 /etc/opt/resmon/log
3568 /etc/opt/resmon/lbin
3512 /etc/opt/samba
3496 /sbin/fs/vxfs
3480 /etc/opt/samba/codepages
3216 /etc/vx/type/static

What you are looking for is a very large directory at the top of your list that is very different than these numbers. Post the output and we can help locate the problem. NOTE: if /dev is in your list, the problem is very simple. Run this command:

find /dev -type f -exec ll {} \;

Anything listed by this command is wrong and does not belong in the /dev directory.

[Geoff Wild]

You need to do a du -sk * from /

Eliminate all the mounted filesystems.

Then investigate all the sub dirs on / (non mounted)

In particular - check /etc

Rgds...Geoff

[Kgreen]

ecc6ta# du -kx / | sort -rn | head -20
304432 /
170312 /etc
131704 /sbin
96048 /etc/vx
82752 /etc/vx/type
34504 /etc/vx/type/static
25784 /etc/opt
22688 /sbin/fs
21720 /etc/vx/type/gen
20064 /etc/lp
19160 /etc/lp/interface
18256 /etc/lvmconf
17632 /etc/opt/resmon
17376 /etc/lp/interface/model.orig
16248 /sbin/fs/vxfs
16224 /etc/vx/type/raid5
12728 /etc/opt/resmon/lbin
12016 /etc/vx/static.d
11256 /etc/vx/static.d/build
10296 /etc/vx/type/fsgen

[Michal Toth]

Bill & Geoff

if you'd read thru the entire post you could save yourself some time :-)

[Michal Toth]

let me rephrase the question:

why does bdf show different data then du:

ecc6ta# du -kx / | sort -rn | head -20
304432 /
..trim...

ecc6ta# bdf
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 2097152 1984392 111928 95% /
...trim...

[Kgreen]

Good observation. I wish I could answer, then I would have resolved this issue.

[Kgreen]

Good observation. I wish if I had known that , then there was no need to post this call

[Geoff Wild]

As you see - you /etc is 170 MB and /sbin 130...check thos to see if there is something abnormal...

In /etc - do you have any core files?

Or check the /etc/lvmconf dir

DO you have a lot of files in there?

Run a du -sk /etc/* |sort -n

Rgds...Geoff

[Michal Toth]

i've filtered the output of your lsof a bit, you can find it attached here... you can use it to check on those files, it's just 33 files that are opened so take a look if all of those exist

if you find a single file that does not, then i'd use fsdb to get the size of the file from it's inode directly

but i'd rather reboot

[Michal Toth]

attachment here