root filesystem slowly filling up

[Bram Vermeulen]

Greetings,

The last couple of weeks the root filesystem is slowly filling up without there being any significantly big files present. I'm guessing there's a process filling up the filesystem but seeing as it's "/" I'm afraid to just kill random processes. A reboot is possible but only a last resort.

Filesystem 512-blocks Used Available Capacity Mounted on
/dev/vg00/lvol10 48912 17072 31840 35% /home
/dev/vg00/lvol4 17284814 8678974 8605840 51% /o
/dev/vg00/lvol5 9408624 5769632 3638992 62% /opt
/dev/vg00/lvol6 1773760 40640 1733120 3% /tmp
/dev/vg00/lvol7 11424340 3629208 7795132 32% /u
/dev/vg00/lvol8 11847840 5479712 6368128 47% /usr
/dev/vg00/lvol9 10408368 2621376 7786992 26% /var
/dev/vg00/lvol1 620624 342000 278624 56% /stand
/dev/vg00/lvol3 1031728 968400 63328 94% /

This is the "top" result:
Load averages: 0.02, 0.02, 0.03
287 processes: 221 sleeping, 65 running, 1 zombie
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.01 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%
1 0.02 0.0% 0.0% 0.2% 99.8% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.02 0.0% 0.0% 0.2% 99.8% 0.0% 0.0% 0.0% 0.0%

Memory: 442488K (121352K) real, 772020K (233104K) virtual, 261652K free Page# 1
/27

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 52 root 152 20 3096K 2752K run 1441:53 2.23 2.23 vxfsd
1 pts/21 25594 jb 152 20 48020K 4164K run 0:00 1.10 0.29 rtsora
0 ? 1449 root 152 20 25664K 5284K run 323:22 0.22 0.21 rds
0 ? 1469 root 152 20 28836K 3824K run 2:39 0.18 0.18 swagentd
0 ? 1619 root 152 20 52004K 13700K run 17:47 0.16 0.16 vxsvc
0 ? 1325 root 152 20 32120K 4396K run 1:31 0.14 0.14 rpcd
1 pts/22 18473 root 178 20 10724K 968K run 0:02 0.13 0.13 top
0 ? 440 root 152 20 8352K 2360K run 9:47 0.12 0.12 utmpd
0 ? 38 root 152 20 288K 256K run 59:02 0.10 0.10 schedcpu
0 ? 1214 root 154 20 7736K 844K sleep 120:12 0.07 0.07 sendmail

Memory: 436964K (130592K) real, 767448K (253624K) virtual, 267172K free Page# 2
/26

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 38 root 152 20 288K 256K run 59:02 0.08 0.08 schedcpu
0 pts/29 18760 jvdm 154 20 5152K 468K sleep 0:00 0.08 0.08 ksh
1 ? 1 root 152 20 2216K 820K run 1:09 0.06 0.06 init
0 ? 19 root 191 20 216K 192K run 27:30 0.06 0.06 ksyncer_
0 ? 905 root 152 20 11076K 1400K run 0:43 0.06 0.06 automoun
0 ? 1214 root 154 20 7736K 844K sleep 120:12 0.05 0.05 sendmail
0 pts/26 20613 eb 152 20 48404K 4612K run 0:00 0.05 0.05 rtsora
1 ? 39 root 191 20 144K 128K run 0:23 0.04 0.04 pagezero
0 ? 40 root 191 20 144K 128K run 0:13 0.04 0.04 cmcd
1 ? 1934 lvdw 152 20 48024K 4172K run 0:00 0.04 0.04 rtsora

There's 25 more pages but none with very long "time" values.

Is there anything there that I can safely kill?
The rtsora processes don't fill up the root filesystem.
I'm guessing I shouldn't kill vxfsd and rds for obvious reasons?

[Suraj K Sankari]

Hi,
Check any open file is there under / also check any backgound process is running which is generating any files under root FS or check any core/zip files is there under /

For checking open files
lsof +L1 +D /

find / -name core -print


Suraj

[Ninad_1]

HI,

Few things to check would be
1. find / -xdev -name core
To find core files in / filesystem (The command suggested in above post will check through all the filesystems)
2. du -kx / | sort -nr | head -20
Will give you top space consuming files/dirs.
3. As suggested in above post - you can also used lsof to see if there are any open files being used in / (which you may observe in the du output) before deleting them.
4. Check in /dev if there is any regular file got created during any backup or so.
find /dev -type f

Regards,
Ninad

[Bram Vermeulen]

Thanks for the responses.

"find / -xdev -name core" didn't give any results, the "find / -name core -print" command gave some but none in the / filesystem.
BHPUX01 root "/"> find / -xdev -name core
BHPUX01 root "/"> find / -name core -print
/opt/java6/sample/jnlp/corba/src/core
/var/opt/cifsclient/core

"du -kx / | sort -nr | head -20" also didn't really show anything abnormal:
BHPUX01 root "/"> du -kx / | sort -nr | head -20
474792 /
265384 /k/2007/jour2007
265384 /k/2007
265384 /k
109624 /sbin
84600 /etc
54376 /etc/vx
46792 /etc/vx/type
22680 /sbin/fs
21200 /etc/opt
17784 /etc/vx/type/static
16248 /sbin/fs/vxfs
14376 /etc/vx/type/gen
10856 /etc/vx/type/raid5
10256 /.swa
10096 /.swa/cache
9728 /etc/opt/resmon
6920 /etc/vx/static.d
6696 /etc/vx/static.d/build
5864 /etc/opt/resmon/lbin

"lsof +L1 +D /" gives 1315 results, is there a way to narrow the search?
In case this is as narrow as it gets I attached the result of the command.

"/dev -type f" gave as result:
BHPUX01 root "/"> find /dev -type f
/dev/stdout
BHPUX01 root "/"> cd /dev
BHPUX01 root "/dev"> ls -l stdout
-rw-r--r-- 1 root sys 0 Jan 25 2007 stdout

I'm not a very experienced unix administrator but I'm all we got here, I didn't see anything out of the ordinary in these results though. Maybe I overlooked something?

Thanks again for the responses.

[Ninad_1]

Hi,


Please see you du output
The following is consuming a lot of space -
265384 /k/2007/jour2007
265384 /k/2007
265384 /k

What is this /k firectory and the contents inside it ? Cant this be outside the / filesystem - in some other filesystem or as a separate filesystem. ??


Regards,
Ninad

[Bram Vermeulen]

It's an archive directory with printfiles from 2007, it shouldn't be open. Does the fact that it's in the du output mean that it is still open after all?
I can move it to another filesystem but wouldn't that just postpone the problem for a month since the filesystem is slowly filling up?

[Ninad_1 · Accepted solution]

Hi,

From the du output you can see the disk usage and not open files (lsof shows you open files).
So before moving the directory, it may be worthwhile if it appears in the lsof or you can do
lsof +D /k

Since this directory consumes around 260 MB, moving this outside / will solve your immediate problem.
Regarding / getting full slowly - I think after this movement of files, you need to observe which directory/files in / are growing and then take appropriate action after understanding the root cause.

Regards,
Ninad

[V. Nyga]

Hi,

of course are always some log files in / .
You can check/cut them in SAM:
sam - Routine Tasks - System Log Files

Also, can you give us your 'bdf'?
Are all lvol's mounted?

Volkmar

[Bram Vermeulen]

Yes, all lvols (except for swap/dump) are mounted.
This is bdf result:
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 516096 484200 31664 94% /
/dev/vg00/lvol1 311296 171000 139312 55% /stand
/dev/vg00/lvol9 5234688 1308280 3895880 25% /var
/dev/vg00/lvol8 5947392 2741840 3182056 46% /usr
/dev/vg00/lvol7 5971968 1815464 3896746 32% /u
/dev/vg00/lvol6 892928 20904 865984 2% /tmp
/dev/vg00/lvol5 4718592 2884816 1819496 61% /opt
/dev/vg00/lvol4 8912896 4339298 4303096 50% /o
/dev/vg00/lvol10 24576 8536 15920 35% /home

The only log files in / are those in /etc:
â â /etc/rc.log 367 37590 10240 â
â â /etc/rc.log.old 99 10220 10240 â

I deleted the old one and trimmed the one currently being used but that didn't make any difference in the bdf result (it actually increased 40kb).

[SKR_1]

you can find the bigger files with below command.

find / -xdev -size +1000000c -exec ll {} \;

you can move /k to some other FS.

Thanks

SKR

[V. Nyga]

Hi again,

repeating the 'du' command you should see the directoy that increases.
You have to compare cronologically different snapshots of your 'du' command.

At the first sight I would suspect /.swa/cache ...
Can you give us a 'll' of your / ?

Volkmar

[Bram Vermeulen]

The result of 'du' hasn't changed much so far, i'll repeat it every day to see. Thanks.

BHPUX01 root "/"> ll
total 1060
-rw------- 1 root sys 505 Jan 7 2008 .Xauthority
-rw------- 1 root sys 8551 Oct 6 16:35 .bash_history
drwx------ 3 root sys 96 Dec 12 2007 .mozilla
-rw-rw-rw- 1 root sys 9 Dec 12 2007 .mozilla-license
-r--r--r-- 1 bin bin 6759 Feb 1 2008 .profile
dr-x------ 3 root sys 96 Nov 30 2007 .secure
drwx------ 2 root sys 96 Feb 1 2008 .ssh
drwxr-xr-x 6 root sys 8192 Dec 1 2006 .sw
drwxr-xr-x 4 root root 96 Dec 7 2007 .swa
drwxr-xr-x 2 root sys 8192 Jan 7 2008 .vnc
drwxr-xr-x 2 root sys 96 Mar 26 2007 SD_CDROM
-rw-rw-r-- 1 root sys 0 Oct 10 23:00 TOT
-rw-rw-r-- 1 root sys 3072 Oct 10 23:00 TOT.idx
lr-xr-xr-t 1 root sys 8 Mar 26 2007 bin -> /usr/bin
drwxrwxrwx 2 root sys 96 Dec 5 2006 cdrom
dr-xr-xr-x 14 bin bin 8192 Feb 25 2008 dev
dr-xr-xr-x 30 bin bin 8192 Oct 13 11:39 etc
drwxr-xr-x 5 root root 96 Jul 12 2007 home
drwxrwxrwx 3 root sys 96 Oct 8 2007 k
lr-xr-xr-t 1 root sys 8 Mar 26 2007 lib -> /usr/lib
drwxr-xr-x 2 root root 96 Dec 1 2006 lost+found
-rw-rw-r-- 1 root sys 2144 Oct 3 2007 mail
-rw-r--r-- 1 root sys 49 Mar 16 2007 make_sys_image.log
drwxrwxrwx 3 root sys 96 Mar 17 2008 mnt
dr-xr-xr-x 1 root root 1 Dec 21 2007 net
-rw-rw-rw- 1 root root 2930 Nov 16 2007 null
drwxrwxr-x 24 tftp hb 1024 Oct 13 08:59 o
dr-xr-xr-x 58 bin bin 8192 Dec 5 2007 opt
lrwxrwxrwx 1 root sys 4 Dec 21 2007 p -> /o/p
-rw-rw-rw- 1 root sys 264 Jan 9 2008 root_orig
dr-xr-xr-x 15 bin bin 8192 Mar 15 2007 sbin
drwxrwxrwx 3 root sys 8192 Oct 13 08:58 script
-rw-rw-rw- 1 root sys 30134 Jan 15 2008 sqlnet.log
dr-xr-xr-x 9 bin bin 8192 Dec 21 2007 stand
dr-xr-x--x 3 root sys 96 Nov 30 2007 tcb
drwxrwxrwt 4 root root 8192 Oct 13 13:39 tmp
-rw-r--r-- 1 root sys 22518 Oct 13 2002 tusc.1
drwxrwxr-x 25 root root 1024 Oct 13 11:51 u
-rwxrwxrwx 1 root sys 304119 Jan 21 2008 unxlink2
dr-xr-xr-x 28 bin bin 8192 Jul 10 2007 usr
dr-xr-xr-x 26 bin bin 8192 Mar 26 2007 var

[V. Nyga]

Hi again,

if you use mozilla then check:
drwx------ 3 root sys 96 Dec 12 2007 .mozilla

This one I would watch:
drwxrwxrwx 3 root sys 8192 Oct 13 08:58 script

HTH
Volkmar

[Dennis Handly]

>-rw-r--r-- 1 root sys 0 Jan 25 2007 stdout

This looks bogus but removing it doesn't save any space.

As mentioned by Ninad move the /k directory, it's 1/4 of your space. If you still need it under the old name, you can add a symlink.

[ManojK_1]

Hi Bram,

Do you enabled auditing in the HP Unix Server. If it is enabled by default the log will generate insdie /.secure folder.

I have faced the same problem and found that auditlog is increasing in size.

you can check the size of each folders and files inside / by the following commnd
#cd /
#du -sk * | sort -rn

The above command will arrange the folders in descending order in size.

Please send the out put of above command.

Manoj K

[Bram Vermeulen]

I checked the logging in SAM like V. Nyga said earlier but didn't see anything about /.secure

Here's the output.

BHPUX01 root /> du -sk * | sort -m
0 SD_CDROM
0 TOT
8 TOT.idx
0 bin
0 cdrom
72 dev
84552 etc
104 home
203256 k
0 lib
0 lost+found
8 mail
8 make_sys_image.log
0 mnt
0 net
8 null
6663608 o
2856904 opt
0 p
8 root_orig
109624 sbin
1784 script
32 sqlnet.log
162432 stand
664 tcb
3472 tmp
24 tusc.1
1863147 u
304 unxlink2
2712120 usr
1292096 var

[ManojK_1]

Dear Bram,

What is this /k.Why it is required. What is the content inside /k.

It has covered around 203MB. If you not required this delete or remove it.

You can check whether auditing is enabled by the command audisp.

Manoj

[Yogeeraj_1]

hi manoj,

can you post the output of the following commands?

file /o

ll / |grep "o "

Seem like there is a directory/file on the root file system which is using much disk space...

revert!

kind regards
yogeeraj

[ManojK_1]

Dear Yogeeraj,

I am not the person who is raised this issue.
As per Bram the person who raised this issue /o mounted separately. So it will not affect the / size increse.

Manoj

[ManojK_1]

Dear Bram,

Sorry for cheking the auditing status you have to use the command audsys

,the output will give the auditing status like

auditing system is currently off
or
auditing system is currently on

Manoj

[V. Nyga]

Hi again,

well, you have .secure in your listing:
-rw------- 1 root sys 505 Jan 7 2008 .Xauthority
-rw------- 1 root sys 8551 Oct 6 16:35 .bash_history
drwx------ 3 root sys 96 Dec 12 2007 .mozilla
-rw-rw-rw- 1 root sys 9 Dec 12 2007 .mozilla-license
-r--r--r-- 1 bin bin 6759 Feb 1 2008 .profile
dr-x------ 3 root sys 96 Nov 30 2007 .secure

In generell - you have to check which directory or file is changing continuosly.

Check directories like .mozilla, .swa, .secure, /script.
Is /.swa/cache changing?

Volkmar

[V. Nyga]

Hi again,

can you repeat the command:
"du -kx / | sort -nr | head -20"
from yesterday and give us the output?

V.