Root Partition 95% full

[Sameer77]

Hi gurus
We r having hp integerity server with HP-UX 11.23 installed. from last 10 days root partition is showing 95% full. bdf is showing 95% full .But du is showing less utilization. I have attached output of both commands. I have rebooted server. Still no change.Please help.

[Mel Burslan]

it is hard to see from the directory sizes in your output. The following script snippet will find all regular files in the / filesystem and add up the sizes together. It is not expected to match the bdf output exactly but it should be within the vicinity. If not, you can edit the /tmp/filesizes and look to see any suspiciously large or small files listed.

code:

find / -xdev -type f > /tmp/filelist
total=0
for file in `cat /tmp/filelist`;
do
templine=`du -sk $file`
echo $templine > /tmp/filesizes
size=`echo $templine | awk {'print $1'}`
(( total=${total}+${size} ))
done
echo ${total}

[Raj D.]

Sameer,

You can find the large file as given below, and can be check if they are normal:

# find / -type f -xdev -exec ls -l {} \; > tmp.list
# cat tmp.list | sort +4 -5nr > tmp1.list
# head -n 30 tmp1.list
You will see the big files on / (root) filesystem.


Hth,
Raj.

[Yogesh M Puranik]

Hi,

Please check is thr any core file present under /, that you can flush,


# find / -type f -xdev -name "core" -exec ls -l {} \;

#>core

It might be possible some times back related files may generate under /.

[Matti_Kurkela]

You have application directory trees /gw, /instgw and /bkindex on the root filesystem. This is not recommended.

Some applications can open temporary files on disk, then immediately unlink() them. This makes the temporary files accessible to the application (only) and guarantees that they will be removed when the application is stopped. If the application does this and is configured to start automatically at system boot, this might explain the discrepancy between bdf and du.

You should set up new LVs (maybe even a new VG) for /gw, /instgw and /bkindex.

To transfer the application that is currently at /gw to a new LV:

1.) Stop the application.

2.) After setting up the filesystem on a new LV, mount the new LV to a temporary location (e.g. /mnt).

3.) Copy everything from the application directory to the temporary location, preserving the file permissions. A "traditional" way to do this is to pipe two tar processes together:

(cd /gw; tar cf - .) | (cd /mnt; tar xpf -)

4.) Make sure the /mnt directory has the same permissions as /gw

ls -ld /gw /mnt
(use chown/chmod to change /mnt as necessary)

5.) Unmount /mnt

6.) Move /gw to /gw.old

7.) Create an empty directory /gw to act as a mount point.

8.) Edit /etc/fstab to make the system mount the new LV (that was at /mnt) to /gw automatically at every boot.

9.) Run "mount /gw"; if your /etc/fstab syntax is correct, this will mount the new LV without errors.

10.) Start the application and test it. It should work exactly the same as before.

11.) If the test is succesful, remove /gw.old. (If the test failed, unmount the new LV, remove the new line you added to /etc/fstab, remove the empty /gw directory and move /gw.old back to /gw. Then figure out what went wrong and try again.)

Do the same with /instgw and /bkindex.

MK

[Michael Steele_2]

Hi

Verify LVM or VXVM please.

If LVM and NOT vxvm the swremove /vxvm/core. Note that this will reduce /etc/vx and / by half.

[Sameer77]

Hello gurus
i have searched for core files. No files.
Attached is the output of ll /tmp.
I can find some big files like crashinfo. Not sure can i remove it ?

[Raj D.]

I guess you have /tmp on root. And not a separate filesystem.
Btw crashinfo can be deleted, it is a diagnostic utility. You can keep a copy of crashinfo file elsewhere and can delte the files:

-rwxrwxrwx 2 202 200 6568736 Oct 15 2007 crashinfo
-rwxrwxrwx 2 202 200 6568736 Oct 15 2007 crashinfo-a-i
-rwxrwxrwx 1 root sys 2491987 Sep 17 13:50 crashinfo.bin

You can feeup 15MB.

Btw,
- what is the size of root filesystem, ( /)

hth,
Raj.

[Michael Steele_2]

Look, you're getting a lot of bad advice. Why are you looking at the /tmp file system? You want to be in lvol3 not lvol4. Now, do what I am telling you and remove vxvm from /etc/vx. Unless you are using it and I don't see any indicatin that you are.

Go back to your bdf output and stop listen to bad advice that doesn't pertain to lovl3, this means, ignore all crashinfo bs.

[Suraj K Sankari]

Hi,
Did you tried with lsof ?
if not try this command

lsof +L1

To see the files, use the utility lsof with the +L1 option. The +L option lists files with less than that many links; thus, +L1 lists files with less than one link (or zero links).

Suraj

[Raj D.]

Sameer77,
Someone given you bad advise to look at /tmp.

Remember /tmp is not root filesystem.
Root file system is /dev/vg00/lvol3 .

Look files under / filesystems.

-Again what is your root file system size, and how much it was before you saw it is 95% full.
- did you checked with lsof , if any program using the root fs space on the fly.

Hth,
Raj.

[Robert-Jan Goossens_1]

Hi Sameer,

Could you post the output of next commands?

# du -ksx / | sort -rn | more

# df -k /

Regards,
Robert-Jan

[Tingli]

What is just under your /? Are there any big files under it?

[Sameer77]

Here is o/p of du & df.
lsof is not installed.
there are no core files.

[Raj D.]

Sameer,
The output doesnt have much data ,
Try runnig below with * , to see the directory listing having data, and then to figure out if anything is wrong.

# cd / ; du -ksx * | sort -rn

Your root fs is 1GB , and it is ok for 11.23 , in normal usage root space usage for 11.23 is around ( 500MB to 600MB).

So it looks like somewhere some non-root data present that might be filling it up to 95%. Around 300MB suspected.

Could you paste the output of the above command.

Cheers,
Raj.

[Raj D.]

Sameer,

- Also what was the usage of : / (root filesystem ) before you saw it is 95%.


tx,
Raj.

[Sameer77]

usage of / was 50%

[Raj D.]

Al right , so here it is clear,


#du -ksx / | sort -rn | more
346696 /

The usage is showing only 346MB.

However: df is showing 95%.


Download lsof : and do the follwing :

# lsof / > lsof.root.txt

-And look in the size coulmn which is the largest size of process is in use.
- Get the pid .
- check the pid with ps -ef.

And you will find the culprit that using the root space. Most probably that process needs to check , who is running it and why it is occupying that much root space.

Cheers,
Raj.

[Sameer77]

# cd / ; du -ksx * | sort -rn
4570000 opt
2955965 aakash2
2925534 aakash1
2795308 aakash3
2717840 usr
2360744 aakash4
2345816 var
1911421 aakash5
424542 ing_backup
140232 etc
136912 stand
126776 sbin
114270 ing_users
71144 gw
66808 tmp
4024 instgw
536 home
88 dev
8
8 root.profile
8 profile-bkup
8 prof1
8 prof
8 bkindex
0 test
0 sapmnt
0 net
0 lost+found
0 lib
0 dvd
0 bin
0 SD_CDROM
0 250209.idx

[Kanagaraj]

Hi Sameer,

as per you attachment,
/var
/opt
/tmp
/home file systems are separately mounted and / file system has only 1 GB and you need to check /gw directory.

This /gw is holding much space and /etc directory holing 140MB(this is system configuration directory so you can omit)

only you need to check /gw directory and try to move this directory to some other filesystem if this is not application directory.

otherwise a create file system and mount it like /gw.new then using tar command create archive file as follows

cd /gw
tar -cvf gw.tar ./*

then go into /gw.new file system and extract tar file in same place.

cd /gw.new
tar -xvf gw.tar.

then you can remove all file in gw directory (if it is application files then make sure to shutdown the application before doing tar and remove)

then un mount /gw.new and mount that file system to /gw directory.

Other wise best idea increase the size of your / file system (if you have installed ON-line JFS)

[Raj D.]

Sameer,

The lsof can be downloaded from here:
Itanium, 11.23 version:


http://hpux.connect.org.uk/ftp/hpux/Sysadmin/lsof-4.82/lsof-4.82-ia64-11.23.depot.gz

- once downloaded, ftp it to the server.
- and gunzip the file.
- install using swinstall (it is online)
- then run # lsof / > lsof_root.txt #(it will take a while to generate the output.)
- sort the SIZE column:
# cat lsof_root.txt | sort -k 7 > lsof_root_sorted.txt

- vi lsof_root_sorted.txt # and check the size column for largest size entry and its process and pid, ( and you got it.)
- do # ps -ef

Cheers,
Raj.

[Raj D.]

Here is the lsof binary, for 11.23
You can check this quickly if not able to download the depot.,
- download. ftp to the server ,
- check : # ./lsof -h
- execute as mentioned above & find out the processes using large root space.

hth,
Raj.

[Sameer77]

Hello Raj
This is production server. I cannot take risk of installation of any new thing. I know lsof will not harm sever. Still customer is not ready to agree.

[Raj D.]

But unfortuantely that is the only way you can find out which program using the root space.
- In that case the pid needs to be find out. And the application owner needs to be informed if they can bounce or kill it. So it will release the root space.

Btw , remember: if root become become 100% ,it may freeze the server and gradually login prompt may not come, and no one can log in.

[Raj D.]

Another option, if you have crashinfo on that box, and if you log a case to HP & if HP ask for
# crashinfo -v -ofiles ,output data, then they could analyze and tell which program/pid using most of the root space on the fly , which is not visible to du command.