unable to determine tmp filesystem usage

[Ninad_1]

Hi Gurus,

I have a HP 11i server on which the /tmp filesystem is 2.3 GB. I am unable to determine the usage of the /tmp filesystem . The output of df -k /tmp and du -sk /tmp show a large difference.Following are the output of df and du of /tmp
df -k /tmp
/tmp 2451328 total allocated Kb
825656 free allocated Kb
1625672 used allocated Kb
66 % allocation used
du -sk /tmp
55152 /tmp

I am unable to understand why df shows 1.6 GB used in /tmp but du shows only 55 MB.

Please guide me in sorting this out.

Regs,
Nad

[Norman_21]

The following commands show the same usage:
#df -k /tmp
#bdf /tmp Refer to attached!


df list by block so I think 1.6GB/2=55MB

Can anyone correct me if I'm mistaken

[Alex Lavrov.]

df indeed lists by blocks, but in this case "-k" option tells df to print the output in KB.

I think you get different values, because inode of some temporary file was removed. so du when it counts the sizze of files does not count it, but df does.

It happens when you remove some files when some program still uses it, so the pointer to the file still exists and the space is not deallocated, but the inode removed and you can't see the file with ls, du etc.

Did you remove some files from /tmp whithout checking is they are still in use by some program?

[Alex Lavrov.]

Forgot something...
To check what programs use /tmp do: lsof /tmp

[Ninad_1]

There seems to be no lsof in my system. Yes I had removed some files but at that time the df -k output showed about 30% used and now it is showing 66% used that means df shows around 800 MB increased whereas du shows only 55 MB.
So I havent deleted any files later on so
1) what could be the cause ?

As suggested may be files were used by some process when they were deleted - in such a case wont the
2) process give any error ?
3) how to determine how the files are getting deleted if any ?
4) What is the solution to this problem ?

Regs,
Nad

[Norman_21]

Sorry I miscalculated?

First, I'll list the files by size in /tmp to find out what files are growing:

du -k /tmp|sort -nbr|pg

The solution, I'll remove /tmp FS and recreate it again?

Backup whatever you need from /tmp and create an Ignite recovery tape before doing anything.

Maybe someone else has a better idea.

Thanks

[Norman_21]

Just to support Alex suggestion, find lsof for free:
http://hpux.connect.org.uk/hppd/cgi-bin/search

It'll help in the long run

[Alex Lavrov.]

Well, I still stick with my idea that you have some temporary file that inode was deleted and space is still allocated.

No, process won't give error message. Well, it depends on how the program was written, but it's not mandatory that an error will be raised, it's not windows after all.

To recreate /tmp? I don't think it's very wise, because it will require boot and as long as you don't have a problem with /tmp, and you don't, because there are still free space, I wouldn't do it.

By the rule, I always delete file in temporary filesystems like /tmp and /var/tmp, in two steps:

1. cat /dev/null >
2. rm

In first step I make sure that the space is deallocated, because I still have the inode and in the second step I remove the inode.

From my point of view, there are 2 solutions:

1. Get lsof and analuze the programs that uses /tmp. Can be difficult and time consuming.
2. Reboot. This way you can be sure that the program that uses /tmp is killed and the space will be free.

[Norman_21]

One last note:
2.3GB for a root file system /tmp is too high.

[Ninad_1]

Alex,

Good method to clear the /tmp filesystem.
I have a 2 questions for you.If your logic that inodes are existing which are occupying space whereas files are deleted -
1) Deleting a file means freeing the inode - otherwise the filesystem superblock will be in invalid state if we say that file is deleted but space is not freed.
2) If your logic is correct then the output of find /tmp | wc -l should vary a lot from the output of bdf -i /tmp which shows the number of inodes.
find /tmp | wc -l gives 6636
and bdf -i gives 6646 inodes used
which are almost same.

Please can you explain this

Regs,
Nad

[Alex Lavrov.]

1. When you use rm on the file, it removes the inode, updates the FS structure and marks the space that was allocated by the file as free. BUT ... it marks the space as free, only when the space is not used by any process. You can do a little experiment. Create a file with some content and write a program that opens the file and reads it in the loop (don't vi it, because vi copies the content to a different place and appends it there). Then from another session delete this file. You'll see that the programs does not fail, file is removed but the space is not freed. When you'll kill the program, the pointer will be destroyed and the space will be marked as free. (I'm away from any HPUX system, so can't try it by myself).

2. Why do you think it's should vary a lot? It's really depends on the system's state. Now you have a difference, which can be a very good explanation of the setuation that you have in /tmp. du output differs from df output.

[Ninad_1]

Hi,

I tried from one session
sh 2.sh | tee test_vol/testfile

2.sh script -

while true
do
dmesg
sleep 10
done

I allowed this to run for some time and then while the program was still running I removed the file , then after some time killed the process. Then found out that the space was released.Is this correct method for checking what you asked.
Can you tell me how I need to check

Regs,
Nad

[RAC_1]

what you checked is correct. If a file was being used by a process and was removed, spce will not be released.

You can check what files are being used as follows.

fuser -cu /tmp
fuser -u /tmp/somefile
lsof +D /tmp
lsof -f -- /tmp

read man pages of those commands.

[Ninad_1]

Thanks for the commands.
1) I have already tried using the fuser command - it shows which processes are using the /tmp filesystem.
2)I do not have the lsof command in my system.
3)The test I carried out released the space even though I removed the file while still the process using it was running - which is contrary to the Alexs explanation.
4) Finally the question still remains that why is the used space in df increasing if noone is removing any more files in /tmp

Please if someone can help me in sorting out this problem.

Regs,
Nad

[Alex Lavrov. · Accepted solution]

lsof is a very useful administrator's tool. It allows you you view what process uses what port, files etc.

I suggest that you take a look at it. It can be found here:

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.74/


About the test that "failed", well, I don't know exactly what you did and maybe there are situations that file is removed after all, but in the most cases the space remains.

Regarding /tmp, I'm out of ideas right now ;)

[John Payne_2]

Nad,

Once you have lsof, issue the following command:

lsof -a +L1 /var

This will list all processes that have open files in /var that have been unlinked.

The output looks like this:

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
syslogd 708 root 10w VREG 64,0x9 62142886 0 16607 /var (/dev/vg00/lvol9)
atswasdO8 1467 cck 6w VREG 64,0x9 181 0 617 /var (/dev/vg00/lvol9)
swagentd 1866 root 5u VREG 64,0x9 41 0 145 /var (/dev/vg00/lvol9)

In the example I give, syslogd has a large file in /var that has been unlinked. (I did this by deleting syslog.log.) Once I kill syslogd (or kill -HUP the process) the open file is released, the space is freed, and life goes on.

Hope it helps

John

[John Payne_2]

(Replace /tmp for /var. Sorry)

If you know the processes that have unlinked open files and fix them, my guess is that your problem will go away.

John

[Ninad_1]

John/Alex

Finally I managed to get lsof onto my system - as this is production system it took time for getting it done.
Finally I ran lsof -a +L1 /tmp and o/p is
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
vxsvc 1841 root 3u unix 64,0x5 0t0 0 32 /tmp/portal (0
x4b2029c0)
sqlplus 2864 dbaauto 0u REG 64,0x5 310 0 1395 /tmp (/dev/vg0
0/lvol5)
sqlplus 2909 dbaauto 0u REG 64,0x5 309 0 1396 /tmp (/dev/vg0
0/lvol5)
sqlplus 2926 dbaauto 0u REG 64,0x5 313 0 1397 /tmp (/dev/vg0
0/lvol5)
sqlplus 3002 dbaauto 0u REG 64,0x5 312 0 1399 /tmp (/dev/vg0
0/lvol5)
sqlplus 3015 dbaauto 0u REG 64,0x5 308 0 1393 /tmp (/dev/vg0
0/lvol5)
tee 7977 controlm 4u REG 64,0x5 7065 0 74 /tmp (/dev/vg0
0/lvol5)
tnslsnr 10998 oracle 3u REG 64,0x5 2377686926 0 329 /tmp (/dev/vg0
0/lvol5)
snmpdm 25922 root 2w REG 64,0x5 0 0 3160 /tmp (/dev/vg0
0/lvol5)
awservice 26532 root 3w REG 64,0x5 262 0 714 /tmp (/dev/vg0
0/lvol5)
aws_orb 26535 root 3w REG 64,0x5 262 0 714 /tmp (/dev/vg0
0/lvol5)
aws_sadmi 26538 root 3w REG 64,0x5 262 0 714 /tmp (/dev/vg0
0/lvol5)
caiLogA2 26539 root 3w REG 64,0x5 262 0 714 /tmp (/dev/vg0
0/lvol5)
caiUxOs 26544 root 3w REG 64,0x5 262 0 714 /tmp (/dev/vg0
0/lvol5)
cai_UxOs_ 26554 root 3w REG 64,0x5 262 0 714 /tmp (/dev/vg0
0/lvol5)

Also please see o/p of lsof +D /tmp - I have pasted only tnslsnr entry related o/p

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
tnslsnr 10998 oracle 12u unix 64,0x5 0t0 330 /tmp/.oracle/s#109
98.1 (0x5bdb0b80)
tnslsnr 10998 oracle 13u unix 64,0x5 0t0 331 /tmp/.oracle/sEXTP
ROC (0x5ec53a00)

This indicates that almost all memory is consumed by the tnslsnr process.
Now I have 2 questions.
1) Will stopping the listener and then starting it will release the space or I need a reboot of the system ? - Since this is a production system please let me know the shortest time taking solution.
2) Why is tnslsnr taking so much space ? this is Oracle 9.2.0.5 version.

Alex/John - I am giving you 9 points each Alex for providing good info on lsof and John for guiding in the use of lsof to determine the prob.

Please answer my questions.

[Alex Lavrov.]

Well, according to the output you have not only tnslsnr that uses this directory. I suggest that you'll start taking those processes down one by one, starting with the less important. This is the only way I can see, without rebooting the system.