tmp file system

[Sarjerao]

when I try to list files from tmp i got foll error :- /tmp =>ll
ls: out of memory
Memory fault(coredump)
***************************************
drwxrwxrwx 26 root root 15555584 May 11 18:32 tmp
can anybody explain why this error is comming and what this 15555584 means ?

[harry d brown jr]

Do a bdf, what does it display? Is /tmp full?

Do a swapinfo, what does it display?

live free or die
harry d brown jr

[Alex Lavrov.]

You have 15555584 (!!!!!!) file in /tmp, so ls, when it takes the files into some array in the memory, cannot held allocate that big array and crashes.

It's a problem and can cause performance issues on your sever. I suggest to boot the box in single user mode, mount /tmp and delete all the files there, then boot the machine normally.

[Steven E. Protter]

It would appear you have so many files in /tmp there is not enough memory to list them.

bdf /tmp

Does it show /tmp or / as the directory.

I know applicaitons like Software AG Natural and Adabas still store files on /tmp but in practice no other applications really should do that.

swapinfo -tam

Is your system under memory pressure.

I would consider checking how well the system is patched and bringing it up to date if its far behind.

If possible, reboot and try again.

Don't specifically know what the 15555584 means.

SEP

[Stephen Keane]

15555584 is the size of the directory (in bytes). Which suggests either (a) you have a lot of files in /tmp, or (b) the /tmp filesystem is trashed. If you don't have a lot of files in /tmp (bdf -i will show you the number of i-nodes in use) You could always go into single user mode, unmount /tmp, fsck it and remount it.

[Alex Lavrov.]

Yes, it's size... 26 is number of links, not 15555584.


Sorry, my mistake, got confused with the columns, please ignore my reply :)

[Sarjerao]

bdf output :-
/dev/vg00/lvol6 819200 197368 621832 24% /tmp

There is no space problem, how i can remove, because it is not allowing me to list files.

[MarkSyder · Accepted solution]

As this is a temporary filesystem I'll assume it's safe to delete anything over a week old.

find /tmp -mtime +7 -exec rm {} \;

Please note the spaces in that command are important.

Mark Syder (like the drink but spelt different)

[Stephen Keane]

What does

# bdf -i

give you? You could have a large number of small (or empty) files in /tmp you need to look at i-node usage too.

[Sarjerao]

I just started command
bdf -i shows folls..
/dev/vg00/lvol6 819200 196465 622735 24% 391259 322853 55% /tmp
so many files in temp;
run foll command....
find /tmp -atime +30 -exec rm {} \;

and waiting for prompt...............

[Alex Lavrov.]

If it's not a critical server, I go back to my suggestion, to boot in single mode and empty /tmp ....

Then, after reboot, when /tmp is empty, monitor it for several hours to see what files are created there and you'll probably find the problem.

[Sarjerao]

after running whole night script for deleting files, now i am able to 'ls'in tmp; But still my tmp showing like this :-
drwxrwxrwx 26 root root 15555584 May 12 10:15 tmp
why this still shows 15555584.

[Naveej.K.A]

Hi

I think a reboot is inevitable at this point of time.

Reboot in the single user mode and delete the files in the /tmp filesystem and see how it behaves after booting to the multi user mode.

Regards,
Naveej

[Sarjerao]

In tmp continously every second "AAAa*, AAAb* ........." these types of files of 0 bytes generating; how i can trace which application generating thses files.
Fuser -u /tmp shows only root user

[Alex Lavrov.]

Who si the owner of these files?


Do you have something in your crontab?

crontab -l

May be in the crontab of some user, check /var/spool/cron/crontabs for other users crontabs.

Check what services you have running when the server boots.

[Sarjerao]

no any crontab process running,
owner is root:root of these files.
I unable to trace which process generates these files.

[Alex Lavrov.]

What's the output of:

lsof /tmp

[Joseph Loo]

hi,

these files r just temporary store there during some application job request and it may also reside in /var/tmp depending on how the job is written.

if u r interested whom generates it, ll command should show the user and group. the last 5 digit is the process id when the job was running, and u may monitor it with lsof when the job is running.

by the way, i am running Oracle apps in the server and always see such files.


regards.

[Sarjerao]

lfof /tmp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
snmpdm 988 root 2w VREG 64,0x6 0 102 /tmp (/dev/vg00/lvol6)
hp_unixag 999 root 1r VDIR 64,0x6 15555584 2 /tmp
hp_unixag 999 root 2r VDIR 64,0x6 15555584 2 /tmp
mib2agt 1007 root 1r VDIR 64,0x6 15555584 2 /tmp
mib2agt 1007 root 2r VDIR 64,0x6 15555584 2 /tmp
trapdesta 1015 root 1r VDIR 64,0x6 15555584 2 /tmp
trapdesta 1015 root 2r VDIR 64,0x6 15555584 2 /tmp
opcctla 1345 root cwd VDIR 64,0x6 15555584 2 /tmp
opcmsga 1346 root cwd VDIR 64,0x6 15555584 2 /tmp
llbserver 1348 root cwd VDIR 64,0x6 15555584 2 /tmp
opcuisrv 1349 root cwd VDIR 64,0x6 15555584 2 /tmp
opcacta 1350 root cwd VDIR 64,0x6 15555584 2 /tmp
opcle 1351 root cwd VDIR 64,0x6 15555584 2 /tmp
opcmona 1353 root cwd VDIR 64,0x6 15555584 2 /tmp
opcmsgi 1355 root cwd VDIR 64,0x6 15555584 2 /tmp
coda 1359 root cwd VDIR 64,0x6 15555584 2 /tmp
sh 8023 root cwd VDIR 64,0x6 15555584 2 /tmp
lsof 12047 root cwd VDIR 64,0x6 15555584 2 /tmp
lsof 12048 root cwd VDIR 64,0x6 15555584 2 /tmp
files generting like .......AAAa10522 AAAa10478 eveery second.

any guess !!!!!!!!

[Alex Lavrov.]

I suggest you to bring down one by one this processes and see when it stops.

The only way to do it, if you don't know for sure what is it.

[Joseph Loo]

hi,

if process id, 10522 and 10478 r the only one, could u show output:

# ps -efx|grep 10522
# ps -efx|grep 10478

or

# lsof |grep 10522
# lsof |grep 10478


regards.

[Stephen Keane]

You could run top
If you have a process that is writing files to /tmp every second it should be fairly near the top of the list. Compare the PIDs of the top few proceses against those reported by lsof. This might give you a clue. As a last resort, you could always take write permissions away from /tmp, but I wouldn't recommend that on a production server and if the process that is writing files to /tmp is running as root it won't stop it in any case.

[Sarjerao]

there is no any such process

[Stephen Keane]

If you mean you can't find top, it should be in /usr/bin/top.

If you mean you can't find lsof, you have to download it, it doesn't come bundled with the operating system.

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.74/

If you didn't mean either of those, then you have me confused.

[Victor BERRIDGE]

Hi
Will you consider a reboot?
I you cant well I offer a almost suicide version of finding out whats going on:
at the console try to umount /tmp
It will not agree so fuser -ku /tmp
If that doesnt work fuser -ku /dev/vg00/lvol(forgot 6? )
umount /tmp
mount it again
Is that any better?
if not umount it again and do a newfs /dev/vg00/rlvol(forgot 6? )
These messy situations often occur when many people write to /tmp and remove files but with some still open...

Does this only happen when you do a ls in /tmp?


All the best
Victor