file system filling up

[bobstar]

hi,

i have a number of identical nodes with the same applications ruuning on them , so therefore, up until recently the files systems were roughly about the same size.

Recently on one node, the / has reached 94% used whilst in other nodes its only about 82%.

I have taken a look at / ( after ignoring other mounted file systems ) and I cant seem to find out what has 'eaten ' up this disk space.

Any ideas ?

Thanks.

[MarkSyder]

The most common reason for / unexpectedly filling is a mis-typed file name for a backup device. This results in a large text file in /dev. This text file can safely be removed.

Mark Syder (like the drink but spelt different)

[Darren Prior]

Another common one is having files 'under' the mountpoint directory. ie you have a mountpoint /blah and normally mount /dev/vgblah/lvblah to it. If you copy files to /blah before the mount command is run, then they will actually be on the root lvol. Once the proper lvol is mounted the files will disappear - but of course the space has still been used.

regards,

Darren.

[bobstar]

thanks for your responses

i have checked /dev and there is no large file

also, i dont think there is a problem with the mount points

what i suspect, is that some app or user/ users have not cleared up their unwanted files

id like to somehow do a sorted listing from / and compare it with another node ..

[RUET]

Do you have home directories in / filesystem ?
You can have core, or log files from users or users applications ..
Did you reboot this node ? Sometimes process can have open files on a filesystem, not completly flushed to disks, not seen by du, but taking space in the bdf ..

[Peter Godron]

Hi,

cd /
du -k | sort -rn | more

Also check for unexpected processes using the disk (fuser or lsof).

Please also read:
http://forums1.itrc.hp.com/service/forums/helptips.do?#33 on how to reward any useful answers given to your questions.

So far you have never awarded points !

[MarkSyder]

Check the newest files:

cd /
ll -tr

The newest files will be at the bottom.

Sort the files in size order:

ll|sort -k5,5nr|pg

The largest files will be at the top.

Mark

[bobstar]

hi

i have tryed to submite points in the past, maybe i need to double check things :)

i have seen some thing a little odd

on the node where the filesystem has filled upp for / the /tmp dir size is *10 bigger than on another node. why is the dir size so much bigger ?

also, if the /tmp is mounted on a diffrent lvol then this should not explain why / has filled up ?

[Peter Godron]

Hi,
as long as /tmp is mounted seperately it will not count towards the / size.

Something else you could try is:
find / -size +1000000c -xdev -exec ll {} \; | grep 2006

This will find all entries with more than 1000000 characters (approx 1Mb) on the root partition. The grep at the end does a very crude elimination of old file. You can try to change the 2006 to 2007 for this year.

[Ganesh Balraman]

Dear Sir,

It could one of the reasons

1.Run lsof and look for fs mismatch.

2./dev directory for large files.

3.du on each main dir to find out large files.

4.core or crash in / dir.

Definetly above one could solve.

Regds
B.Ganesh

[bobstar]

whats lsof ?

[Wouter Jagers]

lsof stands for "list open files". It's a tool which can show you which files (and other things) are opened by processes.

It doesn't come with HPUX, but you can get it at the porting and archiving centre:

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.77/

Cheers
Wout

[Wouter Jagers]

Also, you write:
"up until recently the files systems were roughly about the same size".

Should "recently" be recently enough, you could use the -mtime option to find, to search for recently changed files.

This would look for files which were modified in the last week:

find / -mtime -7 -xdev -exec ll {} \;

You can combine this with size restrictions as described above.

Good luck !

[bobstar]

hi,

find / -size +1000000c -xdev -exec ll {} \; | grep 2006

gives all files (included those mounted on another lvol) ,. 1000000c came back with nothing , 100000c came back with loads but again the info is for all lvols

recently means within the last couple of months its gone over the threshhold of 90%, it was at 94% , but i mamaged to delete some old app logs

im going to try lsof, is there something specific to look for with this exe

[bobstar]

i have ran the lsof exe

and there is alot of info

is there some specific i can look (grep) for ?

thanks

[Ganesh Balraman]

Dear Sir ,

Pls see the attach which we had solved thru lsof.

losf with options can solve ur issue.

Regds
B.Ganesh

[James R. Ferguson]

Hi:

A file that is deleted ('unlink'ed) but still in use by one or more processes does not have its disk space freed until the last process using it terminates. This can cause you to see higher disk space utilization with utilities like 'bdf' than can be accounted for by summation of the sizes of files.

The 'lsof' utility can assist in finding files with link counts of zero (0). If you have instances of these, they will be ones that are still in use by a process and are thus holding disk space that will ultimately be freed. Do:

# lsof +L1

This will list files with a link count (NLINK) less than one. The process pid will allow you to find the process using the file.

Regards!

...JRF...

[bobstar]

hi,

ok, i may have found where the problems lies, but what next ?

command used = lsof +L1

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
syslogd 636 root 13u REG 64,0x3 35656850 0 3491 / (/dev/vg00/lvol3)

[Wouter Jagers]

You could try to kill -1 (restart) the syslogd process, or stop and start it with the init script in /sbin/init.d

Cheers

[Peter Godron]

Hi,
if the problem is your syslogd, check /etc/syslog.conf for where it is logging to, as normally the syslog goes to /var/adm/syslog/syslog.log.

Stop and restart syslogd after checking the log location.

As I mentioned in my first post both fuser and lsof could be used to identify the processes.

Your feedback rating is still 0 for 26 answers to 6 questions!

[bobstar]

the syslogd was restated and file system is down to 74 % :)


i clicked the reward points button and go the follwing message , what am i doing wrong (as i was not given the option to chose a reward value)


Thank you for taking the time to provide feedback on one of the responses you have received. Your review and rating has had a two-fold benefit:

It has improved the knowledge that is being shared among members of the Forum
It has given credit to those who took time to help solve your problem.

Thanks again for completing this valuable feedback. HP and the IT community appreciate it!

[Peter Godron]

Hi,
agianst each answer there should be a field with drop-down values 0 to 10.

If you review EACH of the answers and see how useful they were in working out what the problem and and how to fix it, this will help future viewers of your thread to quickly trace the path to the solution.

Finally summarise the solution and submit.

See http://forums1.itrc.hp.com/service/forums/helptips.do?#28 onwards