Filesystem full, but no files in it?

[Marius Botha]

I check the FS space, use bdf:

/dev/vg00/debuglogs 524288 451776 72512 86% /usr/aethos/snss/logs

Now I check the FS detail, with du:
/usr/aethos/snss/logs>du -sk *
225 CFDSA
2 checkPoint.log
8038 debug
0 lost+found
0 snmp
0 src
0 syslog

In the debug directory on this FS there are some files, but my du calculations dont match what bdf shows .. please advice

[Pete Randall]

Marius,

Unmount the filesystem and then do a 'll' to see if there's stuff underneath the mount point that's using up all the space.

Pete

[harry d brown jr]

Is it possible that a file was deleted but that an application still has it opened?

get a copy of "lsof":

http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.64/

live free or die
harry

[Chris Wilshaw]

fuser /dev/vg00/debuglogs will show any processes using this filesystem.

You should be able to reclaim the space by killing off the process(es) that is(are) using it.

Chris

[Steven Buschman_3]

You could have a large "hidden" file

`du -sk *` doesn't catch hidden files.
do a `ls -a` to check if there are any , or do

du -sk .[a-z]* .{A-Z]* .0* .[1-9]*

You can't do a ./.* because of . and .. dir's.

[Marius Botha]

Hi,

I cannot unmount the filesystem, it is continually written to. I used fsadm to defrag it, still no change. I did fuser -cu and a couple of users have procs in it, but as said cannot kill any of them. Checkmate?

[V. V. Ravi Kumar_1]

hi,

just run du -ks one step obove /usr/aethos/snss
and see how much logs is occupying.
also run du -ks on /usr/aethos/snss/logs/debug

regds

[Marius Botha]

Ravi,

That's in the question .. if you add the amount togethor ... same as du in a directory up .. 8265 for du -sk logs/

[MANOJ SRIVASTAVA]

Well yes check mate , this ahppens sometimes as there are proceesses which are using the mountpoint , all you need to do is to unmount and mount it again schedule so time off and since they are log file may be u can try sneaking it at some other place and remounting it abck


Manoj Srivastava

[S.K. Chan]

It would not make any difference with even after you have defragmented the FS. I think you got a situation where a process is still actively writting to a "accidently-deleted" file and that is a very common symptom for bdf and du mismatch. If you can use lsof which is much more superior than fuser to determine which process has a "deleted" open file. If killing that process does not make any difference, reboot is your only choice.

[V. V. Ravi Kumar_1]

hi,
i understood that, i observed in my system if parent and sub directories are mount points there is difference in du and bdf outputs. any way lsof is the best option, to see is there any process which is writing to the deleted file. iam attaching lsof shar file (unshar it to use - sh lsof64.shar).

regds

[Marius Botha]

Ravi,

Should I use lsof with any special switches when running it on the mounted FS?

[Marius Botha]

Ravi, et all;

Bugger... I need lsof for 64-bit kernel .. I am running N-4000's in a cluster here.

[V. V. Ravi Kumar_1]

hi,

lsof with no options gives entire list of all opened files, thereis no problems with the mounted file systems. but to cut down, u can use grep, if any process that writes to logs directory or grep /usr/aethos/snss/logs

regds

[Pete Randall]

Marius,

64 bit lsof is out there, I just can't remember where off the top of my head. When the search engine is fixed, you can do a boolean search on +"lsof" +"64 bit" and that should, eventually, lead you to it.

Pete

[V. V. Ravi Kumar_1]

hi,

it is for 64 bit kernel only, its working on my system. further i suggest run top and observe top processes and identify if u can, is there any process that is owned by some user and it is writing to logs directory, if it is a stray process just kill it.

regds

[Pete Randall]

Marius,

With no search engine it took a while, but I think this is the site where I got a 64 bit version of lsof:
ftp://coast.cs.purdue.edu/pub/tools/unix/sysutils/lsof/binaries/hpux/

Hope This Helps,
Pete

[doug hosking]

Given that this is used for log files, I'd bet that what has happened is that some person or cron job has tried to delete some large log file(s) while they are still being written to. A feature of the UNIX file system is that open files aren't really deleted in this case until the last user of a file closes it. So the file can continue to grow even after it has been deleted. It just doesn't have a visible name any more.

I'm not familiar with the applications you are running that would be writing to this log directory, but it is not uncommon for applications to close and reopen their log file(s) if you send them a SIGHUP signal. This might be enough to cause the file(s) in question to be closed so they can finish being deleted, without requiring the application to be stopped.

Caveat: I know nothing about your applications, so I can't say what else might happen if you send them a SIGHUP signal.
Please check the documentation for your application before trying this.