Root File system getting full -

[Richard_115]

Hi All
my root file system keeps getting full. I dont seem to have any cause. No files that have filled up this system. I am forced to reboot to clear it. and it reduces to as low as 35%. Does anybody out there have a clue as to what it is and what I can do ?

ooh, I dont have online JFS to extend it.

Suggestions welcome.

Thanks
Rich...

[James R. Ferguson · Accepted solution]

Hi Richard:

From your description it sounds like you have a process that is opending a temporary file(s) in the root directory and immediately removing the file while contining to write to it. This is a very common technique for handling temporary files.

The easiest way to identify the process doing this is to use 'lsof' (available from the Porting Center):

http://hpux.cs.utah.edu/

If you do:

# lsof +D / +L1

Look for an 'NLINK' count of zero (0). These represent open files that will vanish as soon as the last process using them terminates.

Regards!

...JRF...

[A. Clay Stephenson]

This has all the markings of an unlinked temporary file but there is something else to check for. Is /tmp actually mounted? For example, if /tmp were not mounted then everything under the normal /tmp mountpoint would go in /. This means that you could have unlinked temporary files that should be in /tmp actually in /.

You should also note that OnlineJFS would not help you in this case because / must be housed in a contiguously allocated LVOL.

[Richard_115]

Clay and James,

Yes I have /tmp file system mounted. Files underneath are not linked.

James, Do I have to dowmload this lsof from some website ? I dont seem t have it installed on any of my systems. Does it cause a reboot ? I cannot risk since this is a production server !

HELP....

[Bill Hassell]

And just some notes about the / filesystem:

- You can't extend / (or /stand or the primary swap lvol -- these must remain contiguous)

- You don't want to extend / -- it is supposed to remain staic in size

- You probably have root's $HOME in / -- a very bad place (and unfortunately common on almost all Unix flavors). Move root's $HOME to a separate directory so it is easier to manage (/root is fine or /home/root if desired)

- search for bad files in /dev:

find /dev -type f -exec ll {} +

(no regular files in /dev - common errors include om (should be 0m) and /dev/null2)

- No application directories in / -- EVER. They belong in /opt for installs, /var/tmp for temp files and separate lvols for /data-etc

[James R. Ferguson]

Hi (again) Richard:

You can download a binary depot from here:

http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.77/

Regards!

...JRF...

[A. Clay Stephenson]

... and no, a swinstall of /lsof does not trigger a reboot. Lsof is one of those utilities that no UNIX box should be without.

[Richard_115]

Thanks all....
Have installed the lsof .... I am evaluating the results.... will keep u updated...

[Richard_115]

James

I have looked at the output from the lsof command. Now it appears that the NLINK number zero is associated with a given process, which actually is very vital. I cannot kill the process ! In a sense, what I am looking at is " what causes this problem" and how can I stop it from happening. Killing the processes is tantamount to rebooting the system ! Whci case I didnt want to do. Any advise as to the cause and what to do to fix it ?????


Thanks

Rich...

[James R. Ferguson]

Hi (again) Richard:

> it appears that the NLINK number zero is associated with a given process, which actually is very vital

...and so, what is the name of that process? If this is a *local* application, then you probably need to look at where and how that application creates its temporary file(s).

Regards!

...JRF...

[Hein van den Heuvel]

>> Now it appears that the NLINK number zero is associated with a given process, which actually is very vital.

And the file size is relevant? (ls -li )

Sounds like an ill-behaving or ill-used application. Maybe there is a environment variable or .init or .rc setting to redirect its work files?

If this is not a homegrown application but a more or less standard package then you may want to
- share the name with us as some reader might have prior expeirences and resolutions
- contact the vendor for support!

hth,
Hein van den Heuvel

[A. Clay Stephenson]

Since I left my secret decoder ring at home and "The Force" isn't very strong today, it would help to know the name of this vital process --- and is this some application that you are running?

If so, the "fix" may very well be to start the application so that it's CWD is not in the root filesystem. I think this is some "home-grown" or 3rd-party application because well-written applications would observe the TMPDIR convention for temporary files. Of course, it is also possible that this is a well-written application that has been out-bushwhacked by setting TMPDIR=/.