Full file systems that crash a system.

[someone_4]

I had one of my servers crash yesterday morning. I had some crash files but there was not enough space on var to get a good dumb.We looked at other files and came to the conclusion that my server crashed because /dev/vg00/lvol4 other wise known as /tmp got full. I didnt realize that /tmp was so important or that it could crash a system. can a full /tmp dir cause a system panic? if so why? and what other directories are space sensitive.

Thanks
richard

[James R. Ferguson]

Hi Richard:

In the pure sense, /tmp is considered to be root's place to put files it needs. Correspondingly, /var/tmp is for users. That said, I suspect we all abuse this distinction.

Full filesystems can cripple a server.

Regards!

...JRF...

[Jeff Machols]

what could have happened is an import file for the kernel was being written and the file system filled up before the file closed. You have had partial data in a file that corrupted the kernel when it tried to read it back in. I don't think everytime /tmp fills a system will crash, but if you timed it right when the system needed it, this could happen

[Helen French]

Hi Richard,

The server may crash if /tmp is full and is not necessary. When the "/tmp" file system is full, the system can not be expected to behave in a predictable manner.

It is important to ensure that all the root file systems have enough
space, especially /var, /tmp, and /.

HTH,
Shiju

[John Palmer]

Personally, I can't remember a full filesystem actually crashing a HP-UX server. Could /tmp being full be an effect rather than the cause?

The most likely to cause wide ranging problems though are / and /var.

Any filesystem full condition will affect those processes that are trying to write to it so as always, it depends...

Regards,
John

[Rita C Workman]

Filling up a file system will 'generally' not crash a system. It will hang it up - but you should be able to go in and find the culprit and clean it up.
If your system panic'd, and wrote a crash it hit /var..you might want to come up in single user and mount your filesystems individually.
It will come up in single user with only / mounted. Then mount /usr and run bdf...all okay then mount /var...
You should also go and check /var/tombstones and rule out that you didn't panic because of a hardware problem - and see if you got a crashdump at /var/adm/crash.

Because generally filling up /tmp will just cause the system to choke..not panic.

Just my 2cents,
Rita

[Steven Gillard_2]

No, a full /tmp filesystem should never cause a system panic. You might have a patching issue, or it could be a co-incidence. Its a real shame you didn't get a dump.

If it is possible to panic the system by filling up /tmp then thats a serious security flaw.

Regards,
Steve

[fg_1]

Richard

/tmp is critical, if it fills up it will panic the server as you have seen.

In terms of setting up to collect dump information, we always configure a seperate filesystem called /var/adm/crash which we make equivelent to 105% of the amt of physical memory that you have installed on your system.

This has been very useful in properly capturing and analyzing panics on system.

[A. Clay Stephenson]

Hi Richard:

Filling /tmp is a big no-no and can cause exactly the behavior you observed. In the old days both the user applications and the kernel/critical processes wrote to /tmp. Nowadays, the system stuff only is supposed to write to /tmp and everything else is supposed to use /var/tmp. If /var/tmp fills up it is merely inconvenient but if /tmp fills up there is trouble in River City.

Clay

[Sridhar Bhaskarla]

Richard,

/var (/var/tmp) and /tmp can create problems if are full but there is a very minute chance that they will cause the crashes. In my life, I haven't seen systems crashing due to file system full. But there are some indirect effects of this situation and look for the following document in the resource center HPUXERR02.

-Sri

[Krishna Prasad]

I have had /tmp fill up several times and it never caused a panic or a hang for that matter.

The same with /var and / . The worst I have seen is process falling because they could not write to destination neccassary.

But I have never had a crash because of /tmp filling up. Did you look at /var/tombtones/ts99

[Bill Hassell · Accepted solution]

As mentioned, filling /tmp won't cause a panic. HP-UX panics because something unexpected has occurred inside the kernel code and HP-UX uses no files in /tmp (except swap if filesystem swap was enabled in /tmp).

Most likely, the full filesystem error was recorded in syslog.log and that was the last activity shown. Always look at the syslog timestamp: was it the same exact time when the crash occurred? Remember that syslog cannot (and no other file anywhere) record anything about a crash. HP-UX detects a catastrophic error, and to avoid makng more mistakes, stops all processing, branches to doadump (do a dump) and turns control over to the processor ROMs to perform a crash dump.

So how can /etc/shutdownlog record that a panic occurred and even provide a reason (ie, freeing free frag)? By sleuthing through the raw crash dump. When the system reboots, the rc files are run and one of them is savecrash (savecore in 10.xx). If the savecrash program is able to successfully copy the dump area to the crash directory, it will do a quick adb of the files to grab the panic string and records that string in /etc/shutdownlog.

So the first thing to do is to get crashdump capability configured (typically, you'll need to create a new logical volume about 50% the size of RAM (11.xx, or 2xRAM for 10.xx). Then wait for the next crash. While you are waiting, peruse the man page for q4.

As far a system-critical filesystems, the following list in rough order of descending importance for full filesystems. /var is the most critical (almost all processes start failing):

/var /tmp / /usr /opt

Notice I did not include /home...if they fill up their directory, that's their problem. ;-)

In fact, you can make a much more resilient directory structure by creating the followig separate mountpoints:

/var
/var/spool
/var/mail
/var/adm
/var/adm/sw
/var/adm/crash

Now, subsystems that use these directories (like spooling or email) can fill their respective directories and the rest of the subsystems still continue to run.

[Dragan Krnic]

At least a part of the problem is self-inflicted by unnecessarily partitioning the system disc. The usual 9-18 GB disc today can swallow a lot of abuse before it blocks full but the event is called for by fixing any of the tmp, var, usr, opt, home, usually all of them, to its/their own portion of the disc. On my systems /tmp is usually a link to /var/tmp (a subdir of /, not a separate volume) which itself links or mounts the tmp subdir of a huge external subsystem, 70-320 GB, with all else. Sure it's a pain if you need to start it in single-user mode because you have to create the /var/tmp for anything useful to work in that state, but how often do you need to start as single-user?

On the other hand, I have nothing against lavish root VGs, where each partition coincides with a possibly mirrored distinct physical medium. It's those 2 and 4 GB disks cut into 8 LVs that bring tears to my eyes when they spawn a thread like this every now and then.

[Tan Yun Yew_1]

Hi,

Thought that I might bring this thread up again instead of creating a new one.

I just discovered that my /tmp is 95% full.

#bdf
...
/dev/vg00/lvol4 987136 930740 52909 95% /tmp
...

I'm afraid that it might cause the system to crash.

Are there any files that I can safely delete ?

Kindly advise.

regards.

[Michael Tully]

Hi,

It's not good practise to bring back an old posting. Please create and post a new one.

Michael

[Tim D Fulford]

Richard

if you use DCE (& I think Kerbros) Bill's advice is even more crucial as the authentocation is done there, so fill /var & no one can do any work!!!

Tam

(This really should be in a new thread....)

# find /tmp -atime +7 -exec ls -ld {} \;
Check this list, all files that have not been accessed in 7 days.

# find /tmp -atime +7 -exec rm {} \;
This deletes these files. You may need to go round & rmdir some of the directories by hand. do NOT use rm -rf in find cmd, as it could easily delete files that have been accessed less than 7 days.

Tim