/var/spool Filesystem is full -

[VVS]

Hi all,

I accidentely opened the thread in the Linux now I opened the same in the HP-UX Forum.

We were facing problem of /var/spool file system. It was 100%. We checked all the directories and files uder this filesystem. We purged/removed some old log files. Still I was 100%.We checked /var/spool/mail and /var/spool/lp but nothing found there.

Then we found that there were 5 instances runnig of lpsched. So we kill 4 instatnces. Then we stop and start the lp service. Again 5 lpsched process were runnig.

In the top command, we found that "jfmerge" process running which was taking to much CPU usages. So we changed the nice value of that still all the process were taking high CPU usage. There were 5 instance runnig of this process. Please see the attached file. So in the end we killed all the process of "jfmerge" and suddenly the /var/spool size went upto 22%.

Please tells us what is the "jfmerge" process and why it was taking so much CPU usage. What is the reason for this problem.

What is the solution for the same?

Appreciate your early reply...

Thank you!!!

[Steven E. Protter · Accepted solution]

Shalom,

I think at the moment your system is working against you. The system is trying to write data to /var as fast as you clear the data.

It might be good to reboot the system at console and go into single user mode.

hpux -is at the ISL prompt (varies based on system type).

mount /usr
mount /var
after logging in.

Then conduct your cleaning operation knowing the system will not keep writing there.

Your system is in such a state due to the full /var file system. Processed need to write there often, when they can't try go into i/o wait or try over and over, causing problems with CPU consumption.

SEP

[SKR_1]

Is /var/spool is a separate mount point?

Thanks

SKR

[VVS]

Thanks Steven and SKR.
SKR:
/var/spool is a seperate mount point.

Steven:
What was the reason for lpsched running 5 times?
What is the jfmerge process?
The /var/spool filesystem is 2.5 GB and it is 25% used normally. But when it was full, we didnot found any new file created or any large file present. We checked all the directories and files and also calculate their size but it was not more than 700MB. So what is the reason that it was showing 100% full?

After rebooting in single user mode, what are cleaning operation need to be conduct?

Regards,
VVS

[Dennis Handly]

>We checked /var/spool/mail and /var/spool/lp but nothing found there.

Is the filesystem still full? These could be files that were removed but a process still has them open. A reboot or a kill of the process would clear those up.

>Please tells us what is the "jfmerge" process

It has a user jfadm. This appears to be some extra jetform? software you installed from Adobe. google shows it may have to do with creating PDF files?

>We checked all the directories and files and also calculate their size but it was not more than 700MB. So what is the reason that it was showing 100% full?

Have you checked the difference between bdf and du -kxs on the mount point?
If different, then it is the "removed" files as I mentioned above.

[James R. Ferguson]

Hi:

If, as Dennis says, you have open files (i.e. inuse by a process) that have been removed (unlinked), then you will not be able to see them with a simple 'ls', *but* any disk blocks used will be held and accounted for until the last process using the file terminates.

Use 'lsof' against the filesystem. Look for files with an "NLINK" count of zero (0). The "SIZE/OFF" field will be the size in characters consumed. Files with a link count of zero are those that have been unlinked (removed):

# lsof +L1 +D /var/spool

If you don't have 'lsof' installed (and you should) fetch it for instatllation from:

http://hpux.connect.org.uk/hppd/hpux/Sysadmin/lsof-4.80/

Regards!

...JRF...

[VVS]

Thanks Dennis and James,

Dennis:
The /var/spool filesystem is now 22%. When we killed the "jfmerge" process, suddenly the /var/spool filesystem went to 22% from 100%.

James:
We dont have "lsof" installed, so we need to install it.

But now are facing memory dump error.The o/p of swapinfo command is

[dev2]: / # swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 6652 303 6349 5% 0 - 1 /dev/vg00/lvol2
reserve - 6349 -6349
memory 3004 2903 101 97%
total 9656 9555 101 99% - 0 -

Also we found that in /var/spool/lp/request directory there were large files in one perticular printer-name directiory, which are still there.

regards,

VVS

[Dennis Handly]

>When we killed the "jfmerge" process, suddenly the /var/spool filesystem went to 22% from 100%.

Then there must have been some big files still open.

>But now are facing memory dump error. The o/p of swapinfo command is

What's the connection between the two?
But you are all out of swapspace. You need to add some device swap.

Your previous top output showed three d11adm processes with 5 Gb each.

>we found that in /var/spool/lp/request directory there were large files in one particular printer-name directory

Does it show up with lpstat? If it does, you might want to contact that user, otherwise remove it.

[VVS]

Thanks Dennis,

We are facing the same problem again. The "jfmerge" process was taking 99% of CPU usage. So again I killed that process.
Uptil now, I don't have any information on "jfmerge". Why this process was created? Why the process was taking so much CPU usage?

I attached the screen shot for your reference.

Waiting for your reply...

Regards,

VVS

[James R. Ferguson]

Hi (again):

It would appear that your 'jfmerge' process is/was provided by Accelio (now owned by Adobe). I suggest you contact that vendor for more information / support:

http://www.adobe.com/enterprise/accelio_jetform_notice.html

Google is a wonderful tool :-)

Regards!

...JRF...

[Steven E. Protter]

Shalom again,

The jfmerge process either needs to be removed from the system, or updated with a version that doesn't use 99% of the CPU.

It is possible but unlikely that the problem is triggered by OS problems. To eliminate that as a possibility, install the latest QPK or Gold pack for your OS.

SEP

[Dennis Handly]

>The "jfmerge" process was taking 99% of CPU usage. So again I killed that process.

It looks like you need to get support for that product.

>I don't have any information on "jfmerge". Why this process was created? Why the process was taking so much CPU usage?

You should have more info that we do, you bought the product. :-)

If you want to experiment, you could use tusc on that process to see what it's doing. Or use gdb to get a stack trace.

There are lots of orphaned processes.

You do have a 2 Gb file. Either that's the result of the loop. Or the fact that you don't have largefiles enabled and the particular output file needs to be that big. Or jfmerge isn't 64 bit file aware and it can't handle largefiles.

Note: If you need files that large, you don't have enough space for them.

[VVS]

Closing this thread.

Thanks all...