Logical Volume Full and unable to free up space

[HoweIan]

Having an issue with a logical volume on HP UX 11.23
The output of df -k shows below
/var (/dev/vg00/lvol8 ) :
8912896 total allocated Kb
0 free allocated Kb
8912896 used allocated Kb

when i go into /var and do a du -ks i get the following
426368
I have connected to HPUX Server with a Linux box and it reports the /var directory having used 380MB approx.

I did remove a directory in /var/adm/sw/save but a minute later the disk was full again. There is something causing the volume to fill up and im not convinced it's data.

Any ideas on where i start to troubleshoot this?

Ian

[Adam W.]

If you have removed files, but the space is not opened you prolly have a process running that is not allowing the space to "free up"

Use lsof to find this.

Also may want to use cleanup -c 2 to remove patches that have been supercedes at least 2 times. This is incredibly useful for clearing out space.

For finding large files you may want one of the below:
1. find /var -xdev -depth -size +7000 â xdev -exec ll {} \; | more
(finds files larger than 7000k)
2. find / -type f -size +1000000c -xdev -exec ls -l {} \;
(if you want to search for a number of characters try the lower case c after the number)
OR
find / -xdev -size +1000000c -exec ll {} \;
(searches for directories as well as files.)


Or my favorite refence is touch.
To remove dated files. Can be modified to remove sized files as well.


# touch -amt 200504150000 /tmp/ref1
# touch -amt 200802282359 /tmp/ref2

# find /opt/mtmc/ibs/import/reject -xdev -type f -newer /tmp/ref1 -a ! -newer /tmp/ref2 -exec rm {} \+

This will remove files that are newer (more recently modified) than April 15 at 0000 AND NOT newer than Feb, 28th 2359 of this year

[Tim Nelson]

Removing /var/adm/sw/save was BAD.. No longer will you be able to rollback any bad patch. Bummer..

As you mindlessly delete, maybe important stuff, keep in mind that processes are activly writing to /var and some even might keep configuration information there. Removing a log file of an active process may not free up the disk space. You need to be selective about what you are deleting.

Certainly files with extentions like .log are prime candidates but typically processes need to be stopped / restarted or sig_hupped in order to restart their logging.

There is a whole list of HPUX applications that typically use /var.

I would start first by investigating your largests users. A du -sk * under /var is a good place to start. Cleanup of /var/adm/sw should only be done with the cleanup command.
Good candidates for review are:
/var/adm/syslog/syslog.log
/var/adm/wtmp*
/var/adm/btmp*
/var/mail
/var/tmp
/var/preserve
/var/sam/log
and more....


And as another option, depending on your environment, /var/ may just need to be bigger.

Caution is needed before you render your system unusable just to resolve some other issue.

[James R. Ferguson]

Hi Ian:

> I did remove a directory in /var/adm/sw/save

Yikes! Unless you know what you are doing, or (hopefully) used the 'cleanup' command, you may have ruined your server for any future patching!

Your symptoms sound like a process that has opened a temporary file; immediately unlinked (removed) it [a common technique]; and continued to write to the temporary file. When the last process writing to the file in question terminates, then the disk space consumed will be returned to the system.

You can use 'lsof' to see unlinked, open files:

# lsof +D /var +L1

Regards!

...JRF...

[Adam W.]

Howelan,
I hope you have a full system backup handy, cause installing new patches and/or rolling your server back is now impossible. If you have a full backup somewhere you need to push it immediately. THEN try to figure out why /var is full.

[HoweIan]

I don't have a backup of the /var/adm/sw/save directory for this server.
I did run cleanup -c 1 before i removed the contents of this directory. I hope it hasn't prevented me from rolling out new patches to this server.

Regarding the lsof command
This command is not recognised when i run it from the command line.
when i do a "find / -name" lsof i do not find this program.

[Dennis Handly]

>I don't have a backup of the /var/adm/sw/save directory for this server. I did run cleanup -c 1 before I removed the contents of this directory.

You won't be able to remove any patches. The purpose of cleanup is to recover space, there is no need to remove anything else.

>Regarding the lsof command

You can download it from the porting center:
http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.80/

[Bill Hassell]

Removing the sw/save directory may prevent from installing any more software, not just patches. swinstall uses the sw directory to understand what has already been installed. You should try installing a small package (like lsof) to see if it will work. If not, this server has a big problem and will probably have to be rebuilt if you don't have a backup.

In the future, for any mountpoint, NEVER look for big files and just delete them. A lot of new sysadmins have removed those giant files called vmunix and wondered why the server will not boot anymore. You want to find the big directories first. The command (for any directory or mountpoint) is:

du -kx /var | sort -rn | head -20

The directories at the top are the ones to look inside. To sort the files inside the directory, use:

ll /some_dir | sort -rnk5 | head -20

The fact that the directory immediately fills up after removing a lot of space means you have a runaway program. And one of the reasons not to look for big files is that this bad program may have created 50,000 small files, thus eating up your space. You can be assured that the disk is filling up with data. You just have to locate the junk file(s) and/or the program causing the mess.