/var full due to OV backup?

[Priss]

I have a problem with /var constantly filling up so quickly. I noticed the solid.db file is significantly big, do I modify solid.ini to limit the size of this file? What does this file actually store? Also the files in the /var/opt/OV/tmp directory (ovbackup,ovbackup.log,ovcoltosql) seems to take up 13% or more of /var. I used fuser to find the processes that write to /var by root:

root 1472 1 0 Mar 5 ? 2:15 /usr/sbin/swagentd -r
root 583 1 0 Mar 5 ? 0:00 /usr/sbin/ptydaemon
root 1129 1 0 Mar 5 ? 0:11 sendmail: accepting connections on port 25
root 1263 1 0 Mar 5 ? 13:09 /opt/dce/sbin/rpcd
root 1410 1 0 Mar 5 ? 0:00 /usr/sbin/cron
root 1341 1 0 Mar 5 ? 5:29 /usr/sbin/pwgrd
root 1685 1665 0 Mar 5 ? 0:08 /usr/dt/bin/dtlogin
root 16852 16848 0 13:27:55 ? 0:17 ovdbcheck -ovspmd
root 16851 16848 0 13:27:55 ? 0:06 pmd
root 16929 1685 0 13:57:44 ? 0:00 /usr/dt/bin/dtlogin
root 16850 16848 0 13:27:55 ? 0:04 ovwdb -O
root 11555 1685 0 May 9 ? 0:00 /usr/dt/bin/dtlogin
root 23260 1685 0 May 11 ? 0:00 /usr/dt/bin/dtlogin
root 1430 1 0 Mar 5 ? 0:00 /usr/sbin/envd
root 1452 1 0 Mar 5 ? 0:04 /usr/bin/X11/SLSd_daemon
root 17725 1 0 May 14 ? 0:00 /usr/lbin/ntl_reader 0 1 1 1 1000 /var/adm/nettl /var/adm/co
root 16586 1 0 12:45:33 ? 0:00 /usr/sbin/syslogd -D
root 8198 8197 255 Apr 25 ? 33199:08 /opt/netscape/netscape http://unix1:8880/OvCgi/ovlaunch.exe
root 8199 8198 0 Apr 25 ? 0:00 (dns helper)
root 11555 1685 0 May 9 ? 0:00 /usr/dt/bin/dtlogin
root 23260 1685 0 May 11 ? 0:00 /usr/dt/bin/dtlogin

Daily backup cron:

00 23 * * * /opt/OV/bin/ovbackup.ovpl

00 1 * * * /usr/sbin/fbackup -f /dev/rmt/0m -u0g /var/adm/fbackupfiles/graph/fullbackup -I /var/adm/fbackupfiles/log/date"+%Y%m%d.log"

Can someone advice me on how to resolve this problem?

Many thanks..

Priss

[Priss]

Also:

bin 16854 16848 0 13:27:55 ? 0:06 ovrequestd -s
bin 16890 16854 255 13:30:00 ? 24:47 ovdwevent -v -export -trimdetail 35 -trimaggregates 12 -reEx

Priss

[Steven E. Protter]

It would appear you are doing omniback backups and or using other openview products that store temporary files in the location you are indicating.

If you are runing omniback or one of its successors, it keeps a dictionary of the files it backed up in the /var filesystem. If you are using omniback to back up the enterprise, you may find that database reaches several Gigabytes in size if not more.

You need to periodically trim those databases. We limited our omniback database to 30 days and had to run a script to trim it every coupld of days.

In addition, you need to identify which files are the largest.

If you don't trim them once in a while your /var/adm/wtmp and /var/adm/btmp file can get huge. Those two files track user logins correct and b for bad.

A good start that will probably get you some space really fast is thus:

sam
routine tasks
logs

Trim all logs down to their recommended size.

That will give you some breathing room.

Then its time to play detective and find out whats big. Its better to run regular cron jobs to back up large log files to other filesystems.

A wholesale delete of the btmp or wtmp file is a security violation, you need copies of these files in case someone tries to hack your system.

SEP

[Michael Steele_2]

I believe this is just the mount point of your web session. /opt/OV/www/cgi-bin/ovlaunch.exe should be its full path.

I would recommend using SAM to trim down this logs.

SAM > Routine Tasks > view and trim log files

Also use 'cleanup -t' to reduce /var/adm/sw.

Here are some other tools:

quot /var (* list of owners - good for finding owners that shouldn't be using /var *)

[Priss]

Thanks for your reply Steven, I shall check the things you recommended when I am back to work. I am new to all this so you have to bare with me :)

Do you know where I can get documentations/guidelines of how to trim the files and what files I should watch out for? I have manually zero sized some of the log files, they are not even that big. The huge files I found are the ones in /var/opt/OV/tmp, I am planning to symbollically link this directory to another file system. Another file is solid.db - can I reduce the size of it by editing solid.ini? What does it actually store?

[Priss]

Cheers Steven, didn't see your 2nd message... I will try that when I am back to work tomorrow.

[Steven E. Protter · Accepted solution]

Good luck to you, but that was Michael Steele on message #2.

We're going down the same path to get you some /var space quick, but he seems to have an additional idea.

Please post system memory and swap space and the output of this command.

bdf /var

You might need to expand the /var filesystem. I've been generally setting it pretty high on new systems. Disk is cheap.

If you don't have Online JFS to increase /var you must boot into single user or lvm mode.

shutdown -ry now

interupt at the 10 second prompt

bo

Y(es) Interact with the IPL
hpux -is or hpux -lm

Then use lvextend to extend the volume.

If you want it to be 2 Gig then.

lvextend -L 2048 /dev/vg00/lvol8 /dev/dsk/c#t#d0

make the lvol one the one you see associated with /var on the bdf command above and the disk a real one with some space.

extendfs -F vxfs /dev/vg00/rlvol8

then restart again to benefit from the increase space.

This is NOT a job for sam.

Good Luck,

SEP

[echong]

The solid database is actually used by NNM for data warehouse. Check NNM document "Reporting and Data Analysis" on Openview's web site for details. You can control the size of it, but you also need to adjust the trimming in order to keep it from filling up.

You can also change the staging area of your ovbackup.ovpl script to use a different directory/volume when doing NNM backup.

[Priss]

Thanks Steven and Eric, I have already been looking into some of your suggestions :) Shame I don't have access to the box from home so I can't do much now :/

For temporary fix, I have stopped snmpCollect and symbollicaly linked /var/opt/OV/tmp into another file system. /var is not "growing" rapidly now.