The size of /var directory increases too fast

[Leah Chow]

I have a HP 9000 server in my company, the size of /var directory was almost 95% full, I trimed and rotated some of the log files, and also did clean up the patches, the size has dropped, but it still increases everyday (at least 1%). What else do I need to do? Which subdirectory make the size increasing?

Also which patch I should download to make system remove the /var/rbootd files directly?

Thanks
Leah

[Tomek Gryszkiewicz]

I think it will be /var/adm, where logs are stored.
Go to /var and do du -sk * to see which one is biggest

-Tomek

[Pete Randall]

Leah,

All kinds of logs get written to /var on a daily basis. This alone may account for your 1% daily growth. I would suggest you monitor over a period of several days to see which subdirectory is experiencing the growth. Use "du -sk /var/* |sort -n". Put the du command in a cron job or just remember to run it yourself at a consistent time. By the end of the week you should have a good idea where your growth is occurring and you can then figure out how to best address it.


Pete

[T G Manikandan]

Just do a
#du -k|sort -nr|more

check for the large space consuming directories.

Revert

[Leah Chow]

Thank you all, it is really helpful. After I run the command du -sk, I found out the /var/stm/logs/os directory has many *.raw files, I deleted most of them, the size of this directory is very small now. But I still have one question: is there any patches I can install, then system can clean up this directory and /var/rbootd directory automatically?

Thanks
Leah

[Umapathy S]

you can run a cron job every hour like
find . -xdev -size +2048 -print

collect the details over a period of days to find out which file is increasing in size.

Then run lsof on the file to find out which process is causing this.

HTH,
Umapathy

[Robert-Jan Goossens]

Hi Leah,

No not a patch, but take a look at at next doc.

http://www5.itrc.hp.com/service/cki/search.do?category=c0&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=%22%2Fvar%2Frbootd%22&search.y=9&search.x=26&mode=text&searchCrit=allwords&admit=552267591+1066744847869+28353475

Document id: 8606199091

Hope it helps,
Robert-Jan

[Bill Hassell]

/var is the busiest and most critical directory for HP-UX. There are dozens of ways for /var to increase. In this case, the STM program is writing a lot of files which may indicate that you have serious hardware errors. Look at /var/adm/syslog/syslog.log and if you see a hardware errors, you'll need to get them resolved.

As far as /var goes, you'll need to trim the files based on size and keep compressed archives depending on your operational needs. The du command will help you the most in locating busy areas:

du -kx /var | sort -rn

[T G Manikandan · Accepted solution]

Make sure you do not delete the file which ends with raw.cur

please view the log files for any error information.
It would also fill up quickly when there are any errors.

Also make sure that you patch the STM with the latest patches


Also check this

http://forums1.itrc.hp.com/service/forums/parseCurl.do?CURL=%2Fcm%2FQuestionAnswer%2F1%2C%2C0x90617bb04b5cd611abdb0090277a778c%2C00.html&admit=716493758+1066745438166+28353475

[Todd McDaniel_1]

Also regarding /var, If you run Glance, /var/opt/perf/datafiles will fill up quickly...

You may need to run the "utility" command while measureware is down and shrink some of the datafiles. I know one of my parms was set to gather data for 3000 days! way to long. I dont need data for 9+ years.

On a few systems where I keep quite a bit of data, I have broken it off to its own filesystem so that it wont fill up /var.

_________________
One other thing, If you havent committed patches in a while run:

cleanup -p -c 2

And see what uncommitted patches you still have. This may cleanup quite a bit of space.

[Robert-Jan Goossens]

Hi Leah,

Document description: Temporary files C*.reply in /var/rbootd should not be stored
Document id: 8606199091
Product name: VInternet Services
Product number: B1030A
Release id:
System model: 9000
Last update filed: 10/12/03
Status: In Progress

CR# JAGad68278
problem
rbootd is detecting certain bootp reply broadcasts and storing the
header in temporary files in /var/rbootd/C*.reply.

These C*.reply files are ineligible for rbootd clean up. Consequently
they may accumulate in the /var/rbootd directory until removed by rm or
the system is rebooted.

Fix Text

fix
The fix is available in release 11.23. rbootd is not patched often.
Last patch was 1998 (before this fix was done).

Regards,
Robert-Jan

[Leah Chow]

After I checked /var/adm/syslog/syslog.log file, I found out the server has the same messages everyday, here is a copy:

Oct 3 14:04:46 ERP1 vmunix:
Oct 3 14:04:46 ERP1 vmunix: SCSI: Read error -- dev: b 31 0x064000, errno: 126, resid: 2048,
Oct 3 14:04:46 ERP1 vmunix: blkno: 8, sectno: 16, offset: 8192, bcount: 2048.

what does it mean? Do I need to correct this error?

Thanks
Leah

[I.Delic]

Hi,
You have a bad disk or controller.
Back your system up and call hardware support.
To check all of your disk use
ioscan -funC disk ( all shuld be claimd)

Use stm of xstm( gui interface) to see witch disk i damaged (view log)

Use vgdisplay -v vg00 (name of your volumegroup) to see if your disk are available.

Hopefully your system is mirrored.


succes

Idriz

[A. Clay Stephenson]

Disk /dev/dsk (or rdsk)/c6t4d0 is having problems and needs to be replaced.

[I.Delic]

extra info to read your error for the next time

dev: b 31 0x064000,

b blockdevice
31 major number ( kernel driver)
06 is bus
4 scsi id
0 lun nummer
00 fleg

/dev/c6t4d0 is damaged


succes

Idriz

[Dan Copeland]

check your /var/stm/logs/sys directory. The log may be loging excesively.

also try this

find /var -xdev -size +10000 -exec ll -ld {} \;

above would give the big files. You may reduce the value i.s +10000

[Todd McDaniel_1]

the easiest way to find the bad disk is to do this:

cd /dev
ll |grep 0x064000

This will give you the disk.

Then run ioscan -fnC disk > ioscan.out and get the H/W path so you can replace the faulty disk.

Of course run these commands to see if the disk is really bad.

diskinfo /dev/dsk/cXtXdX
pvdisplay -v /dev/dsk/cXtXdX ## should be stale extents usually if the disk is bad...

Then do lvdisplay on the LVols that are associated with that disk and see if any stale extents exist on any of them as well...

Hopefully you have mirroring and it will be fairly easy to do an online replacement.

IF it is a mirrored disk and NON-ROOT Volume, Replace the disk, then::

_______________________________
vgcfgrestore -n /dev/vgXX /dev/rdsk/cXtXdX
vgchange -a y /dev/vgXX
vgsync /dev/vgXX

[Tania Ap. Serpa Martins]

Leah,

Check if /var/stm/logs/sys/activity_log file is increasing, we had the same problem. When the system reboot the file start increasing until I issue mstm and exit the Diagnostic Menu.

Maybe you need to apply the patch for the Diagnostic online.

Tania

[Leah Chow]

After I replace the damaged mirrored dks, and non-root volume, what do I need to do?

[Brian Markus]

There is a doc called "help my file system is full" or something on that line, that was a great help when I was having that problem back on HPUX 9.0. I know the doc has been updated to 10.20 then 11.00. It has tons of tricks and tips to keep a file system clean. Personally I keep var at 2 - 4 gig's on most of my systems. Even when kept trim if you have a lot of software on a system that is very active; it can fill quickly. Disk is a cheap and easy band-aid to your problem. If you check, and the files that grow are normal, you may need to increase the disk. On one of my systems, I have a disk dedicated for the "space-monger" logs that I have to keep around. I have a cron job that tar/gz's the logs after a while, then spools them off to tape automatically on a few of my systems. I hope this helps. If you can't find that document I referred to, contact me, and I'd be glad to dig it up for you.

-Brian.