System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

/var increased to 100% and reduced to 81%

SOLVED
Go to solution
Ajin_1
Valued Contributor

/var increased to 100% and reduced to 81%

Hi Experts

 

In our rp8400 Server ,yersterday we face ome Problem /var file system incresed to 100 . by step by step .Then it was decresed .To sheduled scrips are runing at that time like house keeping ,How to i found the RCA for this ...which logs i have to check for this.Kindly Help.

Thanks & Regards
Ajin.S
Proverbs 3:5,6 Trust in the Lord with all your heart and lean not on your own understanding; in all your ways acknowledge him, and he will make all your paths straight.
7 REPLIES
Patrick Wallek
Honored Contributor

Re: /var increased to 100% and reduced to 81%

If you know the two scripts that were running then have a look at those scripts and see if they write to the /var filesystem.

 

At this point it is somewhat difficult to know what caused /var to fill up since we can't see exactly what processses were running and what files they had open.

Ajin_1
Valued Contributor

Re: /var increased to 100% and reduced to 81%

Hi Patrick Wallek  

Thanks for your kind reply .Those two scripts are only checking some preocess status . For this any other possibilities like ,which users are logged in those server at that time duration ,how to find . kindly suggest.


Patrick Wallek wrote:

If you know the two scripts that were running then have a look at those scripts and see if they write to the /var filesystem.

 

At this point it is somewhat difficult to know what caused /var to fill up since we can't see exactly what processses were running and what files they had open.



 

Thanks & Regards
Ajin.S
Proverbs 3:5,6 Trust in the Lord with all your heart and lean not on your own understanding; in all your ways acknowledge him, and he will make all your paths straight.
Patrick Wallek
Honored Contributor

Re: /var increased to 100% and reduced to 81%

You can use the command 'last' to see what users were logged in at the time the file system filled.

RAJD1
Valued Contributor

Re: /var increased to 100% and reduced to 81%

Ajin,

 

If you have taken some system output duting /var was 100% or reaching towards 100% you can pin point what was causing /var to 100% , or what was wrong. If you have taken lsof output and bdf output during the incident, you could be able to find out what process was using the /var space on the fly. And it seems the moment that process finished /var space released to its normal.

 

Please post bdf and  # lsof > lsof.txt , if you have during that time.  

 

Hth,

Raj.

Ajin_1
Valued Contributor

Re: /var increased to 100% and reduced to 81%

HI Raj

Kindly find the bdf o/p

 

$ bdf
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol3    2097152  331328 1752048   16% /
/dev/vg00/lvol1    1014648   86968  826208   10% /stand
/dev/vg00/lvol13   14942208 11110497 3592287   76% /work-sysmgmt
/dev/vg00/lvol8    8912896 8891632   21264  100% /var
/dev/vg00/lvol9    4194304 2359913 1719768   58% /var/adm/sw
/dev/vg00/lvol10   13631488   19877 12760957    0% /var/adm/crash

Thanks & Regards
Ajin.S
Proverbs 3:5,6 Trust in the Lord with all your heart and lean not on your own understanding; in all your ways acknowledge him, and he will make all your paths straight.
RAJD1
Valued Contributor
Solution

Re: /var increased to 100% and reduced to 81%

Ajin,

 

It is showing that /var is using on its top level , and become 100% full.

 

Filesystem          kbytes       used          avail      %used   Mounted on

dev/vg00/lvol8    8912896   8891632   21264  100%       /var

-------------------------------------------------------------------------------------------

 

You have 8.8GB used   out of 8.9 GB ,   so it seems nothing wrong, but you have to do some cleanup and to recover the free space on /var.

 

 

You can check large directories inside it occupying most space . And you can have some idea what has increased suddenky  or what is using  more space since it was showing normal usage:

# cd /var

# du -sk * | sort -rn  | more

 

 

Hth,

Raj.

Dennis Handly
Acclaimed Contributor

Re: /var increased to 100% and reduced to 81%

>/dev/vg00/lvol9    4194304 2359913 1719768   58% /var/adm/sw
>/dev/vg00/lvol10   13631488   19877 12760957    0% /var/adm/crash

 

At least these typically big filesystems are separate so you don't need to do the standard patch cleanup(1m) and crash dump cleanup.

 

What's left could be cron, mail or user tmp files.