Disk space full problem

[Tejas Lagvankar]

Hello,

I'm working on a HPUX system (B.11.11 U 9000/800).
Due to some reasons our /var volume has become full. I would like to know what happens when a process tries to write on to the disk. Are there any system level parameters that define this behaviour. The reason i'm asking is that if i try to write a file using:

bdf output:
----------->
/dev/vg00/lvol8 4194304 4194304 0 100% /var

-bash-2.05b# pwd
/var
-bash-2.05b# echo 123 > foo
I get the following message:
msgcnt 562 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 block extent)

But if one of our application running on the machine tries to write some files... only zero byte file are created and the data is LOST !! :(
No alarm or above such message is seen.

Can Anyone please explain?

Regards,
Tejas

[Jaime Bolanos Rojas.]

Tejas,

For sure the first thing that you want to do is check what filled up the /var directory and then free up that space.

This is a good link on how to find what filled it up and how to fixed it:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=963344&admit=-682735245+1177597894751+28353475

Regards,

Jaime.

[Piergiacomo Perini]

Hi Tejas (welcome to the hp forums),

maybe why these application don't trap that kind of error messages.

my 2 cent
best regards
pg
and as Peter Godron says

Please also read:
http://forums1.itrc.hp.com/service/forums/helptips.do?#33
on how to reward any useful answers given to your questions.

[Marco A.]

Determine what is /dev/vg00/lvol8, and then you can use the following procedure to fix that, in your case is /var/something....

You can use one of two methods to resolve your problem:

Method 1:

A. Backup the /var/XXX filesystem
B. Execute the newfs command on the /var/XXX filesystem.

C. Restore the data from the /var/XXX filesystem.

Note: Method 1 is the recommended procedure. This method is faster
and easier.

--OR--

Method 2:

A. Add more free contiguous space to the /var/XXX filesystem.

B. Defragment the /var/XXX filesystem.

Hope this helps,

Rgds

Note: Probably you will see a lot of those errors is the syslog.

[spex]

Welcome to the forums, Tejas,

>I would like to know what happens when a
>process tries to write on to the disk. Are
>there any system level parameters that define
>this behaviour.

This behavior depends on the application. Note that a full filesystem will most likely trigger some sort of error condition in the process--not a situation you want to be in.

As others have already said, you need to find out what has caused the /var filesystem to fill up. Here are some commands that will come in handy...

To view the 20 largest directories under /var:
$ find /var -type d -print | xargs du -skx | sort -rn -k1 | head -20
To view the 20 largest files under /var:
$ find /var -type f -print | xargs du -k | sort -rn -k1 | head -20
To view files modified (this includes file creation) within the last 24 hours:
$ find /var -type f -mtime -1 -print | xargs ll

PCS

[Bill Hassell]

> what happens when a process tries to write on to the disk...

The program gets a standard Unix signal ENOSPC, or errno 28. What the program does with this signal is completely out of your hands. Sadly, the vast majority of badly written programs crash, or ignore the error and continue trashing files without comment. Your programmer needs to go back to the code and rewrite the lousy code so it correctly handles error returns from the OS, provides error messages and logs them properly in a well-defined log such as syslog. This is true for any program on any computer.

On the other hand, there is no such thing as an large-enough disk. No matter how big you make /var, it will never be large enough to ignore. All computer systems require hands-on management of disk space and for Unix systems, /var is one of the most common mountpoints to fill up. You must monitor the space at least every 10 minutes to make sure /var (and all the other filesystems) have plenty of space. When you see a filesystem approaching 90%, it's time to analyze the mountpoint to see where the space is allocated. DO NOT look for big files until you have found the biggest directories. Use this command:

du -kx /var | sort -rn | head -20

Then cd to the largest directories and see what is taking all the space in that directory:

ll | sort -rnk5 | head -20

An important command to use regularly is bdf to monitor your system. Another command is crontab which allows you to run a script monitor disk space automatically.

I have attached a script which you can configure to monitor all of your mountpoints and notify you by syslog, by email or by HP-OV (or all 3). Just create the config file mentioned in the script and then add it to cron.

[Tejas Lagvankar]

Hey Folks,
Thank you all for your instant replies !!
I think i have not quoted my problem statement precisely...

Well actually I'm trying to replicate a situation that is happening at our clients end.
Ours is a distributed application. A process on the HP-UX machine gathers records from various real time devices on the network. What is happening at the clients end is, that the /var partition of HP-UX is getting full and after that the process does't gather any records from these devices which further causes the data to be queued up on these devices.
I tried to replicated this problem at our test labs. I deliberately filled up the /var using "dd" upto 99%. and then forced these network devices to send data to the process running on HP-UX. But the process still continues to gather records from these devices even if the disk is full. It's normal behaviour is that it will keep these records in memory till a threshold is reached and then write it to disk.
The main difference that i found out at my end and the clients end is that when the disk is full and the process tries to write the records (when the threshold is reached), at clients end the write is not allowed, but in my test lab this data is just discarded (I was expecting that this write wont be allowed which will force the process to not accept any more records from devices on network which will result in queuing of records on devices) !!
I cannot understand how the same process is behaving differently on two same configuration HP-UX systems !! That's y i thought may be there are some system level parameters that govern the way the system will respond when a process tries to write on to a file system which is full...

I think, by now i might have successfully confused every one...

Regards,
Tejas

[Tejas Lagvankar]

Hi all,

Anyone wishes to comment any more? No responses for quite a long time now.
Otherwise i'll close the thread. :(

Regards,
Tejas

[MarkSyder]

Check the size of the following files in /var/adm:

btmp
sulog
wtmp
cron/OLDlog and log
lp/lodlog and log
syslog/mail.log and OLDsyslog.log and syslog.log

DO NOT delete any of these files. If btmp, sulog, or wtmp are very large, null them:

> btmp

Old logs can be nulled and current logs can be trimmed via SAM.

Mark Syder(like the drink but spelt different)

[Daniel Anthony]

Hi,

The reason why you are not able to write to the /var file system is because there is no space in the file system. The reason why zero byte file is created by the application is that the number of inodes has not yet reached its maximum limit. Once the file system utilization comes down, the files of appropriate size will be created. In this case I guess only the application will have to throw an error.