How does HP-UX handle write system call and fputs function ?

[Vishal Augustine]

Hi,

This doubt is regarding the way in which HP-UX handles write system call and fputs function.

Is the following true for an append operation using fputs ...
1. open a Huge file in append mode using fopen to get a file descriptor (Min memory usage)
2. append a few bytes (Min or Max mem usage ... I am not sure)
3. fclose after appending. (Max memory usage)

Internally the kernel is allocating some buffer for reading the entire file and then adding the bytes given for appending and later writing it to the disk. Am I right ?

Does the same hold good for open/write/close sytem calls ?

What I am interested is in the memory and performance of the whole system. Which one makes use of min resource
1. appending a char to a huge file
2. appending a char to a small file
Or is it that both have the same overhead ?

Thanks and Regards
Vishal

[Carlos Fernandez Riera]

I think it is not so.


When you open a file for append, the system must use lseek function.

See man lseek, or fseek for use in fopen.

[A. Clay Stephenson]

Hi:

In your particular case, it really won't make much difference. The fundamental difference is that files open with fopen() have a buffer space associated with the process (in addition to the UNIX buffer cache so that most I/O operations take place in the buffer and then are transferred to disk in a few operations. Buffered I/O tends to be much more efficient.

I/O done at the file descriptor level using the open(), read(), and write() system calls depend upon the programmer to explicitly do the buffering.

In the case, where you open a file in append mode and write only a few bytes (hopefully with only one write() call) just isn't going to make much difference. The typical buffer sizes are only a few thousand bytes so memory usage usually is not a concern.

Now let's suppost that you want to write 10 bytes. The really dumb way would be to call write 10 times and write 1 byte each time. The smart way would be to call write once and write 10 bytes. If you instead, chose to use fput() or fwrite() and made 10 calls to write 1 bytes, you would not take the perfomance hit because the buffering is done automatically for you 'behind the scenes' and then the actual disk i/o is done either when 1) you fill the buffer, 2) you close the file, or 3) you cross a block boundary by issueing a seek.

Regards, Clay

[Vishal Augustine]

Thanks Clay,

But I guess in a scenario where almost 99 % of the memory is utilized (and 75% of the swap space), appending a few bytes (not calling fwrite or write multiple times but doing it in one shot)to a huge file - say 128 MB file - will affect the memory and CPU utilization. That too if the frequency of such append operation is high - say 20 times a second (open-append-close operation cycle). Instead if the file is broken into smaller chunks, and the same operation is performed on one of the smaller chunk, we can save CPU and memory utilization. Later, all the small chunks can be merged to get the single file. This will reduce swapping in and out, and will increase the performance. Am I right ?

Thanks and Regards
Vishal

[Carlos Fernandez Riera]

Check it yourself:


runing timex yourprogram you will see a report like this one:

real 0.03
user 0.00
sys 0.01


It tells you how may time have run you program in real time ,user and sys mode.

[A. Clay Stephenson · Accepted solution]

Hi:

If your memory utilization is that high and you are paging out constantly then we really don't need to be talking about performance. It's already really bad and the only real cure is more memory.

You seem to be confused over the size of the file; that makes no real difference for a file opened in append mode. It always has to lseek() to EOF before writing but that is not a memory intensive system call and is independent of the size of the file. Obviously anything you can do to reduce memory requirement will help and all you need to do is:
fdes = open("filename",O_RDWR | O_APPEND | O_CREATE,0660);
write(fdes,(void *) mystring,strlen(my_string));
close(fdes);

I would say that the worst thing you could do is to call a separate program when you are already under intense memory pressure.