increasing buf size increases execution time

[Ravi_8]

Hi,

Here is my C program

#include
#include

main()
{

char buff[2621440];

int fd;
int i;


fd=open("/dev/vg00/pacs4", O_CREAT|O_RDWR);

for (i=0; i<200; i++)
{
write(fd,buff,2621440);
}
close(fd);
}

/dev/vg00/pacs4 is a raw device, when the size of pacs4 is less (within 100MB) execution takes very less time(2 to 3 Minutes). when the size increased (say 500MB or 1000MB) execution time takes around 15Min.

whereas the same program on AIX/Solaris takes just a minute irrespective of pacs4 size.

why is so? is there any kernel parameters to be changed to keep execution time less irrespetive of pacs4 size?

thanx in advance
Ravi

[A. Clay Stephenson]

When dealing with large i/o like this, the system silently breaks the i/o into smaller chunks (256KB-1MB or so depending depending upon OS version). The speed also depends upon the underlying hardware.

I must say that it is prrobably not wise to open a file descriptor for a raw device with the O_CREAT flag. If the file exists, no harm, but otherwise you are going to open a regular file in the /dev directory and possibly fill up the / filesystem. It is also possible that you are not, in facxt, writing to a raw device as intended if you mispelled slightly or never created the node/LVOL.

[Dietmar Konermann · Accepted solution]

Ravi,

you're are talking about raw devices, but the file name "/dev/vg00/pacs4" looks more like a block special (buffered) device. In this case the buffercache could be responsible for the effect you're observing. Then you could perform the test again with "/dev/vg00/rpacs4".

Best regards...
Dietmar.

[Hein van den Heuvel]

As replied just now, buffers larger then 1MB or so rarely help. Going from 1KB to 8KB to 128KB one would expect nice rapid increase in speed/bandwitdh and reduction in IO /sec needed (glance at it!). Beyond that, towards 1MB one would expect a modest further speedup. You mileage _will_ vary based on IO infrastructure and LVM settings (notably striping/chunks).

As also hinted to before... you are sure that on the sun box you did not accidently end up going to the filesystem file huh?

The program is writing from 'buff' but as coded that starts at a random/uncontrolled address. The actuall IO will need to come from a page alligned intermediate buffer (best I know, correct me if I'm wrong). Check: man setbuf. It is that intermediate buffer that you really want to grow as your a large private buffer might just make more work for the write function to de-block into the io buffer.
Perhaps Sun does a magic re-allocate based on IO size?

Finally.. we all assume that as you increased the buffer size, you decreased the loop count. I woudl recomamdn hardcoding this in the program statically with defines, or dynamcially with arguments and variables.
Something like:
#define TOTAL_IO 100*1024*1024
#define BUFFER_SIZE 64*1024
#define LOOPCOUNT TOTAL_IO/BUFFER_SIZE
char buff[BUFFER_SIZE]
:
for (i=0,i
Yes, also for 'just a test' program...

Regards,
Hein.

[Ravi_8]

Thanx Dietmar

Your solution writing to raw device /dev/vg00/rpacs4 worked, irrespective of rpacs4 size.