cannot write chunk less than 1024 bytes into raw veritas volume

[Roman Guoussev-Donskoi]

Simple program writes into a raw device (Veritas logical volume) and everything goes file as long as each write size is greater or equal 1024 bytes.
If I try to write 256 or 512 bytes at a time wtite call returns EINVAL.

write(4, 0x7b040770, 512) ....... ERR#22 EINVAL

Before startting to write program skips 1024 bytes from the start of raw device.

I've read that it is possible to write into raw device with up to 256 bytes granularity.
But this definitely does not work on HP.

Does nayone know reason why I cannot write 512 bytes chunk into raw device?
Thank you.

[Jeff Schussele]

Hi,

I would expect that it's because the LVM block size is 1K. Even though it's a raw volume, it's *still* an LVM controlled device & is an LV in a VG. It just has no filesystem on it.
I suspect you'll have to pad the writes up to the 1K boundary.

Rgds,
Jeff

[Roman Guoussev-Donskoi]

Jeff, can you point me to any doc that states that LVM block size is 1K?
I have been trying to remember but could not recall seeing it anyware...
Probably my memory is not good :(

[A. Clay Stephenson]

Do a man scsi_disk. It will explain that the smallest possible I/O quantity is one logical block. That value is set by the value BLKDEV_IOSIZE which must be >= DEV_BSIZE and <= MAXBSIZE. DEV_BSIZE in /usr/include/sys/param.h is defined as 1024 and that is your answer. Man 7 disk for more details. Remember, logical volume I/O whether conventional LVM or VxVM must ultimately be passed on the to underlying disk I/O routines and thus SCSI I/O defines the limits.

[Roman Guoussev-Donskoi]

Thank you.
You definitely got the answer. I lass than 10 points only because further tests showed that part of your reply does not seem correct.
After I have read disk and scsi_disc I have tested a small program which appears in disk 7 just added printout of logical block size as well.
The output as follows:

/usr/local/bin strings /etc/lvmtab
/dev/vg00
/dev/dsk/c1t6d0

./disk_describe /dev/rdsk/c1t6d0
Successful ioctl DIOC_DESCRIBE
model number: ST39103LC
interface: 20 <20=scsi>
logical block size : 512

./disk_describe /dev/vg00/rrawtest
Successful ioctl DIOC_DESCRIBE
model number:
interface: -1 <20=scsi>
logical block size : 1024

From above it seems like scsi allows writing in 512 bytes (which contradicts to part of your reply), it is really Veritas that does require 1024.

[Hein van den Heuvel]

The 1024 from /usr/include/sys/param.h is in the context of FILE SYSTEMS. Each device has it's own native blocksize. For SCSI this typically is 512, as you saw. Drivers that build on top of the device (LVM, filesystems) will add their own restrictions, 1024 in this case. Some file systems refuse to do IO in other then 8192 chunks. If you write 512 bytes to those, then they will pre-read 8192 into a filesystem buffer, merge in the 512, and write 8192.

This is kinda funny to see on Oracle REDO logs, where you _know_ the action is 100% write (no archiving) and yet there are a bunch of reads? Going RAW solves that.


fwiw,
Hein.

[Roman Guoussev-Donskoi]

Actually it is oracle redo logs that has prompted me to get into this test.
I have always put them on raw for reasons of block size writes, indirect access pointers for big files, memory copy overhead etc.
But some customers prefer filesystems and I had to check actual improvement.
To start with I have created veritas filesystem with block size 1024 and raw device and then performed 10K sequential writes of small 2K size into them. To my surprise the timing is almost identical (at least on a disk).

May be a good storage with write cache will make a difference...

[Hein van den Heuvel]

> Actually it is oracle redo logs that has prompted me to get into this test.

I'm not entirely surprised. That's why I put my remark in! Hit!

> I have always put them on raw ... But some customers prefer filesystems

You could propose a compromise. During the main production window, run on large raw devices (1GB? 10GB?). Just for the slow / backup window just create a new group of small (50MB, 500MB?) redo logfiles on a filesystems. switch log and drop raw. When backups are done, recreate the additional large raw redo's, switch and drop the filesystem ones.

> device and then performed 10K sequential writes of small 2K size into them. To my surprise the timing is almost identical (at least on a disk).

Did you do like Oracle does and SYNC after every write? That can make all the difference.

> May be a good storage with write cache will make a difference...

Sure will

Hein.

[Roman Guoussev-Donskoi]

Actually online redo logs are not needed/or even desired for backups (not even online ones). Archived redo is sufficient in proper backup scenario.

It is not backup issue, curstomer reports some oracle bugs which are raw-specific (have not send us actual bug# yet).

I'm opening it with O_DSYNC. This is what Oracle does since >= 7.3.