LVM Slower Write Speed with Mirrors

[Sibel Kamer]

Hi,

I'm hoping someone here can help answer a question for me about LVM performance in a mirrored state.

We are doing a proof of concept for a section which wants to do host based mirroring of SAN disk.
Server is HP rx86xx 11.23, it's using standard LVM and PV Links.
EMC DMX 3 is at the back end and there are 2 sites with a DMX in each, and DMDM Links in between.

One of the tests had us using iozone tool to write 32 X 1GB files simultaneously
The disk at the back end is an 8 Member Meta and each member is an 8GB drive which is RAID 5 (3+1)
The results I got are
Write to only the Local DMX - 160MB/sec
Write to only the Remote DMX - 95MB/sec
Write to Local & Remote DMX at same time - 50MB/sec

In the Local and Remote test the logical volume was lvextended from the Local DMX LUN onto the Remote DMX LUN.
The paths to the Local DMX LUN are down 1 HBA in the server and the paths to the Remote DMX LUN are down the other HBA.

So the questions that came from this is, why is there such a difference between the slower write to the remote DMX and the write to the Local and Remote DMX, should it be as fast as the slowest link? Or is there overhead in the mirroring which accounts for this?

This might be perfectly normal behaviour cause we've various results which tend to point to this.

Can this be explained? Is there any docs you know of with details about this type of stuff?

Any help would be greatly appreciated

Sibel

[Geoff Wild · Accepted solution]

That is interesting, for I always thought that HP's LV mirroring alternates between the mirrors for writes, allowing the other
drive to catch up.

There's some info in this thread - read my last post:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1201182578754+28353475&threadId=1069547

And some info in the sysadmin guide:

http://docs.hp.com/en/5991-6481/ch02s09.html

Rgds...Geoff

[Steven E. Protter]

Shalom,

I think the variance in write times can not be explained by software mirroring.

Software mirroring does cause overhead but there must be some other factor causing the variance.

The first place I'd look at is the setting on the EMC configuration. I've seen some radical peformance problems fixed by having EMC check for esoteric settings made by overeager engineers.

SEP

[Sibel Kamer]

Thanks Guys.

I've just conpleted this test on the internal disks in the server.

Writing to each disk on its own 25MB/sec
Writing to both disk in mirror 15MB/sec

So there seems to be something in it, 40% reduction on internal disk

[Sibel Kamer]

Can i just add, can someone explain how LVM writes in a mirrored set-up.
Does it send the writes to each disk simultaneously or down to one disk and then to the other disk.
It would help clear it up in my head.

[Torsten.]

See
man lvdisplay
for that (keyword schedule).

[TwoProc]

I would have to say, unless write caching is turned on (that is, either from software or hardware, you don't want to wait for writes to complete) - it would have to be slower.

How much slower depends on how you're set up.

Are the mirrored lvs on separate disks?
Are the separate disks on separate firber, scsi or sata controllers ?
If it's fiber, and if you are using a SAN are you coming out of the SAN switch and going to storage on different ports? And on the same topic - are the different ports connected to separate cpus? interfaces?

In other words, there should always be a penalty to write twice the data - more caching helps diminish this, more separation of hardware between the two writes helps as well. But, unless waits for the writes are totally turned off at the hardware and software levels (in this case lvm) - I can't see how it wouldn't be slower. No free ride as far as I know.