LVM/Mirroring - uneven I/O load on Arrays

[Stephen Kebbell]

Hi all,

when using LVM to mirror disks on two separate storage arrays, is there any way to set which disk should be the primary read source? We have the situation where one of our storage arrays (EMC Symmetrix DMX-3) has about double the read I/O load of the other. Write I/O is fairly evenly balanced, as LVM has to write data to both disks in the mirror. Assuming our current growth rate of I/O, soon we will have a performance impact on one array, which will slow down the oracle response times dramatically.

How does LVM choose which disk it reads from?

Thanks in advance.

Stephen
(p.s. I'm not a Unix admin, so I can't provide much technical info on the Host)

[Steven E. Protter]

Shalom,

In reality they are both the read source. The mirror/ux product handles which disk is read on a availablity basis as far as I remember. I do not belive it is user configurable.

Generally this setup improves performance.

SEP

[RAC_1]

I think it depends on the sequence in which you did the mirroring. If disk A was mirrored to B, then reads are from A. If you want reads from B, then reduce mirror A and mirror back using B.

Let gurus confirm this.

[Rita C Workman]

I'm going to suggest something different.
You have EMC storage...do you have EMC PowerPath ? You could so easily just set this up and it works great for such problems.

The cost for PowerPath has come down significantly then what it was when we introduced it here. Call your EMC sales person, see if they will let you 'test drive' the software. That might help mgmt see the benefit and be willing to expend the funds for what really is a great product.

Kindest Regards,
Rita

[Stephen Kebbell]

Hi,

thanks for your replies so far.
PowerPath won't help here, as it would only load-balance between alternate paths for each mirror-member. It won't load balance between mirror-copies on 2 different Symmetrixes.

(Incendentally, we have a site license for PowerPath, but our Unix Admins say "no-go". So we only use it on our Windows boxes to Clariion storage)

Regards,
Stephen

[Steve Lewis]

Stephen

It always reads first from the top PV/disk/lun that is listed in the volume group (as per vgdisplay -v ). If that disk is busy, it then reads from the mirror.

You can try to mitigate the uneven load by un-mirroring some (not all) of the heavily used volumes to take off the primary path, removing their PVs from the volume group, then re-adding the PVs to the end of the volume group and re-mirroring.

Actually in the days of mirrored disks, I was a fan of doing it the unbalanced LVM way, because I had an un-tested/wild/silly hypothesis that you were more likely to get one disk fail at a time, not both disks failing at once. Not that that matters any more.

[Geoff Wild]

This is very interesting.

I posted this thread about issues with an Oracle to Oracle migration on a DMX3:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1067723

As far as setting which disk in a mirror s the source - LVM mirroring doesn't work that way.

Rgds...Geoff

[Tim D Fulford]

Hi.

LVM does not balance the read IO 'till the disk gets busy enough to justify it. so you cant do exactly what you want...

IF you have 4 (or more) physical volumes (say 2 mirrored onto 2); say A[td0] B[td0] A'[td1] B'[td1]. whre A' is the mirror of A and td0 & td1 are the two FC ports the disks are presented down. Under this scenario td0 will redcieve most read and all writes & td1 will recieve a few reads and all the writes. In the lvmtab it probably look like
/dev/dsk/c10t0d0 [A]
/dev/dsk/c10t0d1 [B]
/dev/dsk/c12t0d2 [A']
/dev/dsk/c12t0d3 [B']

You can load balance(ish) the fibre paths td0 & td1 balanced by simply reordering your disks in LVM (vgreduce & vgextend) to get A[td0] B[td1] A'[td1] B'[td0] (you can also use pvchange, but this does not survive reboots)
so in lvmtab terms
/dev/dsk/c10t0d0 [A]
/dev/dsk/c12t0d1 [B]
/dev/dsk/c12t0d2 [A']
/dev/dsk/c10t0d3 [B']

In essence you will NOT beable to load balance the disks, but you can make the fibre channel paths more ballanced

Going back to powerpath. are you positive you cant balance the IO over the FC, the point of powerpath (securepath etc) is to load balance the requests over the SAN cahnnels (as I've sort of done).

regards

tim

[Tim D Fulford]

oops, just re-read your reply about two different DMX-3's... so you are right powerpath will not help here at all. The only way (I can think of) is to interleave the primary/mirror disks on the arrays.

Regards

Tim

[Dave Wherry]

I have a question? Why are you mirroring at the operating systems level? The Symm is already doing some level of Raid protection on the array.
Mirroring at the OS is just creating more I/O operations on the server.