PV-Links question

[Steven E. Protter]

Here is what I've read on PV-Links thus far:

http://search.hp.com/redirect.html?url=http%3A//forums.itrc.hp.com/cm/QuestionAnswer/1,,0x2ba753921f1ad5118fef0090279cd0f9,00.html&qt=PV-Links&hit=3

http://search.hp.com/redirect.html?url=http%3A//forums.itrc.hp.com/cm/QuestionAnswer/1,,0x614d7bb04b5cd611abdb0090277a778c,00.html&qt=PV-Links&hit=7


It would appear that I need to run build pv-links failover with vgcreate.

Problem is that the SAN admin has been a little slow getting things done. The alternate cards aren't plugged in, nor are they zoned at this time.

Of course I've got to present a test system on Tuesday and I'm wondering if I can extend my volume groups later once the SAN admin catches up with me.

I know its silly, but don't even get me started on how long it took them to allocate disk space on the one card they did get plugged in.

A cookbook is good for 10 points, as is a link to a cookbook.

P

[Sridhar Bhaskarla]

Hi Steven,

You can extend your volumegroups with Alternate Links later and online.

Once the links are powered-up, just run "ioscan -f" to grab the devices, then "insf" to create the special files. Only the Controller (cx) will change and the rest will remain the same for the device files of the alternate links.

Once you identified the devices, run vgextend and they will be set as alternate links. Run vgdisplay -v on each volume group and make sure each device has an alternate link. Once it is verified, pull out cable from the controller that appears as Primary and see how the links will switchover. Connect the cables back and the links will failback.

-Sri

[Steven E. Protter]

I'm putting this together from a dozen different posts, so if anyone can point me to a doc that shows me how to do it beginning to end, I'd appreciate it.

I'll point the posts on Monday, after I meet with my SAN people.

P

[Steven E. Protter]

So it doesn't mattter that there's data on the logical volume already. I'll back it up before I start screwing around.

I suppose this is all in the advanced LVM class that I should have taken a year or so ago.

P

[Eugeny Brychkov]

Steven,
of course you can add alternate paths after you'll add additional hardware. Example: you have server with one FC HBA, seeing all LUNs once and you created vgs on the SAN disks. Then you bought 3 more FC HBAs and want to install them to add 3 (or more) paths to your LUN. What should you do: shut down host, physically install HBAs, connect cables, power up hosts, wait it to boot to multi-user mode. Then you check if LUNs are seen on other paths (ioscan -fn) and device files are created. Now you can simply vgextend your vg to alternate paths (1 path, 2 paths or all 3 paths) and if some paths will fail in the future then LVM will use 'still live' paths.
But in case you need to remove some FC HBAs, sequence is the following: vgreduce vg from needed paths, remove device files, then shut down host, remove hardware, and power up host. Do not remove hardware before reducing volume because then vgreduce will not allow you to reduce inexistent path telling 'can not query physical volume'
Eugeny

[James R. Ferguson]

Hi Steven:

Sri and Eugeny have already provided you an ample foundation.

It appears, that you are *adding* alternate links after the initial creation of the volume group(s). In that case, when you 'vgextend' the new device files into the existing volume group, the new paths will be entirely "secondary" ones to the existing "primary" ones, and this is not desirable.

While no more than one path to the same physical disk device is used by LVM at at time, you *do* want to load-balance I/O between/among disk controllers in order to provide optimium performance. Thus, if you had a volume group of eight physical disk with sixteen paths divided between two controllers, you would want to have four primary links from one controller and four primary links from the other controller servicing the group.

If you were creating the volume group from scratch, you would merely order the 'pv_path' arguments to make primary path assignments from both controllers. If you are adding alternate links for the first time, simply add the secondary pv_paths and then apppropiately *exchange* the primary and secondary links to achieve balance. This exchange is done when a primary link is 'vgreduce'd thereby promoting its alternate link to primary status. The original primary link just reduced is then 'vgextend'ed which makes it a alternate link. This exchange can be done while the volume group is active and inuse.

In cases where you have a large number of pv_paths it is sometimes convenient to 'vgexport' and 'vgimport' the volume group to reassign primary and alternate link status to the physical devices. This is easily done by using the '-f outfile' option for 'vgexport'. The current set of pv_paths for the volume group is thus written to the outfile. The outfile may then be used as the 'infile' for the 'vgimport -f' option. Before running the import, simply order the sequence of 'pv_links' within the out/infile to make primary/secondary assignments as desired. Of course, the disadvantage to this method is that you must inactivate the volume group to perform the export/import process.

Regards!

...JRF...

[Steven E. Protter]

I'm thrilled with what I have thus far.

I guess my confusion is because we would not use PV-Links to load balance, we'd use it to provide an alternate path to the same LUN via a second Fiber card.

So I'm concerned about the data sitting out there. There is no way anyone is going to double my disk allocation so I can load balance on two different Virtual disk(LUNS).

So we're looking at a situation where we extend the volume group, have more band width, an alternate path to the very same disk space.

I'm not even sure if its wise or practical to do this.

I appreciate your input and will happily and thankfully read more. Though I usually assign points immediatly, I want to meet with my SAN people, talk to the disk vendor about this and assign higher point values if my problem really is solved.

Thanks, Thanks and Thanks.

P

[Eugeny Brychkov]

Steven,
hpux OS uses only one path to access LUN. If this path fails, then it selects another path and uses it. No load balancing in this way (at least without special software).
People are talking about load balancing over the SAN meaning that if you have disk array and 10 LUNs created on it, you can set primary path to all 10 LUNs through one HBA and one disk array controller, BUT you can split them two by five and share LUNs between two HBAs and two disk array controller. So first 5 LUNs will be accessed by one HBA through disk array controller 1, but another 5 LUNs by another HBA through controller 2. Example: VA74x0 - having two redundancy groups (RG1 and RG2) and perfomance path through C1 for LUNs created within RG1 and C2 for LUNs created within RG2. And, creating LUNs in different RGs you set paths through different controllers and making VA faster twice :o)
Eugeny

[Sridhar Bhaskarla]

Hi Steven,

I am a very favourite of using both links and spreading the IO across both the controllers. But I suggest you do it in two stages. First get your alternate links established. Make sure they work, test them out. Then carefully plan how you would want to balance your IO. There are two ways of load balancing (or sharing) the links.

1. Altering disks in each volume group. For ex., in the following configuration.

vg01 - disk1 - c0t0d0 c1t0d0
disk2 - c0t0d1 c1t0d1

Here you make c0t0d0 as the primary for disk1 and c1t0d1 as the primary for disk2. Repeat the same for other volume groups.

2. Altering disks per volumegroup. Say you have the following configuration.

vg01 - disk1 - c0t0d0 c1t0d0
disk2 - c0t0d1 c1t0d1

vg02 - disk3 - c0t0d2 c1t0d2
disk4 - c0t0d3 c1t0d3

Then you will keep c0 as the primary for vg01 and c1 as the primary for vg02 and so on.

Point to note that it is the order in /etc/lvmtab that determines the primary and alternate links. For ex., if your /etc/lvmtab looks as below.

..
vg01
/dev/dsk/c0t0d0
/dev/dsk/c0t0d1
/dev/dsk/c1t0d0
/dev/dsk/c1t0d1
..

The above will make c0t0d0 and c0t0d1 as the primary links and the other two as alternate links. To change the order, you vgreduce and vgextend one LUN.

#vgreduce vg01 /dev/dsk/c0t0d1
#vgextend vg01 /dev/dsk/c0t0d1

Now your /etc/lvmtab will look as below

..
vg01
/dev/dsk/c0t0d0
/dev/dsk/c1t0d0
/dev/dsk/c1t0d1
/dev/dsk/c0t0d1

Here c0t0d0 and c1t0d1 are primaries.

*Warning*: The above may look straight forward and simple to implement. It is true as long as you have JBODs. However, if you have sophisticated disk systems like XP or EMC, then you may want to sit with your SAN admins and the discuss your solution. This is because you may be load balancing on the system side. However, depending on how the storage is configured, you may affect the loadbalance on the backside. So it is very *important* that you get buy-off from your SAN admins. Also, you will need to do benchmarking with each scenario and implement the one that gives you better performance.

-Sri

[Karamanolis Nikos]

Hi,

Sridhar made a good point there.
You will be load balancing on the system side BUT you should know what is happening on the SANs' side.

In modern arrays usually each LUN is asigned to a Storage Proccessor (known by many-many names). You will have to know this information so you can assign a primary path to that LUN through THAT Proccessor.
Otherwise, performance is affected....

[Steven E. Protter]

Thanks. I'll meet with SAN administration and report back. Then I'll point. I'm thinking I got enough to hand out 7-10 points on some of the answers.

This is a complex area, and I should probably take the advanced LVM class.

P

[Eugeny Brychkov]

Sure. And as soon as you will be working in SAN envrionment then SAN class will be useful also
Eugeny

[Steven E. Protter]

Woohoo!

The SAN vendor sent a doc.

Gonna read it, gonna try it out.

Gonna hand out points before procura does stats.

P