LVM Load Balancing

[Tom Campbell]

Under HPUX 11.00 and earlier I have always understood that LVM does not load balance between primary and alternate links. Alternates are for fail over only. Is it absolute that the alternate link ONLY gets used if the primary path fails completely? Even if the Request Queue for a "drive" seen down the primary link climbs above 3 and beyond, the driver does not consider the primary unavailable and start using
an alternate?

The reason I had to ask this "silly" question was I saw disk I/O coming down two different hardware paths that correspond to the same logical EMC drive while using Perfview. Both had the same time stamp but different values in their fields. One path was my primary and the other was the alternate and I made sure no LVM failures had occurred and I confirmed the drive relationship using a vgscan, ioscan and an EMC syminq. Could this be a quirk in Measureware/ Perfview looking at an EMC? BTW. I just checked sar and it shows I/O on the alternate link as well. If there has been no failover to the alternate path, and we definitely don't have EMC's Power Path, how can this be?

Thanks.

Tom

[Steven E. Protter]

The EMC part might be load balancing.

See this thread on PV-Links. It might provide your answer.

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xc51431ec5e34d711abdc0090277a778c,00.html

Click deep, the answers are there sir.

P

[Sridhar Bhaskarla]

Tom,

What you heard is correct if and only if you set your system to default configuration where one path will be always active and other servers as passive.

However, you can change this behaviour and make use of both the links by changing their order in /etc/lvmtab by selective vgreduce and vgextend commands. Since this is manual, you will need to understand how the disk subsystem is laid out on the back else your will be counterfeiting the loadbalance on the other side.

For the other question look at the order of PVs in your /etc/lvmtab. For ex., if part of your /etc/lvmtab looks like

vg01
/dev/dsk/c0t0d0
/dev/dsk/c0t0d1
/dev/dsk/c1t0d2
/dev/dsk/c1t0d0
/dev/dsk/c1t0d1
/dev/dsk/c0t0d2
..

where c0 and c1 represent controllers of two hbas, then the activity to t0d2 device will always be through the controller c1.

You can also find it out by "vgdisplay" and note down the primary and alternate paths.

-Sri

[Michael Tully]

There have been a number of postings recently in regards to PVLINKs and load balancing.
Steve has pointed on such out for you. In regards to EMC, this would only happen if you have POWERPATH. My guess is that you have some alternate paths set up as primaries or you have a dud controller. Check the outputs of your 'vgdisplay -v' to check.

[Bill Hassell]

As far as I know, read queues will automatically use alternate paths on mirrored disks. I have verified this on a simple Jamaica disk array by mirroring the disks left and right, then starting simple (slow) reads (into /dev/null) followed by busy reads (multiple processes). The mirrors would not be accessed as long as the disk queues were low but lit up immediately as the queues moved past 4 or 5. Naturally, all writes occur on all alternate links.

The idea of alternating links when creating logical volumes works quite well too. lvols 1,2,3 should all be on the same link for sanity (should work OK but these are the most important lvols to get started). But alternating lvols 4+ is an easy way to load balance.

[Tom Campbell]

Thanks for the quick response folks. The link helped me confirm what I've always heard. I'm already alternating primary and alternates to get some type of IO balancing on the EMC. I had fun doing it with 72 drives in the volume group and 4 paths for each drive. I grepped out c11t3d7 and c13t3d7 from a syminq and confirmed they both have the same unique EMC hex serial number. I checked my ioscan and grepped my vgdisplay to confirm c11t3d7 was still my primary and c13t3d7 was still my first alternate and no where else. I set this box up 4 months ago and it looks the same today and I'm the only SA that has touched it. We already use Power Path on some of our servers so I agree that this sounds like something you would see in that environment. However, the customer on this project suffered sticker shock when POWER PATH was quoted and decided to pass. Since logical drives are just that and not physical, I've always wondered how realistic the logical drive appears to the OS and how utilities handle looking down 4 different paths and seeing the same "drive". You are actually looking at EMC software that shows you what it wants you to see. After all they ARE a software company. :-)

[Sridhar Bhaskarla]

Hi Tom,

As Bill pointed out, LVM does access mirror disk for reads. And if the mirror disk is on the other path, that explains it.

However, I doubt that you setup such thing on your system as you are accessing EMC. Apart from perfview, did you try a sar -d 2 200 and confirm if you are seeing the disk being accessed through both links?. I have systems with more than 200 disks and I have never seen this kind of behaviour.

-Sri

[Tom Campbell]

I think I???ve stumbled on to what happened. My bad for not being more diligent. The server had a routine reboot last Sunday at 8:00 PM and syslog.log looks fine. However, OLDsyslog.log had several POWERFAILED messages whose time stamps match up with the ???strange??? sar output. The path recovered each time so we never had a failure to catch. Would have thought Predictive or EMS would have alerted us. Will have to check into that. However, your input did help clear up a debate we???ve been having concerning how LVM actually utilizes alternate links. With Bill???s contribution the answer is: ???It depends??????. Thanks again for your help. Seems I need a SA 101 refresher course.

[Q4you]

We ran into a situation where we had to recreate "lvmtab".

Since we were in hurry, we recreated lvmtab with vgscan option. Originally we did use alternate controllers disk paths ( per Sri) while creating VGs but with new lvmtab due to sequential scanning of the buses, we lost the "alternate path" usage.

We are currently seeing all disks are hitting on one controller. How to fix this online with minimum impact and survive next reboot ?

[Sridhar Bhaskarla]

Hi Q4?,

If you looked at one of the same posts, the way to do it online is given. For ex., if your lvmtab says

vg01
/dev/dsk/c0t0d0
/dev/dsk/c0t0d1
/dev/dsk/c1t0d0
/dev/dsk/c1t0d1

In the above case both c0's are primary. To make c1t0d1 as the primary disk, do

vgreduce vg01 /dev/dsk/c0t0d1
vgextend vg01 /dev/dsk/c0t0d1

This will make lvmtab to look like

vg01
/dev/dsk/c0t0d0
/dev/dsk/c1t0d0
/dev/dsk/c1t0d1
/dev/dsk/c0t0d1

Since c1t0d1 is appearing before c0t0d1, it will be treated as primary link.

-Sri

[Angus Crome]

Check your EMS, STM and other Online Diags. I had all kinds of problems with bogus disk fails (EMC, IBM and HP-XP) when using the March and June 2002 Support CD's versions. The September and December have a newer version of STM and a patch in HWE to fix some problems with EMS. They went away immediately after upgrading the software.

[Tom Campbell]

Angus,

Thanks for the heads up. We are running the June 2002 Diag bundle.

Tom