PowerPath and PVlinks.

[Steve Post]

Two very simple questions.
1. What does the term "PVlinks" mean?

2. I was advised by EMC to not use alternate physical volume paths to the same physical volume. If I do, it would screw up PowerPath.
In the forums a few people said the opposite. That it is a good idea to do this. Which way is it?


When I run vgdisplay -v vg##, I get stuff like this.....
--- Physical volumes ---
PV Name /dev/dsk/c0t0d1
PV Name /dev/dsk/c1t12d2 Alternate Link

I think this concept of having an alternate path to the same physical volume is what people are calling PVlinks. For all I know, this might actually be the term: MoonUnit.

[Christian Tremblay · Accepted solution]

To be safe, set up your pvlinks in LVM the same way you would if Powerpath was not installed. If for any reason Powerpath fails to detect a HBA card failure then you still have the alternate path set and LVM will take care of switching to the alternate path.

We had an incident last year where a HBA card failure was not detected by powerpath. It may not have been Powerpath's fault since HP afterwards issued a patch to fix the problem, but we lost the whole production and almost lost our jobs over that incident.

Setting up LVM alternate links is now mandatory in that environment just to guard against repeating such an incident.

[Steve Post]

Thanks. I'm one third the way there.
I don't know what PVlinks is.
I don't know why EMC says don't do it.
I have a reason to use alternate paths.

I've been trying to contact EMC for the last 2 days. (I kind of hate their powerlink website so far. I guess I'm spoiled with hp's.)

[Christian Tremblay]

A pvlink is simply a different path to the same disk that goes through a different controller and a different fiber card I have seen setups with 4 different physical paths to the disks. This provides redundancy in case of hardware failure of the HBA cards.

[Steve Post]

I guess I shouldn't have assigned points yet. I have one question left to be answered. I'm getting close to having a response from EMC. If they will talk to me, I'll post their answer here.

Steve

[someone_4]

They (EMC) has told us to use alternate links.
They told us that they are needed if the server reboots and it can't find the primary link /dev/dsk/c0t0d1 in your example.

I am very intrested in what EMC tells you.

Richard

[someone_4]

Here is the link the my post on this subject:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1031832

Richard

[Steve Post]

EMC may be too nebulous a term. I always used PVlinks to EMC symmetrix. I have an EMC Clarion CX300 with powerpath now. According to your referenced forum, I should NOT use PVlinks because it forces additional overhead and possible LUN trespass conditions. I tried getting more info on it at powerlink.emc.com. I got kicked out again.

"LUN Trespass". That means the EMC box is looking for the disk to be accessed by host1 through switch 2, and to storage processor A....But emc got a request for the disk from a different path. I can get to the disk from host1 via:
switch1 to spA
switch1 to spB
switch2 to spA
switch2 to spB

Powerpath watches the performance of the disk access and moves it's access through a different path. It's juggling the request from /dev/dsk/c1t2d3 to 3 other devices (behind my back) for performance.

I looked that the AX pdf file you referenced. But when I tried to look for CX300 equivalent, I sent me all over the powerlink site. Then it crashed when it tried to make me give a survey.

I'm still waiting on EMC to talk to me.

[Ted Buis]

Steve,
PVlinks is just the alternate path to the same physical volume and was developed over a decade ago, before PowerPath existed for HP-UX. It is simple failover from one HBA path to another in the event of an HBA path failure. That is what PowerPath attempts to do dynamically via different piece of code. I would use one or the other but not both on the same physical volume. Hopefully, HP will provide a dynamic implementation of PVlinks in HP-UX 11i v3, and you won't have to consider a third party Dynamic Multipathing Driver for LVM.

[Steve Post]

Ok that's the third piece. I'm keeping status quo. I'm not adding Pvlinks.

Christian, I'm NOT saying you are wrong. You have had first hand experience with PowerPath not holding up its end.

When I got the hardware in, I purposely pull the fibre connection out the one the host bus adapters. Powerpath handled it fine. I next plugged it back in. At that time, the box crashed. But it came back fine with a reboot. I was unable to get the error a second time.

Still waiting on emc....

[Christian Tremblay]

No problem Steve, I just wanted to share a real life experience with everybody. Powerpath is very good, but if for whatever reason it was to fail; then having underlying alternate paths configured in LVM is a safeguard against loosing access to your storage.

I still have to see the proof that proceeding that way has any adverse effect on Powerpath.

It's like having the belt and the suspenders.

[sathish kannan]

Steve,

As you know PowerPath and PV Link both are used for path failover. PowerPath has additional advantages than PV links. It does load balance aswell. PV link does not.

Whether you use Symmetrix (Active/Active) or CLARiiON (Active/passve) with PowerPath it is better to leave it to PowerPath to manage the path and the I/O.

You should be careful with using PVlink with CLARiiON. If you don't have PowerPath EMC always recoomeds you to use Default owner as the fisrt path and rest of the path as alternate path.

Regards
Sathish

[Bill Costigan]

I believe the reason that EMC advised you not to use PV links with PowerPath is as follows.

1. HPUX sets up a device file /dev/dsk/cxtxdx for each path to a LUN through the fibric
2. PowerPath identifies all the paths and can use any or all if it wishes
3. PowerPath intercepts I/O calls to any of the devices files and reroutes the I/O to the path it wants.
4. Issuing an I/O on a Path will cause PowerPath to switch its primary path to that path, this might cause it to switch the LUN onwer to a different contoller (which takes some time)< Main reason for recommendation>

Now LVM allows you to add multiple device files for the same LUN to a volume group. This is called PVlinks. If LVM cannot talk over one of the links it automatically tries one of the other paths.

However, when LVM first activates the VG it tests all the links to ensure they work. Let's assume each LUN has 4 links and there are 10 LUNs in the volume group. Each time LVM tests a different link, PowerPath switches its primary path for that LUN. This means that PowerPath might do 40 path switches during the VG activation. Each might take 5 or 10 seconds to complete.

The thought is that LVM should only have a single path defined for each LUN and that path should be selected to be the BEST path.

If that path does fail, PowerPath will route all the I/Os for that path to an available path.

Moreover, the VG activations will be fast because LVM will only issue the I/Os to the path that is already the perfered one.

The downside is the if PowerPath fails and also one of the paths fail, LVM will not know of any alternate path. If you can live with the delays in VG activation, you could configure the primary path and 1 additional PV link for each LUN choosing the alternate path that has no common elements with the primary path. (e.g different, HBAs, switches, CX controller)

[Steve Post]

Well that last answer makes the most sense. I do in fact have 4 paths to each disk.

I am still trying to get an answer from EMC. Instead of using their powerpath (that keeps kicking me out), I'll put in a service call. It seems lame, but I've been trying to get an answer since Monday.
Like I said, once I get an answer from them, I'll put in it here.

Steve

[Bill Costigan]

Oh, when I mentioned 5 to 10 seconds - that was for each path switch, so switching 40 paths could take a few minutes just to activate one VG.

[Scott Riley]

Steve,

This all depends upon whether you are using Symmetrix or Clariion. With Symmetrix, which is active/active like the XP/Hitachi arrays, I recommend using both Powerpath and PVLlinks, to avoid situations like Christian encountered. With Clariion, which is active/passive, use one or the other, but not both.

You didn't state which model Clariion you are running, and what version of firmware it is on, which can change things.

The Clariion should be configured differently depending on whether you are using PVLinks or PowerPath. With PVLinks, it uses "Auto Trespass" which trepasses a LUN automatically between SP's based on I/O access down a non-active path. PowerPath uses "No Auto Trespass" which requires a SCSI trespass command to switch paths. If you have both failover solutions running on the same LUN, obviously there's a conflict in the Auto Trespass setting in the array.

So bottom line, with Clariion only use one of the two solutions. If you have PowerPath, use it, because it's a better solution than PVLinks for EMC hardware.

[Bill Costigan]

Scott,

PowerPath will not even list a path if it is set for 'Auto-Trespass' At least that is what I've seen. You do a 'powermt display dev=all' and no devices are returned.

If powerpath is installed and working then PV links runs throught powerpath so there is no conflict with the trespass. It is set to No-trespass.

But doing this will cause a trespass to occur (via powerpath) whenever LVM does an IO to an alternate path.

Your point is a good one about configuring PV links as a backup to PowerPath. If powerpath is broken then PV links will need you to change the trespass mode on the LUNs. Not exactly, a transparent automatic failover solution.

[Steve Post]

I never did get an answer from EMC directly. I just didn't get the time to call. My email (sent Monday) went unanswered. But I went back through my notes and documentation.

I have a clarion CX300 array with 2 fibre switches to 2 rp4440's. For each disk device, I have 4 paths. I have powerpath 4.5. and flare 02.19., and navisphere 6.19.
The manual for powerpath installation and admin guide says to use PVlinks if you are booting from one of the disks, otherwise do not use PVlinks. I am not using these as bootable disks.

I remember I was told to do this by EMC: In the Navisphere config file, the autotrespass line is commented out, and a line "device auto auto" is added.

Why do I use PVlinks for bootable disks, and not for normal disks? Well, I can make a good guess. Navisphere is software on the hpux box. This software is not loading and running and the point where I'm booting up. PowerPath isn't either. So the only way to go to an alternate AT bootup time is to use PVlinks. Otherwise it's a cute Catch-22.

"I'm booting up. Where's the alternate path?"
"I'm not telling you until you boot up. Will you boot up?"
"Nope. Not until you tell me the alternate path."
"I'm not telling the alternate path, until you boot up."
........catch-22.

[Scott Riley]

Sorry, I meant to comment on your Powerlink woes. I do consulting for an EMC partner, so I deal with these sorts of things a lot. Getting you proper access to Powerlink is the responsibility of your Partner or EMC, depending upon who you got the array from. So I would go yell at those folks, because Powerlink is a really important tool.

Once you're able to get into Powerlink sucessfully, take a look at the "Customer Clariion Procedure Generator." ANY time you add ANY host to a Clariion array, you really should run this procedure generator. Even if you've done it 20 times before, because the Procedure Generator changes frequently, and points you to hot fixes, known issues, etc.

To get to the Procedure Generator, log into Powerlink, then through the menu go to:

Support->Product And Diagnostic Tools->Customer Clariion Procedure Generator

That will take you to a page, where you can click on "Download Clariion Procedure Generator - Customer Version."

After you install this tool and run it, you get a wizard that allows you to specify your environment, then it generates a Word document procedure to follow to do your install. If you follow this religiously, you will be in good shape.

Best of luck!

- Scott

[Christian Tremblay]

I must mention for clarity's sake that my comments only applied to DMX2000 and 3000 frames as there were no Clariion frames in our environment.

Chris

[Steve Post]

Scott, I looked at that procedure generation. THAT works. My problem with the powerlink website is not logging in. It's getting kicked out every 5 minutes. It's kinda annoying! But once I'm at that generator, it seems to work fine. My EMC woes here are just about about non-critical information. If I had a real problem, they usually call me about 5 minutes before the hpux hardware even pages me.

And Christian, thank you for your replies. Even though you use a different version of disk array, I WANTED to know why there were differing opinions on this subject. I didn't like the idea of doing X simply because everyone does it.

thanks.
steve

[Steve Pittenger]

Steve,

Not that this is directly answering your issue but I thought was noteworthy because I am currently working a case with HP and EMC on this. We use PVLinks in our Clariion environment and have for years with great success. After applying patches recently to HP-UX 11.11, we started experiencing excessive lun trespasses in our Clariion logs related to the HP-UX hosts. EMC has a Primus solution (emc143405) that identifies the patch that caused the problem as a LVM patch PHKL_34518. The morale of a long and still being investigated story is that the official HP statement is that they do not support PVLinks in a Clariion environment. Furthermore, once I opened a case with HP they have so far stated there is no plan to remove the code from LVM that is causing this trespass issue. The funny part is that EMC says they do support PVLinks in this type of environment and as stated above has worked well for use for the past 5 years. Currently, I am awaiting the two companies to let me know the outcome. My options as presented to me from both companies are to remove the alternate links from my volume groups (PVLinks), install EMC's Powerpath on these hosts, or back out the LVM patch that caused this problem. The first option is no good unless you perform the second at the same time unless you want to be exposed to a path failure. The third option is not good if you want to keep your LVM code fairly current since there are no plans to remove the code causing the issue. Sounds like you are on the right track with Powerpath as your selection based on what I am hearing currently. As a previous respondent mentioned, just follow the EMC procedure generator for your situation and you should be fine. Good luck.

Steve

[sathish kannan]

Steve Post,
Sorry for taking your question away. I just want to get clarification from Steve Pittenger

Steve Pittenger,
I am not aware of HP or EMC statement about the non support of PV links on CLARiiON. do you have any documentes or link and could you please post us.

I know that HP patch causes a trespass ping-pong with failovermode=1 and EMC recommed to remove those patches.


Regards
Sathish

[Scott Riley]

I would find it really disturbing, if it's confirmed, that HP would not support Clariion in a PV Links environment, and I would ask for some documentation or a written statement confirming that. Pulling the plug on that support would mean pulling the plug on a 15+ year support relationship.

The original PV Links code was written around the time HP was selling the Nike 10 and Nike 20 arrays back in the early 90's. Those were Pre-EMC merger Data General Clariion arrays, and PV Links has supported Clariion arrays ever since.

Perhaps HP is stating they support PowerPath instead of PV Links?

By the way, there is a "free" version of PowerPath available, for the Clariion only, that duplicates PV Link capabilities without the more advanced load balancing that the purchased license gives you.

[Steve Pittenger]

Sathish,
Here is the link HP support provided me that talks about PVLink support. Footnote #4 is the particular note and it's explanation is about halfway down the document. Don't make the mistake I did and look at the footnote at the bottom of the document as that pertains to a different section of the document.
http://www.hp.com/products1/serverconnectivity/mass_storage_devices.html

I use failovermode=0 in all my PVLinks implementations and never had this trespassing issue until I applied the patch I listed in my above posting. That is what caused me to start researching with both companies and am still waiting for a final answer even though the above website(HP) states that it is not supported.

Scott,

I agree with you on the really disturbing part and that is why I continue to have the two companies speak about it. I am also, until recently, a very satisfied PVLinks customer in a Clariion environment. You are correct about the free Powerpath (which I believe they refer to as Powerpath Lite) and this seems to be the way both companies want customers to head. If this thread is still open when I get a definitive answer from both companies, I will post it here.