topic HBA replacement with EMC Clarion SAN in Operating System - HP-UX

HBA replacement with EMC Clarion SAN

Bob Brown_1 — Fri, 15 Aug 2008 13:10:33 GMT

I have an RP8400 vpar with 2 HBA's connected to a fabric connected to an EMC clarion array.
Each HBA is set-up with 2 paths to the SAN.
One HBA is now bad and needs to be replaced. The other is still good and the system hasn't lost any ability to access all of the LUN's (filesystems) on the SAN.

When I have the HBA replaced it will, of course, have a new worldwide name.
I am also running EMC powerpath on the host.

How do I go about getting the HBA replaced and the powerpath and hpux device files etc, all rehooked together to that hpux and powerpath sees the new HBA as just 2 more links to all of the existing LUNS?
thanks.
-Bob

Re: HBA replacement with EMC Clarion SAN

Jon M Zellhoefer — Fri, 15 Aug 2008 17:29:44 GMT

You shouldn't have to do anything more than update your powerpath config. Assuming you took the host offline and provisioned the SAN prior to bringing it back up when replacing your HBA ...

Just run:

powermt config (update paths)

powermt display dev=all class=clariion (verify new disks are there)

powermt save (update configuration)

PowerPath will take care of LB/Failover for you.

If you're hot-swapping the HBA, and then provisioning the SAN, you will need to do an ioscan, insf, and navicli register (sometimes you have to restart the agent /sbin/init.d/agent start/stop), then another ioscan, insf, and proceed to the powermt config section above.

Good luck!

Re: HBA replacement with EMC Clarion SAN

SUDHAKAR_18 — Sat, 16 Aug 2008 09:05:13 GMT

Yes Jon is right.
But you are also register the HBA in clarrion. ie in storage side.

Re: HBA replacement with EMC Clarion SAN

Bob Brown_1 — Mon, 18 Aug 2008 16:12:29 GMT

Ok..just to be clear..first I want to shutdown the hpux instance, right? Do I need to do anything in hpux before I do that?
Next I need the SAN group to make appropriate changes based on the new world-wide-name that the card will have?
When I reboot hpux after the SAN is updated and the card is replaced, I just run the commands you mention? Will it create a whole new set of /dev/dsk files or will it re-use the ones already in place?
thanks.
-Bob

Re: HBA replacement with EMC Clarion SAN

Jon M Zellhoefer — Wed, 20 Aug 2008 13:39:32 GMT

It will create new device files, but it will continue to "use" the old paths to identify the existing vg (it's a power-path thing). To clean up, extend in the new devices and reduce the old ones. The next time the system reboots it will completely clean up the "leftovers".

Re: HBA replacement with EMC Clarion SAN

Bob Brown_1 — Wed, 20 Aug 2008 14:02:41 GMT

Ok, so my steps would be:
1) deconfigure old, bad HBA in the san.
2) shutdown RP8400 and replace the HBA.
3) Upon reboot, it will still be working with the other (non replaced HBA).
4) Reconfig SAN to see the new HBA.
5) Go through steps of having hpux create device files when it sees the LUN's on the new HBA.
6)Add those new device files to the appropriate VG's.
7) Remove the old device files from the appropriate VG's (the device files that were used by the old HBA).
8)Reboot VPAR again to do final cleanup.
At this point I should be back to 4 paths to each LUN (as we were before HBA went bad)?

Re: HBA replacement with EMC Clarion SAN

Steven E. Protter — Wed, 20 Aug 2008 14:44:28 GMT

Shalom,

Any LUN's presented to this system by WWN World Wide Name (Mac address for fiber) need to be updated.

Maybe powerpath deals with that, but I would check with the EMC people on that. Everything I've seen shows WWN's build into the LUNs.\

SEP

Re: HBA replacement with EMC Clarion SAN

Jon M Zellhoefer — Wed, 20 Aug 2008 15:15:01 GMT

Bob -

Assuming you've shut down the server to replace the HBA (you don't need to de-configure prior to shutdown).

When the system comes back online, you will need to get rid of the old device files ....

A quick way to get rid of them all:

ioscan -fnC disk | grep NO_HW | awk '{print $3}' | xargs -i rmsf -H {}

or

for hwpath in `ioscan -fnC disk | grep NO_HW | awk '{print $3}'`
do
rmsf -H $hwpath
done

Once you reconfigure the SAN, you will need to do:

ioscan -fn
insf -e
powermt config
powermt display dev=all class=clariion
(verify your new disks are mapped to your existing luns)
powermt save

Then vgextend in a "new" device (doesn't matter which one, as long as it is from the same EMC LUN), and vgreduce the old.

Verify your old devices aren't tied to the VG anymore and reboot.

When the system comes back online, do:

powermt check

You will be prompted to remove the "dead" paths / devices. Once you have completed this step, the system will be clean.

Re: HBA replacement with EMC Clarion SAN

Jon M Zellhoefer — Wed, 20 Aug 2008 15:20:19 GMT

Steven -

EMC Logical Device ID's are owned by the Clariion array. Once the disk group has been created, the LIDs will remain static - provided you're simply adding a host WWN to the security group already established for the disk.

PowerPath uses these Logical Device ID's to manage the disk and doesn't pay any attention to what HPUX sees, since PP maintains it's own LID->Device database.

Still, it's always a good idea to have current map files for your affected VG's on hand prior to changing anything on the SAN.

Re: HBA replacement with EMC Clarion SAN

Bob Brown_1 — Fri, 22 Aug 2008 12:34:52 GMT

What do you mean about having current map files for vg's on hand before SAN changes?
thanks.
-Bob

Re: HBA replacement with EMC Clarion SAN

Tim Nelson — Fri, 22 Aug 2008 13:33:28 GMT

The map files are in case you need to re-import the vgs.

e.g.
vgexport -p -m vgxxx.map /dev/vgxxx

Also. Don't forget to update the switch zoning with the new WWN.

Here is my 2cents.
I am not so sure you will get new device names for your disk by just replacing the HBA. As long as it is replaced in the same slot the device file names should remain the same ( could be wrong, been a long time since I have had this happen, please chime in if I am wrong).

What will change as you know is the WWN of the HBA, simply update the switch zoning, update the Clariion config, cleanup any powerpath issues. There typically is a replace option with either the switches or the array just for this purpose.

As you have multipathing in place you will still have storage connectivity, just not down the one path until you cleanup the items mentioned.

Print out your current config before executing just in case you get lost.

Cheers

Re: HBA replacement with EMC Clarion SAN

Bob Brown_1 — Thu, 04 Sep 2008 13:02:44 GMT

Well...the hba has been replaced, but the new one wont log in to the SAN fabric.
Any ideas?
The new WW name has been set-up in the switches, the FC card is online, it just wont log in. I've tried a lot of things to get it to wake-up, but still no joy.
any ideas?
thanks.
-Bob

Re: HBA replacement with EMC Clarion SAN

Jon M Zellhoefer — Thu, 04 Sep 2008 16:57:05 GMT

Have you tried an ioscan / insf? If so, try directly querying one of the devices associated to it - many times with EMC you have to force a little I/O activity to get the HBA to show a logged-in state ... even just a diskinfo should change what you see on the array. This only seems to happen with HPUX systems on EMC Clariion arrays ...

Re: HBA replacement with EMC Clarion SAN

Bob Brown_1 — Thu, 04 Sep 2008 17:02:20 GMT

I don't seem to have any devices associated with the interface other than /dev/td0
-Bob

Re: HBA replacement with EMC Clarion SAN

Bob Brown_1 — Fri, 05 Sep 2008 14:30:17 GMT

We are all fine now...some stuff had to be done at the SAN end to cause the HBA's to log-in.
Thanks for the help!
-Bob

Re: HBA replacement with EMC Clarion SAN

Bob Brown_1 — Fri, 05 Sep 2008 14:33:15 GMT

fixed..thanks much for the help!

Re: HBA replacement with EMC Clarion SAN

Michele Musto — Fri, 05 Sep 2008 17:54:40 GMT

Bob,

I have to do the same exact task on my rp7420 attached to the Clarion array. Just so I can be clear, since there were so many replies.
You -
1. Powered down the servers/cabinet
2. Replaced the HBA to the same slot
3. Powered up the cabinet/servers
4. Have SAN folks configure new WWN
now -
Did you have to do any powerpath or system stuff for the new hba to be recognized once the systems were back up?

Thanks for your help.
Michele

Re: HBA replacement with EMC Clarion SAN

Bob Brown_1 — Fri, 05 Sep 2008 19:10:40 GMT

The following are excerpts from the workorders that I put together for this process. Hope this helps.
-Bob
-------------
shutdown rp8400 complex

12) HP replaces the bad HBA and it is then re-connected to the SAN
13) poweron RP8400 and boot-up to vpar monitor and boot jupiter

15) Insure that jupiter's /morestore (vg02/lvol12) and /db (vg01/lvol11) are up and running.
16) jupiter: powermt display dev=all and see that we have 2 paths to "/db", "/standby" and "/morestore".
16a) vgreduce /dev/vg01 /dev/dsk/c#d#t# /dev/dsk/c#d#t# to remove old dead paths to vg01
vgreduce /dev/vg01 /dev/dsk/c#d#t# /dev/dsk/c#d#t# to remove old dead paths to vg02

17) Run fcmsutil on replaced fc card (td0?) and get the new worldwide name to the LAN group.
25) LAN group adjusts SAN config so that jupiter's LUN's are mapped to the new FC card as well as the old one. Info from EMC follows:
Replace HBAs
First verify all paths are recognized by the CLARiiON array:
From Navisphere Manager, right-click the array and go to Connectivity Status.
Sort by Server Name and make sure all paths for the host say YES in the logged in column and the registered column.
Highlight each path and choose the info button. Make sure failovermode is set to 1.
Close out of Connectivity Status.
Reconnect HBAs to Storage group, Option #1:
Enter engineering mode: Ctrl + Shift + F12 with password: messner.
Right-click the Storage Group for the affected host and go to Connect Hosts.
On the right side of the window click the Advanced button.
Make sure all paths are present and have a check mark next to them.
Reconnect HBAs to Storage Group, Option #2:
In the Enterprise Storage Window, Click on the Hosts tab.
Right-click the problematic host and choose Connectivity Status.
Click the reconnect button.
---all below on jupiter during a window where I can do an outage----

26) ioscan -fC disk
27) insf -e
28) /sbin/init.d/agent stop
29) /sbin/init.d/agent start
30) powermt config
31) powermt display dev=all -- verify that we have devices showing alternate paths (a total of 4) to each used
lun (/db and /morestore). Compare this report to the one generated before work began.
32) powermt save
33) vgextend /dev/vg01 /dev/dsk/c#d#t# /dev/dsk/c#d#t# to add new paths to vg01
34) vgextend /dev/vg01 /dev/dsk/c#d#t# /dev/dsk/c#d#t# to add new paths to vg02
35) stop bigbrother monitoring on jupiter
36) reboot jupiter
37) powermt check (and let it remove the dead paths/devices)