topic Re: RX6600 root mirror rebuild steps in Operating System - HP-UX

RX6600 root mirror rebuild steps

J. Deverel — Fri, 14 Jan 2011 22:11:13 GMT

First time in my hpux admin career, I have been given an rx6600 which is supposedly have a boot mirror failure. I have investigated forums her and elsewhere, but can not locate a definitive document or better a cheat sheet, showing the step-by-step attack plan for this. On PA RISC machines, I have perfected this down to a script but on IA64 with s0, s1 s2 etc. slicing, I am as lost as someone in Sahara desert without a compass.

my "vgdisplay -v vg00" says I have these two drives in this vg:

--- Physical volumes ---
PV Name /dev/dsk/c1t0d0s2
PV Status available
Total PE 4346
Free PE 1536
Autoswitch On
Proactive Polling On

PV Name /dev/dsk/c1t1d0s2
PV Status available
Total PE 4356
Free PE 1546
Autoswitch On
Proactive Polling On

sasmgr get_info -D /dev/sasd0 -v -q lun=all -q lun_locate

returns this output

LUN LUN HW Path Enc Bay Locate LED
=== =========== === === ==========
/dev/rdsk/c1t0d0 0/4/1/0.0.0.0.0 1 5 OFF
/dev/rdsk/c1t1d0 0/4/1/0.0.0.1.0 1 6 OFF
/dev/rdsk/c1t2d0 0/4/1/0.0.0.2.0 1 7 OFF

and none of the logical volumes in vg00 are reporting a stale extent as a result of scanning the lvdisplay -v output ran against each lv.

Am I going crazy here trying to figure out which drive has failed ? I was basically got handed this task of replacing the failed root mirror disk, late yesterday and I spent my whole day today trying to figure it out with no success. I am strongly thinking, I am a victim of an overzealous operation monitoring team member.

Any ideas ?

Thanks.

Re: RX6600 root mirror rebuild steps

Mel Burslan — Fri, 14 Jan 2011 22:31:03 GMT

This excerpt below is from HP document number KBRC00014036. I am not sure if it is still available or you can find it with your credentials here at the ITRC site but here is the information you need, below.

But, as you have guessed, the root disk and its mirror looks fine from the output you provided and analysis you made. Check other volume groups to see if they have any stale extents. Also, in the front panel, visually check to see how many disk drives this server has. It definitely have disk numbers 5, 6 and 7 out of the 8 slots. If you only have 3 drives, which is kind of odd but not totally out of question, you do not have any failed drives. If you have drives in any other disk slots, those are the failed disks.

Hope this helps.

Check the document below...

C) Replacing a root disk on Itanium. Partition the new disk and populate.

1. Use vi to create a partition description file by doing the following:

# vi /tmp/partitionfile

The 3 line entry for 11.22 should look like this:

2
EFI 100MB
HPUX 100%

The 4 line entry for 11.23 should look like this:

3
EFI 500MB
HPUX 100%
HPSP 400MB

2. Use idisk to setup the disk partitioning using the file created above:

# idisk -wf /tmp/partitionfile /dev/rdsk/cXtXdX

NOTE: There will be a prompt with a message saying the operation may be
destructive and asks to continue. Be sure to answer 'yes' for the
operation to be successful. If the prompt is answered with 'y' only,
an error is received along with a message saying "user aborting".

3. Use insf to create the new device dfiles (cXtXdXs1, cXtXdX2 and cXtXdXs3)

# insf -e

4. Use mkboot to format and populate the newly created EFI partition:

# mkboot -e -l /dev/dsk/cXtXdX

D) Use standard LVM commands to restore the mirror information. For root
disks, be careful to specify the s2 device file (cXtXdXs2) when
performing LVM commands.

# vgcfgrestore -n vg00 /dev/rdsk/cXtXdXs2
# vgchange -a y vg00
# vgsync vg00
# lvlnboot -r /dev/vg00/lvol3 /dev/vg00
# lvlnboot -b /dev/vg00/lvol1 /dev/vg00
# lvlnboot -s /dev/vg00/lvol2 /dev/vg00
# lvlnboot -d /dev/vg00/lvol2 /dev/vg00

# lvlnboot -v (to verify the results)
# vgdisplay -v /dev/vgxx (to verify the results)

E) Modify the contents of NVRAM using the setboot(1M) command.

If an EFI boot option entry (held in NVRAM) exists for the replaced disk,
it is still linked to the old disk GUID. Use setboot with the appropriate
option and argument to modify NVRAM and point it to the replacement disk.

For Primary boot disk:

1. setboot -p
2. for HA Alternate boot disk:
3. setboot -h
4. and for alternate boot disk:
5. setboot -a

F) VERIFY YOUR WORK!!!

* Manually boot the system using each boot option.
* Run the following command after booting HP-UX. # lvlnboot -v (to verify both disks are setup as bootable) # vgdisplay -v vg00 (Each LV should be "SYNCd" and "Used PV = 2")

Re: RX6600 root mirror rebuild steps

donna hofmeister — Fri, 14 Jan 2011 23:01:45 GMT

i believe what mel said is correct.

however, i think you need to distinguish between 11.23 and 11.31 notation when referring to your disk(s):

/dev/rdsk/cXtXdXs2 # for 11.23
/dev/rdsk/cXtXdX_p2 # for 11.31

Re: RX6600 root mirror rebuild steps

J. Deverel — Sun, 16 Jan 2011 01:28:56 GMT

Thanks to you both for the answers. My OS is v.11.23 so the procedure is on the spot for me. Now I am facing another issue. I replaced the disk drive with the replacement sent by hardware support. Now I can not see it neither in ioscan nor in sasmgr outputs. My sasmgr command from my original post, comes back with the same exact output. On my ioscan output the replaced drive path comes up with no_hw even though I rescanned the bus with ioscan -fn command. Is there any additional command I need to run to make this drive recognized by the server ?

Oh by theway, thanks for the confirmation, it was not part of vg00 volume group. As a matter of fact, it was not being used at all in any other volume group but there are plans to incorporate it into some other operation on this server. So, it is crucial to make it recognized by the server. Any further help is appreciated.

Re: RX6600 root mirror rebuild steps

Ranveer — Sun, 16 Jan 2011 14:59:05 GMT

Please invest some time in this "Famous" doc:
Sure you will get a lot info for your support process!

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c01911837/c01911837.pdf

Re: RX6600 root mirror rebuild steps

TTr — Sun, 16 Jan 2011 16:34:30 GMT

> an rx6600 which is supposedly have a boot mirror failure

How was this determined? What was the error?

Is there another volume group other than vg00?

A listing of ioscan -fn would help.

And of "sasmgr get_info -D... -q phy=all

Re: RX6600 root mirror rebuild steps

nijokj — Mon, 17 Jan 2011 04:07:59 GMT

you should try

insf -C disk for creating device files newly installed disk drive.

Re: RX6600 root mirror rebuild steps

Zinky — Mon, 17 Jan 2011 05:36:06 GMT

The rx6600 has a SmatArray controller. Your vg00 likely sits on top of a HW raid volume out of that controller:

Do a:

sasmgr -N get_info -D /dev/sasdN -q raid

for all /dev/sasdN (sasd0, sasd1...)

Re: RX6600 root mirror rebuild steps

J. Deverel — Mon, 17 Jan 2011 14:53:00 GMT

Thanks again. Here is the output from ioscan -fnC disk command:

Class I H/W Path Driver S/W State H/W Type Description
====================================================================================
disk 0 0/0/2/1.0.16.0.0 sdisk CLAIMED DEVICE TEAC DV-28E-N
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
disk 1 0/4/1/0.0.0.0.0 sdisk CLAIMED DEVICE HP DG146ABAB4
/dev/dsk/c1t0d0 /dev/rdsk/c1t0d0
/dev/dsk/c1t0d0s1 /dev/rdsk/c1t0d0s1
/dev/dsk/c1t0d0s2 /dev/rdsk/c1t0d0s2
/dev/dsk/c1t0d0s3 /dev/rdsk/c1t0d0s3
disk 2 0/4/1/0.0.0.1.0 sdisk CLAIMED DEVICE HP DG146ABAB4
/dev/dsk/c1t1d0 /dev/rdsk/c1t1d0
/dev/dsk/c1t1d0s1 /dev/rdsk/c1t1d0s1
/dev/dsk/c1t1d0s2 /dev/rdsk/c1t1d0s2
disk 3 0/4/1/0.0.0.2.0 sdisk CLAIMED DEVICE HP DG146ABAB4
/dev/dsk/c1t2d0 /dev/rdsk/c1t2d0
disk 4 0/4/1/0.0.0.3.0 sdisk NO_HW DEVICE HP DG146ABAB4
/dev/dsk/c1t3d0 /dev/rdsk/c1t3d0
disk 5 0/4/1/0.0.0.4.0 sdisk CLAIMED DEVICE HP DG146ABAB4
/dev/dsk/c1t4d0 /dev/rdsk/c1t4d0

and I only have one sas controller as follows:

escsi_ctlr 0 0/4/1/0 sasd CLAIMED INTERFACE HP PCI/
PCI-X SAS MPT Adapter
/dev/sasd0
ext_bus 1 0/4/1/0.0.0 sasd_vbus CLAIMED INTERFACE SAS Devi
ce Interface

and the sasmgr command Alzhy suggested, gave an error as follows:

# sasmgr -N get_info -D /dev/sasd0 -q raid
sasmgr: illegal option -- N
sasmgr [-h] get_info -D
[-h] get_info -D -q raid
[-h] get_info -D -q phy={all | }
[-h] get_info -D -q phy_in_port={all | }
[-h] get_info -D -q target={all | }
[-h] get_info -D -q vpd
[-h] get_info -D -q pci_id
[-h] get_info -D -q smp_addr
[-h] get_info -D -q lun={all | }
[-v][-h] get_info -D -q lun={all | }
-q lun_locate
[-h] get_info -D -q reg={all | }
[-q reg_type={pci_config | mmio}]

since I am not very familiar with sas controllers.

and finally @ nijoki:
I believe you did not understand my issue here. ioscan shows no_hw for the device path I just replaced. What do you think that an insf -C will do for me ?

Re: RX6600 root mirror rebuild steps

Torsten. — Mon, 17 Jan 2011 14:55:47 GMT

>> The rx6600 has a SmatArray controller

SmartArray P400 is optional!
It has an LSI chip that presents the "plain" disks or manages a hardware mirrored pair of disks.
The sasmgr command will show you what you have.

In case of "plain" disks there are many steps needed, unfortunately my stored link is dead ...

Re: RX6600 root mirror rebuild steps

J. Deverel — Mon, 17 Jan 2011 15:06:34 GMT

Oh by the way, when I skipped the offending -N switch on sasmgr command provided by Alzhy, here is the output I got:

# sasmgr get_info -D /dev/sasd0 -q raid

Mon Jan 17 06:57:02 2011

---------- PHYSICAL DRIVES ----------
LUN dsf SAS Address Enclosure Bay Size(MB)

/dev/rdsk/c1t0d0 0x5000c500040a56d9 1 5 140014
/dev/rdsk/c1t1d0 0x5000c500040a263d 1 6 140014
/dev/rdsk/c1t2d0 0x5000c5000409a035 1 7 140014
/dev/rdsk/c1t4d0 0x5000c50008cf3055 1 8 140014

after running this command, lun_locate was able to figure out the new drive in bay #8, as follows:

# sasmgr get_info -D /dev/sasd0 -v -q lun=all -q lun_locate
LUN LUN HW Path Enc Bay Locate LED
=== =========== === === ==========
/dev/rdsk/c1t0d0 0/4/1/0.0.0.0.0 1 5 OFF
/dev/rdsk/c1t1d0 0/4/1/0.0.0.1.0 1 6 OFF
/dev/rdsk/c1t2d0 0/4/1/0.0.0.2.0 1 7 OFF
/dev/rdsk/c1t4d0 0/4/1/0.0.0.4.0 1 8 OFF

but my ioscan still lists the hardware path to this drive as no_hw.

Re: RX6600 root mirror rebuild steps

Torsten. — Mon, 17 Jan 2011 15:11:43 GMT

>>

disk 4 0/4/1/0.0.0.3.0 sdisk NO_HW DEVICE HP DG146ABAB4
/dev/dsk/c1t3d0 /dev/rdsk/c1t3d0
disk 5 0/4/1/0.0.0.4.0 sdisk CLAIMED DEVICE HP DG146ABAB4
/dev/dsk/c1t4d0 /dev/rdsk/c1t4d0

I'm sure the bad disk was c1t3d0, but the new disk is c1t4d0.

This is normal for these SAS disks, a new disk (new identifier) gets a new device file.

What you need to do now is

# sasmgr replace_tgt -D /dev/sasd0 -q old_dev=/dev/dsk/c1t3d0 -q new_tgt_hwpath=0/4/1/0.0.0.4.0

Make sure you did run "pvcgange -a n /dev/rdsk/c1t3d0" before.

Re: RX6600 root mirror rebuild steps

nijokj — Mon, 17 Jan 2011 19:09:33 GMT

As Torsten's above reply ,sometimes if you replace(insert) a disk on the same hardware path it is not necessary that you will get the same device file , in that case if you run insf -C disk then it will create device file only for newly inserted disk.This way you can identify the newly inserted disk only if your replacement is online.

Re: RX6600 root mirror rebuild steps

Torsten. — Tue, 18 Jan 2011 05:44:36 GMT

A new disk get always a new device file if connected to this controller.

Re: RX6600 root mirror rebuild steps

J. Deverel — Tue, 18 Jan 2011 23:46:13 GMT

Thanks for all information