vgextend error while attempting to mirror boot disk

[Jeff Thayer]

I am attempting to mirror the boot disk on a BL60p blade server, runing 11.31, using HP MirrorDisk/UX for mirroring. The blade has 2 identical disks. I already did this successfully on an identical blade, and it works fine. I am following the same cheat sheet I made while I mirrored the 1st blade.

I get the following error when I run:

bash-3.2# vgextend vg00 /dev/disk/disk2_p2
Couldn't open physical volume "/dev/rdisk/disk5_p2":
Verification of unique LVM disk id on each disk in the volume group
vg00 failed.

Researching this error I have only found this reference:

vgimport(1M)
Message Text:
Verification of unique LVM disk id on each disk in the volume group /dev/vgname failed.

Cause:
There are two possible causes for this message:

1.The vgimport command used the –s option, and two or more disks on the system have the same LVM identifier; this can happen when disks are created with BC copy or cloned with dd.
2.LVM was unable to read the disk header; this can happen when you create new logical units on a SAN array.

Recommended Action:
1.Do not use the –s option to vgimport. Alternatively, use vgchgid to change the LVM identifiers on copied or cloned disks.
2.Retry the vgimport command.
--------------------------------------
I have never run the vgimport command on this box. It was simply built from the HP-UX DVD using one of 2 disks. I built the other blade identically, and it mirrored ok.

I think this is the source of the problem:

bash-3.2# vgscan -pv
Physical Volume "/dev/dsk/c0t0d0s2" is not part of a Volume Group

/dev/vg00
/dev/disk/disk5_p2

bash-3.2# strings /etc/lvmconf/vg00.conf
CONFIG01
/dev/vg00
/dev/rdisk/disk5_p2
LVMREC01",
HPLVMBDRG
VGDA0001
VGSA0001G

bash-3.2# diskinfo /dev/rdisk/disk5
io_search failed: I/O tree node does not exist.

bash-3.2# diskinfo /dev/rdisk/disk3
SCSI describe of /dev/rdisk/disk3:
vendor: COMPAQ
product id: BF0728A4BA
type: direct access
size: 71132000 Kbytes
bytes per sector: 512

bash-3.2# diskinfo /dev/rdisk/disk2
SCSI describe of /dev/rdisk/disk2:
vendor: COMPAQ
product id: BF0728A4BA
type: direct access
size: 71132000 Kbytes
bytes per sector: 512

It appears that vg00 thinks it's using disk5, but it is actually using disk3.

How can I fix this error so I can continue the mirroring process?

Thanks in advance!

[Michael Steele_2]

vgreduce -f /dev/vg## will attempt to reduce out any disks without specifying the disk. The -f option is a forced reduction. It's worked well for me in the past and may offer satisfaction to you here.

Regarding what caused this error, did you pvcreate -f -B this mirror before doing anything else? Note the -B option for boot disks is absolutely necessary.

[Sandman!]

Try recreating the /etc/lvmtab file by running vgscan(1M). Save the original lvmtab file just in case its needed again and run vgscan ie:

# cp /etc/lvmtab /etc/lvmtab.orig
# vgscan -a -v

~hope it helps

[Jeff Thayer]

I used pvcreate -B /dev/rdisk/disk2_p2 before the vgextend command.

I just now tried vgreduce -f vg00, but it didn't help.

I had already tried to recreate lvmtab yesterday, but I followed Sandman's advice and tried his commands exactly as he specified:

bash-3.2# mv lvmtab lvmtab.1.30.08

bash-3.2# vgscan -a -v
Creating "/etc/lvmtab".
Physical Volume "/dev/dsk/c0t0d0s2" is not part of a Volume Group

/dev/vg00
/dev/disk/disk5_p2

Scan of Physical Volumes Complete.
*** LVMTAB has been created successfully.
*** Do the following to resync the information on the disk.
*** #1. vgchange -a y
*** #2. lvlnboot -R

I followed up with the vgchange and lvlnboot commands as the system suggested. vgextend still fails with the exact same error:

vgextend vg00 /dev/disk/disk2_p2
Couldn't open physical volume "/dev/rdisk/disk5_p2":
Verification of unique LVM disk id on each disk in the volume group
vg00 failed.

[Michael Steele_2]

Oh, I missed the unique comment. I think you have a problem with your /etc/slvmvg file. Look at this thread:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1197800

[Jeff Thayer]

I looked at that thread. I don't have either /dev/slvmvg or /etc/slvmvg files on this box.

[F Verschuren]

readyin the errors the lvmhadder of your primare root disk is corupted...

can you do a diskinfo /dev/rdisk/disk5_p2
and a vgdisplay -v vg00

to check the vgheader on the disk:

echo '0X2010?2X' | adb /dev/dsk/disk5_p2 | expand | tr -d '' | sed 's/2010:/VGID/'
ps i am not sure this still works on 11.31...

[Jeff Thayer]

bash-3.2# diskinfo /dev/rdisk/disk5_p2
diskinfo: can't open /dev/rdisk/disk5_p2: Permission denied

bash-3.2# vgdisplay -v vg00
--- Volume groups ---
VG Name /dev/vg00
VG Write Access read/write
VG Status available
Max LV 255
Cur LV 9
Open LV 9
Max PV 16
Cur PV 1
Act PV 1
Max PE per PV 4294
VGDA 2
PE Size (Mbytes) 16
Total PE 4285
Alloc PE 4284
Free PE 1
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0
VG Version 1.0.0

--- Logical volumes ---
LV Name /dev/vg00/lvol1
LV Status available/syncd
LV Size (Mbytes) 1792
Current LE 112
Allocated PE 112
Used PV 1

LV Name /dev/vg00/lvol2
LV Status available/syncd
LV Size (Mbytes) 8192
Current LE 512
Allocated PE 512
Used PV 1

LV Name /dev/vg00/lvol3
LV Status available/syncd
LV Size (Mbytes) 2048
Current LE 128
Allocated PE 128
Used PV 1

LV Name /dev/vg00/lvol4
LV Status available/syncd
LV Size (Mbytes) 7792
Current LE 487
Allocated PE 487
Used PV 1

LV Name /dev/vg00/lvol5
LV Status available/syncd
LV Size (Mbytes) 10992
Current LE 687
Allocated PE 687
Used PV 1

LV Name /dev/vg00/lvol6
LV Status available/syncd
LV Size (Mbytes) 1024
Current LE 64
Allocated PE 64
Used PV 1

LV Name /dev/vg00/lvol7
LV Status available/syncd
LV Size (Mbytes) 16000
Current LE 1000
Allocated PE 1000
Used PV 1

LV Name /dev/vg00/lvol8
LV Status available/syncd
LV Size (Mbytes) 12000
Current LE 750
Allocated PE 750
Used PV 1

LV Name /dev/vg00/lvol9
LV Status available/syncd
LV Size (Mbytes) 8704
Current LE 544
Allocated PE 544
Used PV 1


--- Physical volumes ---
PV Name /dev/disk/disk5_p2
PV Status available
Total PE 4285
Free PE 1
Autoswitch On
Proactive Polling On


I'm completely unfamiliar with the method suggested for checking the vgheader on the disk, and some of the syntax didn't work. I substituted small x for the caps in the echo part cause the cap x failed:

bash-3.2# echo '0x2010?2x'|adb /dev/disk/disk5_p2|expand|tr -d ''|sed 's/2010:/VGID/'
adb: warning: /dev/disk/disk5_p2: Unknown file type.
adb: warning: File '/dev/disk/disk5_p2' is not an object file.
0xVGID
0x222cc0ab 0x470c0fa1

[Sandman!]

Could you post the output of the following commands:

# ioscan -funC disk
# ll /dev/rdisk/disk5_p2

Imho /dev/rdisk/disk5_p2 maynot be a device special file

[Jeff Thayer]

bash-3.2# ioscan -funC disk
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE COMPAQ BF0728A4BA
/dev/dsk/c0t0d0 /dev/dsk/c0t0d0s3 /dev/rdsk/c0t0d0s2
/dev/dsk/c0t0d0s1 /dev/rdsk/c0t0d0 /dev/rdsk/c0t0d0s3
/dev/dsk/c0t0d0s2 /dev/rdsk/c0t0d0s1
disk 0 0/1/1/0.1.0 sdisk CLAIMED DEVICE COMPAQ BF0728A4BA
/dev/dsk/c0t1d0 /dev/dsk/c0t1d0s3 /dev/rdsk/c0t1d0s2
/dev/dsk/c0t1d0s1 /dev/rdsk/c0t1d0 /dev/rdsk/c0t1d0s3
/dev/dsk/c0t1d0s2 /dev/rdsk/c0t1d0s1

bash-3.2# ll /dev/rdisk/disk5_p2
crw-r----- 1 bin sys 22 0x000003 Oct 9 16:37 /dev/rdisk/disk5_p2

[Michael Steele_2]

Jim, if I understand this right this is an initial install that succeeded for the primary but now the mirror won't take.

I'd seriously consider reinstalling. This choose the mirror to be the primary and let the O/S sort it out.

[TTr]

Something is really strange here!

Your vgdisplay listing shows that the /dev/rdisk/disk5_p2 is already a member of vg00.

> --- Physical volumes ---
> PV Name /dev/disk/disk5_p2
> PV Status available
> Total PE 4285
> Free PE 1
> Autoswitch On
> Proactive Polling On

You can not be using the same disk to extend vg00 and mirror the boot disk.

Also, your ioscan shows that the valid disk devices are /dev/dsk/c0t0d0s2 and /dev/dsk/c0t1d0s2

> bash-3.2# ioscan -funC disk
> Class I H/W Path Driver S/W State H/W Type > Description
> ==================================
> disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE
> COMPAQ BF0728A4BA
> /dev/dsk/c0t0d0 /dev/dsk/c0t0d0s3
> /dev/rdsk/c0t0d0s2
> /dev/dsk/c0t0d0s1 /dev/rdsk/c0t0d0
> /dev/rdsk/c0t0d0s3
> /dev/dsk/c0t0d0s2 /dev/rdsk/c0t0d0s1
> disk 0 0/1/1/0.1.0 sdisk CLAIMED DEVICE
> COMPAQ BF0728A4BA
> /dev/dsk/c0t1d0 /dev/dsk/c0t1d0s3
> /dev/rdsk/c0t1d0s2
> /dev/dsk/c0t1d0s1 /dev/rdsk/c0t1d0
> /dev/rdsk/c0t1d0s3
> /dev/dsk/c0t1d0s2 /dev/rdsk/c0t1d0s1
>
> bash-3.2# ll /dev/rdisk/disk5_p2
> crw-r----- 1 bin sys 22 0x000003 Oct 9
> 16:37 /dev/rdisk/disk5_p2

The /dev/rdisk/disk5_p2 does not show up in ioscan. Does this blade have an internal RAID controller and the two disks are bound to that /dev/rdisk/disk5_p2 volume? Then the other two disks should not show up in ioscan.

Please compare all of these commands with the "other" blade whre everything worked fine.

Also post the output of "lvlnboot -v"

[Jeff Thayer]

To answer Michael Steele's question; yes this was newly installed on the primary in mid Oct 07, and has been working fine so far, though I don't know how much my customers have been using it. I installed this blade 1st, and then did another identical BL60p blade later, choosing the same installation options. I was able to mirror the 2nd box without errors.

I did have one problem with both of these blades:
Symptoms:
Both NIS and automountint worked originally, but I was getting an rpc error of some sort when starting the automounter. I think it was rpc.statd, but am not sure. I didn't have time to investigate further at that point, and both NIS and automounting
were working then. After vacation and the emergency power shutdown during the 10/07 San Diego wildfires, neither NIS nor automounting worked anymore.

I found this fix on the web:
----------------------------------
HP moderator Dave Olker
Aug 14, 2007 23:40:42 GMT 10 pts
Hi Mario,

I think this is a known problem that was supposed to be fixed prior to 11i v3 shipping
but it may have reappeared somehow. It may be a corrupted /stand/ioconfig file.

Try this:

# /sbin/init.d/nfs.server stop
# /sbin/init.d/autofs stop
# /sbin/init.d/nfs.client stop
# /sbin/init.d/lockmgr stop
# /sbin/init.d/nfs.core stop

Then rebuild your /stand/ioconfig file by issuing the command:

# insf -e

Then see if you can get all the RPC daemons to start normally:

# /sbin/init.d/nfs.core start
# /sbin/init.d/lockmgr start
# /sbin/init.d/nfs.client start
# /sbin/init.d/autofs start
# /sbin/init.d/nfs.server start

Let me know if this works.

Regards,
Dave
--------------------------------
That fixed the problem. As I said, this happened with both of these BL60p blades, the one I mirrored successfully and this one.

I'm not sure if I could get customer permission to rebuild this blade right now. I'd rather avoid that if at all possible.

[Michael Steele_2]

I don't know. This is a mess. You've been running w/out a boot mirror for 2 1/2 months. And now the disk you're trying to vgextend onto is already apart of the vg. I think the problem's between the chair and keyboard.

[Jeff Thayer]

To answer TTR's questions:

This is a BL60p blade and they do not have the HP SmartRaid array controller.

bash-3.2# lvlnboot -v
bash-3.2#
--------------------------------------------
Commands on the other blade that mirrored OK:

vgdisplay -v (partial output)
--- Physical volumes ---
PV Name /dev/disk/disk5_p2
PV Status available
Total PE 4285
Free PE 1
Autoswitch On
Proactive Polling On

PV Name /dev/disk/disk4_p2
PV Status available
Total PE 4285
Free PE 1
Autoswitch On
Proactive Polling On

-bash-3.2# ioscan -funC disk
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================
disk 1 0/1/1/0.0.0 sdisk CLAIMED DEVICE COMPAQ BF0728A4BA
/dev/dsk/c0t0d0 /dev/rdsk/c0t0d0
/dev/dsk/c0t0d0s1 /dev/rdsk/c0t0d0s1
/dev/dsk/c0t0d0s2 /dev/rdsk/c0t0d0s2
/dev/dsk/c0t0d0s3 /dev/rdsk/c0t0d0s3
disk 0 0/1/1/0.1.0 sdisk CLAIMED DEVICE COMPAQ BF0728A4BA
/dev/dsk/c0t1d0 /dev/rdsk/c0t1d0
/dev/dsk/c0t1d0s1 /dev/rdsk/c0t1d0s1
/dev/dsk/c0t1d0s2 /dev/rdsk/c0t1d0s2
/dev/dsk/c0t1d0s3 /dev/rdsk/c0t1d0s3
---------------------------------------------

[TTr]

Whatever it is, the /dev/disk/disk5_p2 is NOT a physical device. It is an HP-UX 11.31 new construct for mapping the "real" physical hardware.

take a look at

http://forums12.itrc.hp.com/service/forums/questionanswer.do?threadId=1160821&admit=109447627+1202179775059+28353475

and at the mentioned docs.

The command to use is

ioscan -m dsf

and that will give you a translation of the new devices to the old ones.