Resync after failure problem

[Igor I. Shulz]

I have /dev/vg00 on /dev/dsk/c0t6d0 /dev/dsk/c2t6d0 with 8 lvolumes, each is mirrored by HP MirrorDisk/UX. I phisically detach one of the disks (/dev/dsk/c2t6d0). This member of vg00 goes to "unavailable". Now I attach the disk back, run ioscan, to see if it is CLAIMED (it is), and run vgsync /dev/vg00. But it fails:

# vgsync /dev/vg00
vgsync: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
vgsync: Couldn't resynchronize logical volume "/dev/vg00/lvol1".
vgsync: Couldn't resynchronize volume group "/dev/vg00".

But if I than reboot server, this pv goes online and resync procedure is completed as needed.

My question: how I can force to resync mirrored vg after returning broken disk to working state without rebooting?

[Eugeny Brychkov · Accepted solution]

Igor,
after running ioscan try
vgchange -a y /dev/vg00
You know vg00 can not be deactivated, but can be reactivated
Hope this will help
Eugeny

[Robert-Jan Goossens]

Hi,

Here is copy of doc KBAN00000347 covers all possibilities, mirrored boot disk, non-mirrored boot disk, mirrored data disk, non-mirrored data disk, etc.

http://forums.itrc.hp.com/cm/components/FileAttachment/0,,0x677db47b9a27d6118ff40090279cd0f9,00.txt

Robert-Jan.

[Igor I. Shulz]

1. vgchange -a y /dev/vg00 doesn't help:

# vgchange -a y /dev/vg00
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c2t6d0":
Unknown error
Volume group "/dev/vg00" has been successfully changed.
#

2. KBAN00000347 is very usefull document but it didn't help me much: it stated that "reboot may not be necessary in case of hot-swap disk" (it's my case). And for resync I should run
# vgcfgrestore -n /dev/vg00 /dev/rdsk/cNdNtN
# vgchange -a y /dev/vg00

but it doesn't work:

# ioscan -fnC disk
Class I H/W Path Driver S/W State H/W Type Description
===========================================================================
disk 0 0/0/0/2/0.6.0 sdisk CLAIMED DEVICE HP 36.4GST336752LC
/dev/dsk/c0t6d0 /dev/rdsk/c0t6d0
disk 2 0/0/0/3/0.6.0 sdisk CLAIMED DEVICE HP 36.4GST336752LC
/dev/dsk/c2t6d0 /dev/rdsk/c2t6d0
................
# vgcfgrestore -n /dev/vg00 /dev/rdsk/c2t6d0
vgcfgrestore: IO error on Physical Volume device /dev/rdsk/c2t6d0: #
# vgchange -a y /dev/vg00
vgchange: Warning: Couldn't attach to the volume group physical volume "/dev/dsk/c2t6d0":
Unknown error
Volume group "/dev/vg00" has been successfully changed.
#

But if I reboot server this disk is attached to VG00 and synced properly :-(

[Robert-Jan Goossens]

Take a look at next document

http://www4.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000063238152

Robert-Jan.

[Rajeev Shukla]

Does it always complain about /dev/vg00/lvol1.
Can you try to sync other logical volumes using "lvsync" command. If other logical volumes get synced and only /dev/vg00/lvol1 has problem, i sucpect the Disk. That PE might be bad and gets stale. Try this with other disk then.

Rajeev

[T G Manikandan]

What does diskinfo on that disk give

First do a
#diskinfo /dev/rdsk/c2t6d0

to check the disk is working fine returning the proper number of bytes.

[T G Manikandan]

What is the output of
pvdisplay -v /dev/vg00/c2t6d0

You should get the logical volumes that were under this disk.
Also
a lvdisplay of the logical volumes which were under c2t6d0.

lvdisplay -v /dev/vg00/lvol..

REvert

[Igor I. Shulz]

To Rajeev Shukla:
lvsync for other volumes doesn't work too:

# lvsync /dev/vg00/lvol3
lvsync: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
lvsync: Couldn't resynchronize logical volume "/dev/vg00/lvol3".
#

To T G Manikandan:

It seems disk works fine:
# diskinfo /dev/rdsk/c2t6d0
SCSI describe of /dev/rdsk/c2t6d0:
vendor: HP 36.4G
product id: ST336752LC
type: direct access
size: 35566480 Kbytes
bytes per sector: 512

pvdisplay shows:

# pvdisplay /dev/dsk/c2t6d0
--- Physical volumes ---
PV Name /dev/dsk/c2t6d0
VG Name /dev/vg00
PV Status unavailable
Allocatable yes
VGDA 2
Cur LV 9
PE Size (Mbytes) 8
Total PE 4340
Free PE 2602
Allocated PE 1738
Stale PE 79
IO Timeout (Seconds) default
Autoswitch On

# lvdisplay -v /dev/vg00/lvol1
--- Logical volumes ---
LV Name /dev/vg00/lvol1
....
Mirror copies 1
..
--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t6d0 38 38
/dev/dsk/c2t6d0 38 38

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c0t6d0 00000 current /dev/dsk/c2t6d0 00000 stale
00001 /dev/dsk/c0t6d0 00001 current /dev/dsk/c2t6d0 00001 stale
........

[Rajeev Shukla]

Hi Iqor,
I see both the disks on a different SCSI bus. Can you confirm about the SCSI termination on the other bus of whose disk shows stale.
It could be issues with bad SCSI terminator or no termination.

[Igor I. Shulz]

Hmm. These are pre-installed by HP internal disks in our just delivered (a week ago) new rp8400 box. Is it possible that such famous and reliable company installed new disks in new system inproperly? Anyway I don't know how I can make sure that termination is set correctly. Besides - this is only problem concerning disk system that I have now on the system.

[Rajeev Shukla]

Though the disks are internal the buses will be terminated externally. Try swapping the terminator and see if the problem persistists with the same disk or goes to another disk which can confirm about the terminator.

[harry d brown jr]

Igor,

How about posting the output of

vgdisplay -v vg00

And you don't have a termination issue. You probably need to "force" the resync, but first I want to see the output from above.

live free or die
harry

[Armin Feller]

Hi,

perhaps there are entries in the bad block area. Is the 'bad block' relocation switched on for the lvols (lvdisplay)?

Please check if you can read from the disk without errors:

dd if=/dev/dsk/c2t6d0 of=/dev/null bs=1k

If the 'dd' fails, perhaps you have to re-add this disk to vg00.

Regards,
Armin

[Igor I. Shulz]

Wow!!!

This is an answer from Great Guru. Problem is gone!

Hello Igor,

Please see below reply on your problem with mirror disks.

For your case may to try solve problem by two ways:

1. to try spin-up inserted disk with next command (diskinfo, scsictl,
pvcreate), then vgsync.

but better

2. install next patch PHKL_28096.

Best regards,
Sergei