problem encoutered dealing with rx6600 failed disk replacement.

[stephen peng]

guys,
today I tried to replace the failed disk of that rx6600, lvdisplay of lvol8 told that there were two stale PEs:
00063 /dev/dsk/c3t0d0s2 00912 stale /dev/dsk/c3t1d0s2 01056 current
00070 /dev/dsk/c3t0d0s2 00919 stale /dev/dsk/c3t1d0s2 01063 current
and PEs on c3t1d0s2 were all ok(every lv), there were 2 free disks on that rx6600, so I tried to add one(c3t3d0s2) to vg00 and lvextend all lvs to c3t3d0s2, then:
lvextend: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
lvextend: Couldn't resynchronize logical volume "/dev/vg00/lvol8",
and lvdisplay -v /dev/vg00/lvol8 showed that
00063 /dev/dsk/c3t0d0s2 00912 stale /dev/dsk/c3t1d0s2 01056 current /dev/dsk/c3t3d0s2 01056 stale
00070 /dev/dsk/c3t0d0s2 00919 stale /dev/dsk/c3t1d0s2 01063 current /dev/dsk/c3t3d0s2 01063 stale
then c3t1d0 producing numberic stale PEs in serveral lvs, and then synced by os, pv status of c3t1d0 showed that it was unavailable, and it became available when I pulled it out and then inserted it in again.
so, c3t0d0 had failed PE, and c3t1d0 was failing, there were some files unreadable under /var, and I could not lvreduce c3t0d0 and c3t3d0 from vg00, telling:
lvreduce: Couldn't reduce the logical volume:
Device busy
lvreduce: The LVM device driver failed to reduce mirrors on
the logical volume "/dev/vg00/lvol8".
is there any way to fix this problem, not needing os restore? there was problem when I tried make_tape_recovery.

it is really urgent for me, thank you for your replay!
stephen

[Raj Briden]

try pvmove command

[Anoop Sivan]

Hi Stephen

Please provide below out puts.

1. uname -a
2. vgdisplay -v vg00 | egrep "LV Status|LV Name|PV Name"
3. lvdisplay -v /dev/vg00/lvol8 | egrep "Mirror copies|Allocation"
4. diskinfo /dev/rdsk/c3t1d0 and c3t3d0

[stephen peng]

Anoop,
1,HP-UX CHNFSCC2 B.11.23 U ia64 2794265756 unlimited-user license
2,before
LV Name /dev/vg00/lvol8
LV Status available/stale
LV Size (Mbytes) 8704
Current LE 272
Allocated PE 544
Used PV 2
after I add c3t3d0:
LV Name /dev/vg00/lvol8
LV Status available/stale
LV Size (Mbytes) 8704
Current LE 272
Allocated PE 816
Used PV 3
3,Mirror copies 2
Allocation strict
4,#diskinfo /dev/rdsk/c3t3d0
describe of /dev/rdsk/c3t3d0:
14Hvendor: HP
10Hproduct id: DG146ABAB4
16Htype: direct access
16Hsize: 143374744 Kbytes
4Hbytes per sector: 512

I did not record c3t1d0's diskinfo, but I am sure that c3t1d0 was a disk with problem.

[SoorajCleris]

Hi,

It seems original disk contains badblock area. Do you have a valid ingnite backup?


It seems one of the disk has issue ( badblock ). I have seen this issue before .

The solution was replacing the disks one at a time (following root disk replacement procedure) , Check which disk has issue.

Remember to keep one original disk when you replace the other one. you may plan downtime.

Regards,
Sooraj

[stephen peng]

Sooraj,
I could not lvreduce lvol8 from c3t0d0 or c3t3d0, so I could not use regular root disk replacement procedure. Do you mean vgcfgrestore?
And I don't think /var was consistent now, would it be filesystem consistency problem?
It is a critical choice which maybe lead to os re-install, and I don't have a valid os backup and it does not seem to be achievable to backup the os at present situation, I tried, but failed.

[Anoop Sivan]

Hi Stephen,

Here you have to the use pv-key of the failed disk to reduce all those from vg00.

lvdisplay -v -k /dev/vg00/lvol1

Note the pv-key for the failed disks
(Insted of disk name it shows pv-key)

first reduce (disc-c3t0d0) use the key


lvreduce -A n -m 0 -k /dev/vg00/lvol1 0
-->(-m "0" will be changed)

like remove all lvol mirrors

repeat the same procedure on c3t3d0s2

then

vgreduce -f /dev/vg00

[stephen peng]

Anoop,
what I met did not like the situation stated in "When_Good_Disks_Go_Bad-WP", so I did not try lvreduce -k as the early option, I want some other opinions. I do concern how to deal with inconsisten /var.

regards
Stephen

[DeafFrog]

Hi Stephen ,

I think drd clone should go thru on the still to fail (failing disk) c3t1d0 .
Also , i remember there are threads on forum that talks about breaking internal raid on rx6600 on-line and offline , right now i was able to google this
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1215857

Hope this helps.

regards,

[stephen peng]

DeafFrog,
I don't think I need to break an internal raid.

[SoorajCleris]

Hi,

Are those disks IR Volumes?

Regards
Sooraj

[stephen peng]

how to find out whether they were IR?

[P Arumugavel]

hi,

could you post the output of

#ioscan -fnC disk

rgds..

[P Arumugavel]

# sasmgr get_info -D /dev/sasd0 -q raid

[stephen peng]

---------- PHYSICAL DRIVES ----------
5LUN dsf SAS Address Enclosure Bay Size(MB)
5
5/dev/rdsk/c3t0d0 0x5000c50006381011 1 5 140014
5/dev/rdsk/c3t1d0 0x5000c500063a3201 1 6 140014
5/dev/rdsk/c3t2d0 0x5000c500063a3745 1 7 140014
5/dev/rdsk/c3t3d0 0x5000c500063a7db9 1 8 140014

[stephen peng]

replacement done!

regards
Stephen