topic Re: problem encoutered dealing with rx6600 failed disk replacement. in Operating System - HP-UX

problem encoutered dealing with rx6600 failed disk replacement.

stephen peng — Wed, 17 Nov 2010 10:06:24 GMT

guys,
today I tried to replace the failed disk of that rx6600, lvdisplay of lvol8 told that there were two stale PEs:
00063 /dev/dsk/c3t0d0s2 00912 stale /dev/dsk/c3t1d0s2 01056 current
00070 /dev/dsk/c3t0d0s2 00919 stale /dev/dsk/c3t1d0s2 01063 current
and PEs on c3t1d0s2 were all ok(every lv), there were 2 free disks on that rx6600, so I tried to add one(c3t3d0s2) to vg00 and lvextend all lvs to c3t3d0s2, then:
lvextend: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
lvextend: Couldn't resynchronize logical volume "/dev/vg00/lvol8",
and lvdisplay -v /dev/vg00/lvol8 showed that
00063 /dev/dsk/c3t0d0s2 00912 stale /dev/dsk/c3t1d0s2 01056 current /dev/dsk/c3t3d0s2 01056 stale
00070 /dev/dsk/c3t0d0s2 00919 stale /dev/dsk/c3t1d0s2 01063 current /dev/dsk/c3t3d0s2 01063 stale
then c3t1d0 producing numberic stale PEs in serveral lvs, and then synced by os, pv status of c3t1d0 showed that it was unavailable, and it became available when I pulled it out and then inserted it in again.
so, c3t0d0 had failed PE, and c3t1d0 was failing, there were some files unreadable under /var, and I could not lvreduce c3t0d0 and c3t3d0 from vg00, telling:
lvreduce: Couldn't reduce the logical volume:
Device busy
lvreduce: The LVM device driver failed to reduce mirrors on
the logical volume "/dev/vg00/lvol8".
is there any way to fix this problem, not needing os restore? there was problem when I tried make_tape_recovery.

it is really urgent for me, thank you for your replay!
stephen

Re: problem encoutered dealing with rx6600 failed disk replacement.

Raj Briden — Wed, 17 Nov 2010 10:15:47 GMT

try pvmove command

Re: problem encoutered dealing with rx6600 failed disk replacement.

Anoop Sivan — Wed, 17 Nov 2010 12:45:06 GMT

Re: problem encoutered dealing with rx6600 failed disk replacement.

stephen peng — Wed, 17 Nov 2010 12:59:54 GMT

Anoop,
1,HP-UX CHNFSCC2 B.11.23 U ia64 2794265756 unlimited-user license
2,before
LV Name /dev/vg00/lvol8
LV Status available/stale
LV Size (Mbytes) 8704
Current LE 272
Allocated PE 544
Used PV 2
after I add c3t3d0:
LV Name /dev/vg00/lvol8
LV Status available/stale
LV Size (Mbytes) 8704
Current LE 272
Allocated PE 816
Used PV 3
3,Mirror copies 2
Allocation strict
4,#diskinfo /dev/rdsk/c3t3d0
describe of /dev/rdsk/c3t3d0:
14Hvendor: HP
10Hproduct id: DG146ABAB4
16Htype: direct access
16Hsize: 143374744 Kbytes
4Hbytes per sector: 512

I did not record c3t1d0's diskinfo, but I am sure that c3t1d0 was a disk with problem.

Re: problem encoutered dealing with rx6600 failed disk replacement.

SoorajCleris — Wed, 17 Nov 2010 13:00:40 GMT

Hi,

It seems original disk contains badblock area. Do you have a valid ingnite backup?

It seems one of the disk has issue ( badblock ). I have seen this issue before .

The solution was replacing the disks one at a time (following root disk replacement procedure) , Check which disk has issue.

Remember to keep one original disk when you replace the other one. you may plan downtime.

Regards,
Sooraj

Re: problem encoutered dealing with rx6600 failed disk replacement.

stephen peng — Wed, 17 Nov 2010 13:11:00 GMT

Sooraj,
I could not lvreduce lvol8 from c3t0d0 or c3t3d0, so I could not use regular root disk replacement procedure. Do you mean vgcfgrestore?
And I don't think /var was consistent now, would it be filesystem consistency problem?
It is a critical choice which maybe lead to os re-install, and I don't have a valid os backup and it does not seem to be achievable to backup the os at present situation, I tried, but failed.

Re: problem encoutered dealing with rx6600 failed disk replacement.

Anoop Sivan — Wed, 17 Nov 2010 13:48:30 GMT

Hi Stephen,

Here you have to the use pv-key of the failed disk to reduce all those from vg00.

lvdisplay -v -k /dev/vg00/lvol1

Note the pv-key for the failed disks
(Insted of disk name it shows pv-key)

first reduce (disc-c3t0d0) use the key

lvreduce -A n -m 0 -k /dev/vg00/lvol1 0
-->(-m "0" will be changed)

like remove all lvol mirrors

repeat the same procedure on c3t3d0s2

then

vgreduce -f /dev/vg00

Re: problem encoutered dealing with rx6600 failed disk replacement.

stephen peng — Wed, 17 Nov 2010 14:07:04 GMT

Anoop,
what I met did not like the situation stated in "When_Good_Disks_Go_Bad-WP", so I did not try lvreduce -k as the early option, I want some other opinions. I do concern how to deal with inconsisten /var.

regards
Stephen

Re: problem encoutered dealing with rx6600 failed disk replacement.

DeafFrog — Wed, 17 Nov 2010 23:25:16 GMT

Hi Stephen ,

I think drd clone should go thru on the still to fail (failing disk) c3t1d0 .
Also , i remember there are threads on forum that talks about breaking internal raid on rx6600 on-line and offline , right now i was able to google this
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1215857

Hope this helps.

regards,

Re: problem encoutered dealing with rx6600 failed disk replacement.

stephen peng — Thu, 18 Nov 2010 02:27:57 GMT

DeafFrog,
I don't think I need to break an internal raid.

Re: problem encoutered dealing with rx6600 failed disk replacement.

SoorajCleris — Thu, 18 Nov 2010 03:34:56 GMT

Hi,

Are those disks IR Volumes?

Regards
Sooraj

Re: problem encoutered dealing with rx6600 failed disk replacement.

stephen peng — Thu, 18 Nov 2010 05:46:53 GMT

how to find out whether they were IR?

Re: problem encoutered dealing with rx6600 failed disk replacement.

P Arumugavel — Thu, 18 Nov 2010 06:00:57 GMT

hi,

could you post the output of

#ioscan -fnC disk

rgds..

Re: problem encoutered dealing with rx6600 failed disk replacement.

P Arumugavel — Thu, 18 Nov 2010 06:05:18 GMT

# sasmgr get_info -D /dev/sasd0 -q raid

Re: problem encoutered dealing with rx6600 failed disk replacement.

stephen peng — Thu, 18 Nov 2010 06:58:54 GMT

---------- PHYSICAL DRIVES ----------
5LUN dsf SAS Address Enclosure Bay Size(MB)
5
5/dev/rdsk/c3t0d0 0x5000c50006381011 1 5 140014
5/dev/rdsk/c3t1d0 0x5000c500063a3201 1 6 140014
5/dev/rdsk/c3t2d0 0x5000c500063a3745 1 7 140014
5/dev/rdsk/c3t3d0 0x5000c500063a7db9 1 8 140014

Re: problem encoutered dealing with rx6600 failed disk replacement.

stephen peng — Tue, 23 Nov 2010 17:34:39 GMT

replacement done!

regards
Stephen