disk IO errors connected with disk mirroring operation

[Constance]

Hi all, earlier this week I posted a message asking for direction in mirroring a disk. When I followed the directions I got errors when mirroring 3 of the 12 lvols. I searched the site and found someone who had had the same errors and the recommended solution was to run a command to test the lvols then if they check out ok there was instructions to fix the mirror problem. Unfortunately there was no info on how to proceed if the disks did not check out. Mine did not check out. The following is how I tested the lvol and the result.

# dd if=/dev/vg00/lvol3 of=/dev/null bs=4096k
dd read error: I/O error
0+1 records in
0+1 records out
#

I am pretty clueless about this box and the OS. If the solution to this issue or further problem definition requires much more than a couple of commands that are pretty much the same on any system I'm going to need detailed instruction. I'm not even sure what info I should pass along to define the issue. If the answer is on the site then I don't know enough to know how to find it, I have tried and I'm coming up dry. General direction and links to detailed docs are great.
Any help I can get is greatly appreciated.
Thanks,
Constance Cate

[S.K. Chan]

Can you provide these output .. (this will help give us more details)

1) # vgdisplay -v vg00
2) # strings /etc/lvmtab
3) # cat /etc/fstab
4) # lvdisplay -v /dev/vg00/lvol3
From 2) you would know which devices are on vg00, and please run these on those devices (for example : c2t2d0)

5) # pvdisplay -v /dev/rdsk/c2t2d0

[S.K. Chan]

Sorry .. 5) should be

# pvdisplay -v /dev/dsk/c2t2d0

[Helen French]

Hi,

What is the error you getting while doing the mirror ?

Did you check the file system with 'fsck' ?

what is the output of lvdisplay, pvdisplay and vgdisplay ?

HTH,
Shiju

[Constance]

OK
I did the diag commands and I'm attaching them to this reply.
I did step 4 once for each of the logical volumes that gave me the error.
I did not use the fsck command, I am not famailar with it. (Told ya I'm ignorant ;)
The error when I did the mirror was;

# lvextend -m 1 /dev/vg00/lvol3 /dev/dsk/c3t5d0
The newly allocated mirrors are now being synchronized.This operation will
take some time. Please wait ....
lvextend: Couldn't re-synchronize stale partitions of the logical volume:
I/O error
lvextend: Couldn't resynchronize logical volume "/dev/vg00/lvol3".

[Helen French]

Hi,

Check this out:

http://us-support2.external.hp.com/cki/bin/doc.pl/sid=5761bbcf1b841bd82d/screen=ckiDisplayDocument?docId=200000054370930

HTH,
Shiju

[Patrick Wallek]

My guess is that the disk drive at c3t5d0 is going bad. It has some spots that are apparently bad on it now. It will probably only get worse.

Place a hardware call with HP and have them replace the drive.

Your mirroring should then succeed.

[Helen French]

Hi,

Run STM or dd command on /dev/dsk/c3t5d0 and check for any hardware errors:

# stm

# dd if=/dev/rdsk/c3t5d0 of=/dev/null

The second disk may have some bad blocks in it. If you found any hardware errors, replace the disk.

HTH,
Shiju

[S.K. Chan · Accepted solution]

This is what I've concluded
- All lvols are mirrored (c3t6d0 is primary and c3t5d0 is secondary)
- You got 3 stale LVs (lvol3 - /; lvol7 - /usr and lvol10 - /var/opt/OV)
- All of the stale extents are in c3t5d0.

This is what you should do ..
o Shutdwon the system
o Replace c3t5d0 (make sure same model disk and SCSI id)
o Boot up the system all the way to multiuser mode but escape the quorum check.
ISL > hpux -lq
o Then run these to synchronize both disks..
# mkboot -l /dev/rdsk/c3t5d0
# mkboot -a "hpux -lq" /dev/rdsk/c3t5d0
# vgcfgrestore -n vg00 /dev/rdsk/c3t5d0
=> recovers LVM info to disk
# vgchange -a y vg00
=> enables new root disk
# vgsync vg00
=> configure LVs and copies data from primary disk
=> this will take awhile
o You should be good to go ..

[Constance]

Wow. Wow. Bad disk.
Who knew? ;)
NOT what I wanted to hear. I'll try the solution with replacing the disk.
Once I have the hardware in I may post again if I can't figure out from my own research the procedure about the quorum thing.
Thanks for all your help with this.
Constance

[S.K. Chan]

Since you got 2 disks in vg00, if one disk is "dead" you will not be able to activate the VG after the system reboot. By telling it to escape the quorum check, at least you can boot up the system to multiuser mode and do you fixing later. You can choose to boot up in LVM maintenance mode ..
ISL> hpux -lm
and do all your work there too but user would not be able to access the system after you've done with it.

[Helen French]

Hi,

Another thing to make sure is that you always have a good full backup of the system and good RECOVERY tapes ( make_tape_recovery ). In case of any failures, you can always restore your system back !

Read man pages of 'hpux' for getting more about the quorum check.

HTH,
Shiju

[Constance]

I'm thinking that until I get the bad disk replaced I would like to undo what I did to add the new (bad) disk to vg00. How would I go about that so as not to cause myself problems?

[S.K. Chan]

In my opinion you need not undo what you did. The reason for that is it can get messy with the lvreduce'ing now that you got a bad disk. Even if you've "cleanly" reducing those LVs you have to start the mirroring process all over again after the new disk is replace. Just leave it as it is, replace the disk and follow the process. Hope eveything goes smoothly ..:)

[Helen French]

Hi,

You don't need to "undo" all the things, unless you want to. This won't cause any problems for you. This is just like replacing a bad mirrored root disk.

HTH,
Shiju

[Constance]

Hang on, this is likely to be a really stupid question...
Can any of the "bad" from the bad disk move to the good one?
That's why I was gonna un-do the mirror.

[Helen French]

Hi,

Not a stupid question !

Here you are going to replace the disk, from which you got errors while mirroring. So there is no chance that the mirror HDD ( second one) is currupting the primary HDD. The data will be synchronized after the mirroring is successfully done.

Another thing is, if your HDD has a currupted file/block ( by any chance ), undo mirroring will not store the original data back to the disk. If some thing happen like this, you have to restore the file from backup.

HTH,
Shiju