IO problem or ??

[S.K. Chan · Accepted solution]

At a glance it seems you have 2 problems ..
1) Possible IO (from "LVM .. POWERFAILED")
2) Network link (from "MII Link ...")

The "LVM..POWERFAILED" error is accompanied by "SCSI Request Timeout" error and this usually means you got some kind of SCSI connection/termination problem. Check for loose SCSI cable, loose SCSI terminator, etc. ANd from the message it's pointing to c4t15d0 (1f04f000). I do not see this error ...

"pv# return to vg#"

which is good because if that appears the likelihood of a bad disk is high, given we rule out low timeout value of the disk driver.
Anyway you have to run STM to confirm your findings...
# cstm
cstm> map
cstm> sel dev
cstm> info
cstm> infolog

See if there is any error here. If not run exerciser. I can't remember the syntax but you can run "help" to list the command ..

cstm> help

The exerciser tool will run for a while. Run "map" occasionally to check its status. Once it is completed, view the exerciser log. Again use "help" to find out the exact syntax. If you can't see any error here, then it's due to SCSI connection/termination like I mentioned earlier.

[Daniel_25]

Thanks for your respond Chan,.
you 're right

one of the disk is not detectek by system
disk 13 0/3/0/0.14.0 sdisk CLAIMED DEVICE SEAGATE ST39175LC
/dev/dsk/c4t14d0 /dev/rdsk/c4t14d0
disk 8 0/3/0/0.15.0 sdisk NO_HW DEVICE SEAGATE ST39173WC
/dev/dsk/c4t15d0 /dev/rdsk/c4t15d0
disk 10 0/7/0/0.12.0 sdisk CLAIMED DEVICE IBM DGHS09Y
/dev/dsk/c6t12d0 /dev/rdsk/c6t12d0
disk 9 0/7/0/0.13.0 sdisk CLAIMED DEVICE IBM DGHS09Y
/dev/dsk/c6t13d0 /dev/rdsk/c6t13d0
disk 5 0/7/0/0.14.0 sdisk CLAIMED DEVICE SEAGATE ST39236LC
/dev/dsk/c6t14d0 /dev/rdsk/c6t14d0
disk 3 0/7/0/0.15.0 sdisk CLAIMED DEVICE SEAGATE ST39236LC
/dev/dsk/c6t15d0 /dev/rdsk/c6t15d0

Maybe this is scsi termination problem or the hardisk problem,..right?

But question again :-),..after I fix it ,.what should I do ?,..this disk is mirror copies of /dev/vgoracle/lvol1.

If the problem is the disk,..what should I do to get it back on again as mirror copies of /dev/vgoracle/lvol1 ?

Do I have to reboot the machine (..unfortunately this is production machine )

[Tom Geudens]

Hi,
Oh Oh ... not good.
Are your disks connected to the server with a hub/switch ? If so, check the hub/switch loggings for a link which has a lot of transmit/receive errors (and reconnects as well probably). That's the first thing you have to do, locate which link is failing. Next, you'll have to detect which component in the link is bad. This can be the card in the server, the cable to the hub, the port in the hub, the cable to the disks. In the best case scenario, somebody pulled one of the cables loose.

I don't want to make you panic, but I would address this problem as swiftly as possible !

Regards,
Tom

[Daniel_25]

Hi Tom,

I think that error comes from another app,..that we use veritas...

But again,..
How do I recover the disk ( if the disk is damage ) & put it back again as mirror coppies of one lvol,..need reboot ?

Thanks

[Tom Geudens]

Hi,
Sorry, didn't see your answer ...
First, lvreduce the logical volume (to remove the mirror)
lvreduce -m 0 /dev/vgoracle/lvol1 /dev/dsk/c4t15d0
Then, vgreduce the volumegroup
vgreduce /dev/vgoracle /dev/dsk/c4t15d0

Replace the disk ...
If you find it in an ioscan ... the problem was the disk.

Next, vgextend the volumegroup
vgextend /dev/vgoracle /dev/dsk/c4t15d0
And finally, recreate the mirror
lvextend -m 1 /dev/vgoracle/lvol1 /dev/dsk/c4t15d0

Hope this helps,
Tom

[Tom Geudens]

Hi,
Extra add-on :-)
After you replace the disk and ioscan it, you also have to pvcreate it before you can add it to the volumegroup.

Regards,
Tom

[S.K. Chan]

If the disk is not detected you won;t be able to do lvreduce, vgreduce and all that, so DO NOT do that, it'll hang !
Since you system is still running fine you're probably ok for the time being until you can schedule a downtime to get c4t15d0 replaced. What you need to do is ..
o Shutdown and replace damaged disk.
o Boot up in single user mode ..
ISL> hpux -is (;0)/stand/vmunix

Then do this to recover your LVM structure to the new disk..
# pvcreate -f /dev/rdsk/c4t15d0
# vgcfgrestore -n /dev/vgoracle /dev/rdsk/c4t15d0
# vgchange -a y /dev/vgoracle
# vgsync /dev/vgoracle
# shutdown -r 0

At set !! Thank goodness you got mirroring !

[S.K. Chan]

Since you system is still running fine you're probably ok for the time being until you can schedule a downtime to get c4t15d0 replaced. What you need to do is ..
o Shutdown and replace damaged disk.
o Boot up in single user mode ..
ISL> hpux -is (;0)/stand/vmunix

Then do this to recover your LVM structure to the new disk..
# pvcreate -f /dev/rdsk/c4t15d0
# vgcfgrestore -n /dev/vgoracle /dev/rdsk/c4t15d0
# vgchange -a y /dev/vgoracle
# vgsync /dev/vgoracle
# shutdown -r 0

At set !! Thank goodness you got mirroring ! Assumption made..
1- all lvols in c4t15d0 have 1 mirror copy
2- your /etc/lvmconf/vgoracle.conf file is good

[S.K. Chan]

sorry ... for the double posting ..

[Michael Tully]

If the disk is replaced, the system will still think that the disk exists in /etc/lvmtab. You should only need to run these commands.

# vgcfgrestore -n /dev/vgoracle /dev/rdsk/c4t15d0
# vgchange -a y /dev/vgoracle
# vgsync /dev/vgoracle

[Tom Geudens]

Hi,
My apologies I did put you on the wrong track ... my procedure would indeed only work if the disk is still available.

S.K. and Michael, thanks for correcting me (without flaming me - which would have been just). I printed Michael's version because I want to try that myself.

Daniel, can you make sure these guys get full marks ?

Regards,
Tom

P.S. N/A for all my answers to this post please.