topic Re: Disk Problem in Operating System - HP-UX

Disk Problem

Waqar Razi — Fri, 12 Jun 2009 13:35:49 GMT

The output of the dmesg command is showing some disk problem. Also there is an amber light on the disk on the server. I have checked the syslog, nothing is showing up there. I have checked the ioscan, all disks seem to be CLAIMED there. I have checked the vgdisplay, nothing seems to be stale or Unavailable there.

I am managing this server remotely and the local operators say that they see an amber light on the disk. But I dont see any thing in the syslog or vgdisplay output. But the dmesg output shows some problems with the disk.
Can any one please help me in this regard.

Here is the dmesg output:

LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000484b8800), from raw device 0x1f0e0500 (with priority: 0, and current flags: 0x40) to raw device 0x1f0f0500 (with priority: 1, and current flags: 0x0).
LVM: Recovered Path (device 0x1f0e0500) to PV 4 in VG 1.
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000484b8800), from raw device 0x1f0f0500 (with priority: 1, and current flags: 0x0) to raw device 0x1f0e0500 (with priority: 0, and current flags: 0x0).
LVM: Restored PV 4 to VG 1.
LVM: vg[1]: pvnum=3 (dev_t=0x1f0e0400) is POWERFAILED

SCSI: Write error -- dev: b 31 0x0e0400, errno: 126, resid: 8192,
blkno: 3389016, sectno: 6778032, offset: 3470352384, bcount: 8192.

SCSI: Write error -- dev: b 31 0x0e0400, errno: 126, resid: 8192,
blkno: 168464, sectno: 336928, offset: 172507136, bcount: 8192.
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000484b8040), from raw device 0x1f0e0400 (with priority: 0, and current flags: 0xc0) to raw device 0x1f0f0400 (with priority: 1, and current flags: 0x0).
LVM: Recovered Path (device 0x1f0f0400) to PV 3 in VG 1.
LVM: Recovered Path (device 0x1f0e0400) to PV 3 in VG 1.
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000484b8040), from raw device 0x1f0f0400 (with priority: 1, and current flags: 0x0) to raw device 0x1f0e0400 (with priority: 0, and current flags: 0x80).
LVM: Recovered Path (device 0x1f0e0700) to PV 6 in VG 1.
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000484b8800), from raw device 0x1f0e0500 (with priority: 0, and current flags: 0x40) to raw device 0x1f0f0500 (with priority: 1, and current flags: 0x0).
LVM: Recovered Path (device 0x1f0e0500) to PV 4 in VG 1.
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000484b8800), from raw device 0x1f0f0500 (with priority: 1, and current flags: 0x0) to raw device 0x1f0e0500 (with priority: 0, and current flags: 0x0).
LVM: vg[1]: pvnum=6 (dev_t=0x1f0e0700) is POWERFAILED
lv_readvgdats: Could not read VGDA 1 header & trailer from disk H/W path 1/0/4/0/0.1.18.0.0.0.4 (error = 5)
LVM: Failed to restore PV 3 to VG 1!
LVM: Restored PV 4 to VG 1.
lv_readvgdats: Could not read VGDA 1 header & trailer from disk H/W path 1/0/4/0/0.1.18.0.0.0.7 (error = 5)
LVM: Failed to restore PV 6 to VG 1!
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000484b8040), from raw device 0x1f0e0400 (with priority: 0, and current flags: 0x40) to raw device 0x1f0f0400 (with priority: 1, and current flags: 0x0).
LVM: Recovered Path (device 0x1f0e0400) to PV 3 in VG 1.
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000484b8040), from raw device 0x1f0f0400 (with priority: 1, and current flags: 0x0) to raw device 0x1f0e0400 (with priority: 0, and current flags: 0x0).
LVM: Performed a switch for Lun ID = 0 (pv = 0x000000004af8c840), from raw device 0x1f0e0700 (with priority: 0, and current flags: 0xc0) to raw device 0x1f0f0700 (with priority: 1, and current flags: 0x0).
LVM: Recovered Path (device 0x1f0e0700) to PV 6 in VG 1.
LVM: Performed a switch for Lun ID = 0 (pv = 0x000000004af8c840), from raw device 0x1f0f0700 (with priority: 1, and current flags: 0x0) to raw device 0x1f0e0700 (with priority: 0, and current flags: 0x80).
LVM: Restored PV 3 to VG 1.
LVM: Restored PV 6 to VG 1.

Re: Disk Problem

Sunny123_1 — Fri, 12 Jun 2009 13:42:20 GMT

Hi

Check with lvdisplay command and see if there is any stale extents.

Regards
Sunny

Re: Disk Problem

Durvesh Mendhekar — Fri, 12 Jun 2009 13:49:31 GMT

Hi,

In logs you can see

LVM: vg[1]: pvnum=6 (dev_t=0x1f0e0700) is POWERFAILED
lv_readvgdats: Could not read VGDA 1 header & trailer from disk H/W path 1/0/4/0/0.1.18.0.0.0.4 (error = 5)
It means there is some problem with power supply.

Regards,
Durvesh

Re: Disk Problem

Mel Burslan — Fri, 12 Jun 2009 13:55:43 GMT

ioscan might say a disk is claimed which only tells you that, it can querry the disk controller and gets a response back from the unit. It does not necessarily tell you that the device as a whole unit is functional. YOu might have a bad plate on the disk or disk might not be spinning at all. The constantly lit amber light in front of the disk device is usually a good indicator of trouble.

Since you are remote, I'd suggest contacting your hardware support and have them replace this disk.

Re: Disk Problem

Waqar Razi — Fri, 12 Jun 2009 13:56:57 GMT

Are there any more commands to check the power supply status?

Re: Disk Problem

Sunny123_1 — Fri, 12 Jun 2009 13:59:45 GMT

Hi

checked in /var/opt/resmon/log/event.log

did you check lvdisplay |more
for stale extents????

Regards
Sunny

Re: Disk Problem

Mel Burslan — Fri, 12 Jun 2009 14:00:51 GMT

the POWERFAILED phrase does not refer to the power supply as the couple of posts above references. It is telling you that power is not reaching to some parts of your disk drive or maybe the whole drive (more than likely intermittently)

Re: Disk Problem

Mel Burslan — Fri, 12 Jun 2009 14:03:33 GMT

exercise your disk with this command and see if it completes successfully or errors out :

dd if=/dev/dsk/cXtXdX of=/dev/null bs=65535

it will take a long while, especially longer if this is a high capacity drive.

Re: Disk Problem

Dennis Handly — Fri, 12 Jun 2009 16:02:43 GMT

>Mel: dd if=/dev/dsk/cXtXdX of=/dev/null bs=65535
>it will take a long while

Any reason you didn't suggest a larger blocksize? bs=1024k

Re: Disk Problem

sujit kumar singh — Fri, 12 Jun 2009 17:18:40 GMT

Hi,

LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000484b8800), from raw device 0x1f0e0500 (with priority: 0, and current flags: 0x40) to raw device 0x1f0f0500 (with priority: 1, and current flags: 0x0).
LVM: Recovered Path (device 0x1f0e0500) to PV 4 in VG 1.
LVM: Performed a switch for Lun ID = 0 (pv = 0x00000000484b8800), from raw device 0x1f0f0500 (with priority: 1, and current flags: 0x0) to raw device 0x1f0e0500 (with priority: 0, and current flags: 0x0).

These are pertaining to LVM Path switching that is automatically done to get paths in case of One path is having some problem

SCSI: Write error -- dev: b 31 0x0e0400, errno: 126, resid: 8192,
blkno: 3389016, sectno: 6778032, offset: 3470352384, bcount: 8192.

SCSI: Write error -- dev: b 31 0x0e0400, errno: 126, resid: 8192,
blkno: 168464, sectno: 336928, offset: 172507136, bcount: 8192.

these should take your attention more ...

Try doing dd as suggetsed to have a look at the IO Read from that device as that shows write errors.

cd /dev/dsk

ll | grep 0x0e0400

try identifying which disk is that.
SCSI write failures are not a good indication as far as SCSI IO is concerned.

Monitor this closely -- might be a failing disk.

try to refer syslog and eventlogs that is /var/opt/resmon/log/event.log

Monitor that closely and have data backup in place proactively.

regards
sujit