LVM: vg[2]: pvnum=0 (dev_t=0x1f008000) is POWER FAILED

[David Burgess]

I got the above error this morning on disk c0t8d0 on a D350. The disk is mirrored to a HASS disk.

syslog.log had lots of

[Sebastian Galeski_1]

Powerfailed messages in syslog - general recommendations - check HW, patches,
pvtimeouts, bus performance.

[BFA6]

Looks as though the disk is on the way out.

Does it still show up in ioscan ?

Hilary

[David Burgess]

This site is slow today. My browser went mad and submitted half the message. I'll try again!

I got the above error this morning on disk c0t8d0 on a D350. The disk is mirrored to a HASS disk.

OS is 10.20

vgdisplay hung when it got to c0t8d0 as did vgsync. The disk isn't powerfailed as the light is still flashing. Informix is moaning about it. A reboot fixed the problem.

Any ideas on what happened. Could I be short of a patch or two?

I attached the syslog errors to the previos posting.

Regards,

Dave.

[BFA6]

I would check all the disks on the bus, could be a problem with another disk that's causing the bus to hang.

If the disks are very busy, then you could try changing the pvtimeouts as previously suggested.

May also be worth logging call with HP & getting them to check it out, if problem persists.

Hilary

[Stefan Farrelly]

Youve definitely got a disk problem which will need replacement. The question is to identify which disk. The error above is sometimes wrong or misleading. The only way to confirm which disk needs replacement is to use LOGTOOl - you can run it from XSTM -> TOOLS -> UTILITY then select LOGTOOL and when it comes up select RAW SUMMARY. This will list all hardware errors by device. You should see from this tons of errors logged mostly on one disk - this is your troublesome disk - replace it.

Normally a powefail message means the disk spun down then up again which is an indication the disk is on its way to disk heaven, but if you get a hang then you know it needs replacement asap.

[S.K. Chan · Accepted solution]

It can only mean 3 things
1) Disk is failing
2) Low timeout value
3) SCSI termination and connection.

You just have to go through the process of elimination to determne your actual root cause. As suggested earlier, check for IO error using STM..
# cstm
cstm>map
cstm>sel dev
cstm>info
cstm>infolog

If you don't see any error, double check connection, cable, termination,etc. If that is ok, then is it likely to be that the timeout value of the disk needs to be increased. For example ..
# pvchange -t 180 /dev/dsk/c0td0
increases it to 180 secs. Check man page of pvchange for details.

[PIYUSH D. PATEL]

Hi,

It seems to be a disk problem. But sometimes the SCSI cards or SCSI cables can also cause this problems.
Check for loose connections.

This is a serious matter and you need to contact the HP guys.

How frequently are you getting these errors.

Piyush

[David Burgess]

The scsi connections are ok.
cstm doesn't show any errors.
Apart from y2k patches this server was last patched in February 1999!
Apparantly it's being upgraded to 11.00 soon. If not I feel a patching sesion coming on!

The server is rebooted every night. The error hasn't come back yet.
I'll keep and eye on it.

Thanks for all your help

Dave.

[T G Manikandan]

Hello David,
We had this errors of POWERFAILED disks,and the HP Engineer told us to upgrade the disk firmware version.
The model of the disks were seagate and the disk firmware version was HP01.
He upgraded it to HP04 for these errors.
We have not check that specific disk after that.

[T G Manikandan]

Just check this link

http://support2.itrc.hp.com/service/patch/patchDetail.do?patchid=PF_DSEACH3HP04&context=firmware:disk

Thanks

[Nick Wickens]

Just on a point I would make on the fact that your disk is being used for an Informix database. Informix invariably has problems for us if a disk in a mirror group becomes powerfailed (or in my recent experience swaps to an alternate fibre device).

I think that this is due to the fact that Informix (IBM) insist on using the raw device (rlvol) path to the disk ie not via LVM (lvol). This is because Informix don't like the idea of losing data in cache following a system fail. There is an option to mount LVM volumes without buffer cache (mincache) so at some point I am going to try this out as this has been a regular pain for us following disk failures for the last few years.

[David Burgess]

TG,

The disk 4GB and the firmware is already at HP03. It's not in the list. Does that mean that it can't go any higher?

Regards,

Dave.