LVM and Compaq SAN

[Keely Jackson]

Hi

We are running 2 k580's one at 10.20 the other at 11.0 both attached via fibre channel to a compaq switch to a compaq Storage Area Network. Patch level on both machines should be fine.

We are getting lots of EMS errors reported both in the syslog and event.log regarding disk power failures, scsi read, write errors. More errors are logged in the syslog than the event log. The users are complaining about poor performance (possibly is co-incidental). I have increased the disk time out period to 180 seconds but it made no difference.

Compaq are saying there is nothing wrong with the storage their end, no errors are reported by the switch or controllers and HP are saying nothing wrong their end, no errors on the adaptor card.

I notice Kurt Beyers had a similar problem back in december but unfortunately no resolution was mentioned.

If anybody has any ideas I would be most grateful.

Thanks

Keely

[Clemens van Everdingen]

Hi,

1. Make sure that the latest SCSI/LVM patch (and its dependencies) are installed.


2. Check for an I/O bottleneck on the disk.
sar -d

C.

[Michael Tully]

You also might look at turning off the EMS monitoring for the time being. This might not help with the all the traffic being generated.

[S.K. Chan]

Did you made a hardware call to HP response center showing them these errors ? There has to be something wrong or if not they should provide possible explanation why these error logs show up.

..LVM: vg[2]: pvnum=0 (dev_t=0x1f052200) is POWERFAILED
==> c5t2d2
..LVM: VG 1 : PV 0 (device 0x1f033100) is POWERFAILED
==> c3t3d1

Check a few thing..
1) In LVM are there any stale extents ?
2) diskinfo on those disk paths, do they hungup?
3) run exc (exercise) on the suspected disk via cstm to test the disk.

[Kurt Beyers.]

Hi Keely,

We installed quiet recent Secure Path Software of Compaq to perform load balancing across the physical paths towards the LUN's. We had to uninstall first an old Compaq driver for the fibre channel cards (don't know it's name anymore) before the Secure Path could be installed. After this uninstall and with Secure Path, we didn't receive any errors anymore (at least until now).

My guess is that the Compaq driver for the fibre channel cards was casung conflicts with the HP fibre channel card. I don't know if you have such a driver installed as well.

Kurt

[Duncan Edmonstone]

COnsider replacing fibre cables for starters, then consider replacing fibre cards...

Check out the man page for fcmsutil...

there's an option on here which allows you to gathers stats for the fibre cards, specifically the number of errors reported - I'm not at a HP system right now so i can't give you the exact syntax, but I seem to remember it was something like:

fcmsutil /dev/td0 stat

Where the device name is grabbed from an ioscan...

This might at least tell you if you have a real physical problem on the fibre, or just logical problem in LVM/SAN

HTH

Duncan

[Keely Jackson]

The following seems to have resolved the problem:

upgraded switch and controllers to the latest version of the firmware

applyed 2 fc adaptor patches and once scsi patch (PHKL_23939, PHKL_25769, PHKL_25938) on 11.0 and an fc adpator patch (PHSS_23581) on 10.20

iostat and sar stats now show sensible numbers and ems errors are no longer reported. Fantastic.

Thanks to you all for taking the time to make your suggestions. Much appreciated.

Cheers
Keely