lun path has come online / intermittant disk problem

[Alan Casey]

 I am having intermittant problems with disk i/o

Sar -d shows 100% busy on SAN Fiber attached disks.

14:55:03   c5t0d0  100.00    1.67       4      38    0.16  602.71
           disk11    5.39    0.71      14     424    1.34    8.60
           disk15  100.00    1.67       4      38    0.16  602.71
           disk41    0.20    0.50       5      43    0.00    0.32

Syslog shows at the same time:

 vmunix: lun path (class = lunpath, instance = 24) belonging to LUN (default minor = 0x16) has come online

repeated

There are no errors suggesting anything going offline

HPUX 11iv3, the disks are being accessed by Oracle ASM and are share with another Oracle cluster node.

Any ideas?

[Bill Hassell]

14:55:03   c5t0d0  100.00    1.67       4      38    0.16  602.71
           disk11    5.39    0.71      14     424    1.34    8.60
           disk15  100.00    1.67       4      38    0.16  602.71
           disk41    0.20    0.50       5      43    0.00    0.32

 Having the storage links coming online is a definite hardware failure -- that is not supposed to happen. Something is wrong with your LUNS. c5t0d0 and disk15 are taking 50-75 times too long to complete I/Os. With no other indication in the HP-UX syslog, the disk storage logs, possibly SAN switches will provide the  answer.

[chris huys_4]

> .. Oracle ASM

There is your problem. ;)

>14:55:03   c5t0d0  100.00    1.67       4      38    0.16  602.71

You shouldnt need to see legacy device files on a hp-ux 11.31 system in the sar -d output. Probably lvm volumegroup still defined partially with legacy instead of agile (/dev/rdisk) device files ?

>disk15  100.00    1.67       4      38    0.16  602.71

Not a lot of IOs. and small IOs btw. but 602msec per IO is a lot of time needed by the storage, for a on average 4kb IO, when now IO service times of on average 5msec are expected.

> vmunix: lun path (class = lunpath, instance = 24) belonging to LUN (default minor = 0x16) has come online

repeated

With ioscan -fnN output, you should be able to link the lunpath with instance 24 with the associated /dev/rdisk devicefile.

Probably fcddiag/fclpfiag ;escsi_diag will show more details.

supportshow/supportsave on the sanswitches. if they are brocade sanswitches. should also show something.

and logs on the san storage.