Disk io error at oracle end (RAC) and getting lost device id at syslog and dmesg

[shikhar_1]

Hi All,

Its regarding the oracle team facing some disk io errors on their RAC and on the same time i can below errors in dmesg and syslog.

This is an BL870c i2 server.

Mar 27 08:00:19 urrtw41 vmunix: fclp driver at 1/0/0/7/0/0/0 (/dev/fclp2) : has lost a device with id 0x32b700, PWWN 0x50060e801534c10e.

Mar 27 08:00:19 urrtw41 vmunix: This device has disappeared from the Name Server
Mar 27 08:00:19 urrtw41 vmunix: database, LUNs behind this port will not be seen.
Mar 27 08:00:29 urrtw41 vmunix: class : tgtpath, instance 45
Mar 27 08:00:29 urrtw41 vmunix: Target path (class=tgtpath, instance=45) has gone offline. The target path h/w path is 1/0/0/7/0/0/0.0x50060e801534c10e
Mar 27 08:00:29 urrtw41 vmunix:
Mar 27 08:00:32 urrtw41 CIM Indication[2368]: Indication (default format):ProviderName = HPUXFCLPIndicationProvider, PerceivedSeverity = 7, EventID = 55
Mar 27 08:00:39 urrtw41 vmunix: Target path (class=tgtpath, instance=45) has gone online. The target path h/w path is 1/0/0/7/0/0/0.0x50060e801534c10e
Mar 27 08:01:38 urrtw41 inetd[8978]: registrar/tcp: Connection from urrtw41 (10.231.167.111) at Wed Mar 27 08:01:38 2013
Mar 27 08:00:39 urrtw41 vmunix: class : tgtpath, instance 45 

And again we had got the same issue but with the different path

Mar 28 16:16:50 urrtw41 vmunix: fclp driver at 1/0/0/7/0/0/1 (/dev/fclp3) : has lost a device with id 0x3cb700, PWWN 0x50060e801534c11e.

Mar 28 16:16:50 urrtw41 vmunix: This device has disappeared from the Name Server
Mar 28 16:16:50 urrtw41 vmunix: database, LUNs behind this port will not be seen.
Mar 28 16:17:01 urrtw41 vmunix: class : tgtpath, instance 46
Mar 28 16:17:01 urrtw41 vmunix: Target path (class=tgtpath, instance=46) has gone offline. The target path h/w path is 1/0/0/7/0/0/1.0x50060e801534c11e
Mar 28 16:17:01 urrtw41 vmunix:
Mar 28 16:17:11 urrtw41 vmunix: Target path (class=tgtpath, instance=46) has gone online. The target path h/w path is 1/0/0/7/0/0/1.0x50060e801534c11e
Mar 28 16:17:11 urrtw41 vmunix: class : tgtpath, instance 46
Mar 28 16:17:11 urrtw41 vmunix:

As per HP the HBA card is ok and storage end saying there is no issue from their end.

Please find the attachment as the errors at oracle end and details of the HBA details.

Please reply.

[Torsten.]

What kind of storage are you using?

Looks like you loose connectivity in both SANs.

[shikhar_1]

We are using Hitachi disks and as per Storage and HP, they did not find any issue