Adapter at hardware path 0/7/0/0/0/0 : Driver has lost a device

[Aamir14]

Hi,

I saw the error message on syslog file.

Error Description:----

tail -1000000 /var/adm/syslog/syslog.log |grep CRITICAL

Mar 13 17:43:33 prodata1 EMS [2866]: ------ EMS Event Notification ------   Value: "CRITICAL (5)" for Resource: "/adapters/events/fclp_adapter/0_7_0_0_0_0"     (Threshold:  >= " 3")    Execute the following command to obtain event details:   /opt/resmon/bin/resdata -R 187826181 -r /adapters/events/fclp_adapter/0_7_0_0_0_0 -n 187826177 -a

CURRENT MONITOR DATA:

Event Time..........: Wed Mar 13 17:43:33 2013

Severity............: CRITICAL

Monitor.............: dm_fclp_adapter

Event #.............: 55

System..............: prodata1

Summary:

     Adapter at hardware path 0/7/0/0/0/0 : Driver has lost a device.

Description of Error:

 lbolt value: 541651584

 has lost a device with id 0x10400, PWWN 0x500110a0008cf060.

This device has disappeared from the Name Server

database, LUNs behind this port will not be seen.

Probable Cause / Recommended Action:

Driver has detected that a previously discovered device (Nport_id) is no longer

in the GPN_FT response, or it's corresponding World-Wide Name has changed.

Driver won't be able to communicate with the device any more, please check that

device. If the problem persists, please report the problem to Hewlett Packard.

Additional Event Data:

     System IP Address...: 192.168.4.5

     Event Id............: 0x51406d6d00000000

     Monitor Version.....: A.01.00

     Event Class.........: I/O

     Client Configuration File...........:

     /var/stm/config/tools/monitor/default_dm_fclp_adapter.clcfg

     Client Configuration File Version...: A.04.00

          Qualification criteria met.

               Number of events..: 1

     Associated OS error log entry id(s):

          0x51406d6c00000000

     Additional System Data:

          System Model Number.............: ia64 hp server rx6600

          OS Version......................: B.11.31

          EMS Version.....................: A.04.20.31.02

          STM Version.....................: D.04.00

     Latest information on this event:

          http://docs.hp.com/hpux/content/hardware/ems/dm_fclp_adapter.htm#55

v-v-v-v-v-v-v-v-v-v-v-v-v    D  E  T  A  I  L  S    v-v-v-v-v-v-v-v-v-v-v-v-v

Component Data:

    Physical Device Path....: 0/7/0/0/0/0

    Vendor Id...............: 0x000010DF

    Serial Number(WWN)......:

I/O Log Event Data:

     Driver Status Code..................: 0x00000037

     Length of Logged Hardware Status....: 0 bytes.

     Offset to Logged Manager Information: 0 bytes.

     Length of Logged Manager Information: 227 bytes.

Manager-Specific Information:

Raw data from FCLP Adapter driver:

01000000 2048F280 00000001 00000000 00000000 00000000 00000000 00000000

00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

00000000 68617320 6C6F7374 20612064 65766963 65207769 74682069 64203078

31303430 302C2050 57574E20 30783530 30313130 61303030 38636630 36302E0D

0A546869 73206465 76696365 20686173 20646973 61707065 61726564 2066726F

6D207468 65204E61 6D652053 65727665 720D0A64 61746162 6173652C 204C554E

73206265 68696E64 20746869 7320706F 72742077 696C6C20 6E6F7420 62652073

65656E

Kindly Share me what is a actual problem 

and how can we solve the error message.

Thanks

Aamir

[Jaffer]

Hi,

As per event log, It seems all the devices presented via hw path 0/7/0/0/0/0 (FC port) lost its devices.

Please check the same in ioscan -fn and verify whether all the devices under this hw path is not getting detected.

Also you can use the below command to check the FC HBA card status.

# /opt/fcms/bin/fcdutil /dev/fcd0
# /opt/fcms/bin/fcdutil /dev/fcd1

If it shows its status as "Awaiting_Link_Up" then you check the FC cable and switch ports for connectivity failure.

Also please check on the storage side for the nport id 0x10400.

Please check the fiber channel driver as well and make sure its having the latest one.

Regards.

[Matti_Kurkela]

If the HBAs are OK, then the message might be caused by the storage administrator un-presenting a LUN that was originally presented to this system, or disabling some LUN paths or storage ports when reorganizing the SAN fabric.

Use "ioscan -fnNCdisk" and/or "ioscan -P health -C disk" to see the state of your disks/LUNs: if any of them are in NO_HW or "offline" state, you should see if those disks/LUNs are used by anything on your system. This message can happen when intentionally removing a LUN, if the sysadmin has forgotten to run "rmsf" for the LUN to be removed, before telling the storage administrator to un-present the LUN.

If no disks are in NO_HW state, but the "ioscan -P health -C disk" says "limited" for some LUNs, check the individual lunpaths: "ioscan -P health -C lunpath". If some paths are offline, check with your SAN admin: it might be the result of a planned SAN reorganization, or a zoning mistake in SAN switches.

If the missing/offline LUNs or lunpaths are intentionally removed, use the "rmsf" command to remove them from the HP-UX configuration, so the operating system will stop alerting about them.

[Aamir14]

hi,

Please see the output.

prodata1#ioscan -P health -C lunpath

Class     I  H/W Path  health

===============================

lunpath   6  0/3/0/0/0/0.0x500110a0008cf05a.0x0                 online

lunpath   2  0/3/0/0/0/0.0x50014380025d94d9.0x0                 online

lunpath  13  0/3/0/0/0/0.0x50014380025d94d9.0x4001000000000000  online

lunpath  14  0/3/0/0/0/0.0x50014380025d94d9.0x4002000000000000  online

lunpath  16  0/3/0/0/0/0.0x50014380025d94d9.0x4003000000000000  online

lunpath  17  0/3/0/0/0/0.0x50014380025d94d9.0x4004000000000000  online

lunpath  18  0/3/0/0/0/0.0x50014380025d94d9.0x4005000000000000  online

lunpath  19  0/3/0/0/0/0.0x50014380025d94d9.0x4006000000000000  online

lunpath  20  0/3/0/0/0/0.0x50014380025d94d9.0x4007000000000000  standby

lunpath  21  0/3/0/0/0/0.0x50014380025d94d9.0x4008000000000000  online

lunpath  47  0/3/0/0/0/0.0x50014380025d94d9.0x400a000000000000  online

lunpath  50  0/3/0/0/0/0.0x50014380025d94d9.0x400b000000000000  standby

lunpath  54  0/3/0/0/0/0.0x50014380025d94d9.0x400c000000000000  online

lunpath  55  0/3/0/0/0/0.0x50014380025d94d9.0x400d000000000000  online

lunpath  62  0/3/0/0/0/0.0x50014380025d94d9.0x400e000000000000  online

lunpath  67  0/3/0/0/0/0.0x50014380025d94d9.0x400f000000000000  online

lunpath  75  0/3/0/0/0/0.0x50014380025d94d9.0x4010000000000000  online

lunpath  71  0/3/0/0/0/0.0x50014380025d94d9.0x4014000000000000  online

lunpath  87  0/3/0/0/0/0.0x50014380025d94d9.0x4032000000000000  online

lunpath  78  0/3/0/0/0/0.0x50014380025d94d9.0x4064000000000000  online

lunpath   5  0/3/0/0/0/0.0x50014380025d94dd.0x0                 online

lunpath   9  0/3/0/0/0/0.0x50014380025d94dd.0x4001000000000000  standby

lunpath  10  0/3/0/0/0/0.0x50014380025d94dd.0x4002000000000000  standby

lunpath  11  0/3/0/0/0/0.0x50014380025d94dd.0x4003000000000000  standby

lunpath  12  0/3/0/0/0/0.0x50014380025d94dd.0x4004000000000000  standby

lunpath  15  0/3/0/0/0/0.0x50014380025d94dd.0x4005000000000000  standby

lunpath  23  0/3/0/0/0/0.0x50014380025d94dd.0x4006000000000000  standby

lunpath  24  0/3/0/0/0/0.0x50014380025d94dd.0x4007000000000000  online

lunpath  25  0/3/0/0/0/0.0x50014380025d94dd.0x4008000000000000  standby

lunpath  48  0/3/0/0/0/0.0x50014380025d94dd.0x400a000000000000  standby

lunpath  49  0/3/0/0/0/0.0x50014380025d94dd.0x400b000000000000  online

lunpath  60  0/3/0/0/0/0.0x50014380025d94dd.0x400c000000000000  standby

lunpath  61  0/3/0/0/0/0.0x50014380025d94dd.0x400d000000000000  standby

lunpath  63  0/3/0/0/0/0.0x50014380025d94dd.0x400e000000000000  standby

lunpath  69  0/3/0/0/0/0.0x50014380025d94dd.0x400f000000000000  standby

lunpath  77  0/3/0/0/0/0.0x50014380025d94dd.0x4010000000000000  standby

lunpath  73  0/3/0/0/0/0.0x50014380025d94dd.0x4014000000000000  standby

lunpath  88  0/3/0/0/0/0.0x50014380025d94dd.0x4032000000000000  standby

lunpath  80  0/3/0/0/0/0.0x50014380025d94dd.0x4064000000000000  standby

lunpath   0  0/6/0/0/0/0/4/0/0/0.0x0.0x0                        online

lunpath   1  0/6/0/0/0/0/4/0/0/0.0x0.0x4000000000000000         online

lunpath   8  0/7/0/0/0/0.0x500110a0008cf060.0x0                 online

lunpath  53  0/7/0/0/0/0.0x500110a0008cf060.0x1000000000000     online

lunpath   3  0/7/0/0/0/0.0x50014380025d94d8.0x0                 online

lunpath  31  0/7/0/0/0/0.0x50014380025d94d8.0x4001000000000000  online

lunpath  32  0/7/0/0/0/0.0x50014380025d94d8.0x4002000000000000  online

lunpath  33  0/7/0/0/0/0.0x50014380025d94d8.0x4003000000000000  online

lunpath  35  0/7/0/0/0/0.0x50014380025d94d8.0x4004000000000000  online

lunpath  36  0/7/0/0/0/0.0x50014380025d94d8.0x4005000000000000  online

lunpath  37  0/7/0/0/0/0.0x50014380025d94d8.0x4006000000000000  online

lunpath  38  0/7/0/0/0/0.0x50014380025d94d8.0x4007000000000000  standby

lunpath  39  0/7/0/0/0/0.0x50014380025d94d8.0x4008000000000000  online

lunpath  46  0/7/0/0/0/0.0x50014380025d94d8.0x400a000000000000  online

lunpath  52  0/7/0/0/0/0.0x50014380025d94d8.0x400b000000000000  standby

lunpath  56  0/7/0/0/0/0.0x50014380025d94d8.0x400c000000000000  online

lunpath  57  0/7/0/0/0/0.0x50014380025d94d8.0x400d000000000000  online

lunpath  64  0/7/0/0/0/0.0x50014380025d94d8.0x400e000000000000  online

lunpath  66  0/7/0/0/0/0.0x50014380025d94d8.0x400f000000000000  online

lunpath  74  0/7/0/0/0/0.0x50014380025d94d8.0x4010000000000000  online

lunpath  70  0/7/0/0/0/0.0x50014380025d94d8.0x4014000000000000  online

lunpath  86  0/7/0/0/0/0.0x50014380025d94d8.0x4032000000000000  online

lunpath  79  0/7/0/0/0/0.0x50014380025d94d8.0x4064000000000000  online

lunpath   4  0/7/0/0/0/0.0x50014380025d94dc.0x0                 online

lunpath  27  0/7/0/0/0/0.0x50014380025d94dc.0x4001000000000000  standby

lunpath  28  0/7/0/0/0/0.0x50014380025d94dc.0x4002000000000000  standby

lunpath  29  0/7/0/0/0/0.0x50014380025d94dc.0x4003000000000000  standby

lunpath  30  0/7/0/0/0/0.0x50014380025d94dc.0x4004000000000000  standby

lunpath  34  0/7/0/0/0/0.0x50014380025d94dc.0x4005000000000000  standby

lunpath  41  0/7/0/0/0/0.0x50014380025d94dc.0x4006000000000000  standby

lunpath  42  0/7/0/0/0/0.0x50014380025d94dc.0x4007000000000000  online

lunpath  43  0/7/0/0/0/0.0x50014380025d94dc.0x4008000000000000  standby

lunpath  45  0/7/0/0/0/0.0x50014380025d94dc.0x400a000000000000  standby

lunpath  51  0/7/0/0/0/0.0x50014380025d94dc.0x400b000000000000  online

lunpath  58  0/7/0/0/0/0.0x50014380025d94dc.0x400c000000000000  standby

lunpath  59  0/7/0/0/0/0.0x50014380025d94dc.0x400d000000000000  standby

lunpath  65  0/7/0/0/0/0.0x50014380025d94dc.0x400e000000000000  standby

lunpath  68  0/7/0/0/0/0.0x50014380025d94dc.0x400f000000000000  standby

lunpath  76  0/7/0/0/0/0.0x50014380025d94dc.0x4010000000000000  standby

lunpath  72  0/7/0/0/0/0.0x50014380025d94dc.0x4014000000000000  standby

lunpath  89  0/7/0/0/0/0.0x50014380025d94dc.0x4032000000000000  standby

lunpath  81  0/7/0/0/0/0.0x50014380025d94dc.0x4064000000000000  standby

prodata1#

prodata1#ioscan -P health -C disk
Class I H/W Path health
===============================
disk 1 0/0/2/1.0x0.0x10 N/A
disk 0 64000/0xfa00/0x1 online
disk 40 64000/0xfa00/0xf online
disk 53 64000/0xfa00/0x18 online
disk 58 64000/0xfa00/0x19 online
disk 76 64000/0xfa00/0x1f online
disk 77 64000/0xfa00/0x20 online
disk 78 64000/0xfa00/0x21 online
disk 79 64000/0xfa00/0x22 online
disk 80 64000/0xfa00/0x23 online
disk 81 64000/0xfa00/0x24 online
disk 82 64000/0xfa00/0x25 online
disk 83 64000/0xfa00/0x26 online
disk 84 64000/0xfa00/0x27 online
disk 87 64000/0xfa00/0x28 online
disk 92 64000/0xfa00/0x29 online
disk 97 64000/0xfa00/0x2a online
disk 104 64000/0xfa00/0x2b online
disk 105 64000/0xfa00/0x2c online
disk 113 64000/0xfa00/0x2e online
prodata1#

every disk and LUN are online.